Azcopy throws error while executing via Terraform

user_11225799 0
Viewed 1334

I am using the Azcopy tool to copy a storage account to another. While executing the command using terminal it executes perfectly. But while executing the same using Terraform's local-executioner it throws an error. Please find the code and error below.

Code:

resource "null_resource" "backup" {
      provisioner "local-exec" {
     command= <<EOF
        azcopy cp "https://${var.src_storage_acc_name}.blob.core.windows.net${var.src_sas}" "https://${var.dest_storage_acc_name}.blob.core.windows.net${var.dest_sas}"
        EOF
  }
}

Error:

 Error running command '        azcopy cp "https://strsrc.blob.core.windows.net?[SAS]" "https://strdest.blob.core.windows.net?[SAS]"
': exit status 1. Output: INFO: The parameters you supplied were Source: '"https://strsrc.blob.core.windows.net?[SAS]-REDACTED- of type Local, and Destination: '"https://strdest.blob.core.windows.net?[SAS]-REDACTED- of type Local
INFO: Based on the parameters supplied, a valid source-destination combination could not automatically be found. Please check the parameters you supplied.  If they are correct, please specify an exact source and destination type using the --from-to switch. Valid values are two-word phases of the form BlobLocal, LocalBlob etc.  Use the word 'Blob' for Blob Storage, 'Local' for the local file system, 'File' for Azure Files, and 'BlobFS' for ADLS Gen2. If you need a combination that is not supported yet, please log an issue on the AzCopy GitHub issues list.

failed to parse user input due to error: the inferred source/destination combination could not be identified, or is currently not supported

Please provide your thoughts on this.

terraform0.12+ terraform-provider-azure terraform
2 Answers

Today I needed to implement a similar task, and I used the azcopy cp command with --recursive=true option which is given in the document.

It successfully copied all contents of the source container to the destination.

Copy all blob containers, directories, and blobs from storage account to another by using a SAS token:

  - azcopy cp "https://[srcaccount].blob.core.windows.net?[SAS]" "https://[destaccount].blob.core.windows.net?[SAS]" --recursive=true
user_108917500

azcopy only support certain combinations of source and destination types (blob, Gen1, Gen2, S3, Local file system, ...) for copy sub-command.

azcopy tries to guess source/destination types based on URL & params.

This error means that

  1. you're trying to use a combination that isn't supported OR
    • Nothing you can do. Raise a issue as suggested. They'll probably just ignore it like this or this.
  2. there is something wrong with your URL. E.g. you have blob.core.windows.net when you should've had dfs.core.windows.net or vice versa. This in turn causes mis-identification of source and destination types.
    • If you're sure that the combination is supported then you can tell azcopy the types using --from-to. Ironically, when you use a combination that isn't supported (e.g. BlobFSBlobFS), it gives the same error message instead of saying "source destination combination not supported).
    • When dealing with Gen2, you could use blob instead of dfs in the URL to make it use older (blob/Gen1) APIs to interact with your Gen2 account. Though less performant, it still might work.
  • 'Blob' for Blob Storage
  • 'Local' for the local file system
  • 'File' for Azure Files
  • 'BlobFS' for ADLS Gen2

As of now following combinations are supported per documentation:

  • local <-> Azure Blob (SAS or OAuth authentication)
  • local <-> Azure Files (Share/directory SAS authentication)
  • local <-> Azure Data Lake Storage Gen 2 (SAS, OAuth, or shared key authentication)
  • Azure Blob (SAS or public) -> Azure Blob (SAS or OAuth authentication)
  • Azure Blob (SAS or public) -> Azure Files (SAS)
  • Azure Files (SAS) -> Azure Files (SAS)
  • Azure Files (SAS) -> Azure Blob (SAS or OAuth authentication)
  • Amazon Web Services (AWS) S3 (Access Key) -> Azure Block Blob (SAS or OAuth authentication)
  • Google Cloud Storage (Service Account Key) -> Azure Block Blob (SAS or OAuth authentication) [Preview]
user_4962890
Related