Move folders from azure containers which is having 2 levels of sub-folders to container level with the same name as the sub folder using ADF

user_15521809 0
Viewed 323

There is a blob storage account, the name of the blob storage account is azureblob11 it has a container with the name source.

The container source contains 2 level subfolders.

enter image description here

The folder structure looks similar:

source (container)
  |--a (folder)
     |--ana (sub-folder)
     |--hem (sub-folder)
     |--thg (sub-folder)
     |--oud (sub-folder)

The goal that I am trying to achieve is that copy all the subfolders (ana, hem, thg, oud) along with their data to the container level and create these 4 subfolders as new containers along with the data in them as is.

The expected goal looks like this:

source (container)
  |--a (folder)
     |--ana (sub-folder)
     |--hem (sub-folder)
     |--thg (sub-folder)
     |--oud (sub-folder)
ana (container)
hem (container)
thg (container)
oud (container)

To achieve this goal, I have used the copy data pipeline in ADF. Selected the source. Chose the source data set -> select New -> chosen the type as Azure blob storage -> chosen the format as binary -> chosen the linkeservice as movecontainer

enter image description here

Selected the sink -> select New -> chosen the type as Azure blob storage -> chosen the format as binary

enter image description here

I am stuck on how to configure the parameters or the dynamic content for creating the containers.

azure-data-factory-2 azure-data-factory azure-blob-storage azure-devops azure
1 Answers

Use Get Metadata activity to get the list of subfolder names and pass it to ForEach activity to copy the folders to sink as shown below.

Source structure in Azure data lake:

enter image description here

ADF pipeline:

  1. Using the Get Metadata activity, get the list of subfolder names under the folder “a” and container “source”.

    • Create a dataset for the source path and add it to the Get Metadata dataset.

enter image description here

• Select child items under the field list in dataset properties.

enter image description here

• Output of Get Metadata:

enter image description here

  1. Pass this output to ForEach activity.

• Under items property, add Get Metadata output child items.

enter image description here

  1. Add Copy data activity inside ForEach activity.

• Create source dataset and parameterize the subfolder (directory) path in the source dataset as shown below.

enter image description here

• In copy data activity, pass current ForEach item name to the dataset parameter value in source properties.

enter image description here

• Create sink dataset and parameterize the sink container path in the sink dataset.

enter image description here

• In the Copy activity sink dataset, pass the current ForEach item name to the sink parameter.

enter image description here

  1. Subfolders and files from each subfolder are copied to the sink. It creates a sink container with the current item name if does not exist.

enter image description here

user_159691070
Related