I need to transfer around 20 CSV files inside a folder named ActivityPointer in an azure blob storage container to Azure SQL database in a single data factory pipeline, but ActivityPointer contains 20 CSV files and another folder named snapshots inside it. So when I try to create a pipeline and give * to select all the CSV files inside ActivityPointer it includes the snapshots folder too, which should not be included. Is there any possibilities to complete this task. Also I can't create another folder to transform the snapshots folder into it. What can I do now? Anyone can please help me out.
2 Answers
Assuming you want to copy all CSV files within ACtivityPointer folder, You can use wildcard expression as below :
you can provide path till Active folder and than *.csv
Copy data is also considering the inner folder while using wildcards (even if we use .csv in wildcard file path). So, we have to validate whether it is a file or folder. Please look at the following demonstration.
- First use
Get Metadataon the required folder with field list asChild items. The debug output will be:
- Now use this to iterate through child items using
For eachactivity.
@activity('Get Metadata1').output.childItems
- Inside for each, use
if conditionactivity to check whether the current item is a file or not. Use the following condition.
@equals(item().type,'File')
- When this is true, you can use
copy datato complete copying the file to target table (Ignore the false case). I have createfile_nameparameter in my source dataset passing its value as@item().name().
- This will help you to achieve your requirement. The following is the debug output. I have 4 files and 1 folder. The folder will be ignored, and the rest will be copied into the target table.
Related