how to merge all files using copy activity or other tools in ADF

Question

I have json files sitting under differnt subfolders. folder structures are like this below

/UserData/data/json/A/2021/01/28/03/
A_2021_01_28_03_file1.json
A_2021_01_28_03_file2.json
A_2021_01_28_03_file3.json
/UserData/data/json/A/2021/01/28/02/
A_2021_01_28_02_file1.json
A_2021_01_28_02_file2.json
/UserData/data/json/B/2021/03/27/02/
A_2021_03_27_02_file1.json
A_2021_03_27_02_file2.json

/UserData/data/json/C/2021/04/21/01/
A_2021_04_21_01_file1.json
A_2021_04_21_01_file2.json

I want to merge all the files available under A folder/B folder/C folder and ingest them as A table, B table and C table in Azure Data Explorer.

the schema is

name string
timestamp date
value string

I don't see merge feature in copy activity, how could I achieve this?

appreciate your help

user_6788885 · Answer

You need 3 copy activities. In each copy activity, in source -> select "Wildcard file path" and choose * to select all files.(see attached picture).

it will copy all the files under the specific folder

please read more: https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage?tabs=data-factory

how to merge all files using copy activity or other tools in ADF

1 Answers