I need to merge more than 800 CSV files that have an identical structure into 1 file on Azure Data Lake Storage GEN2 using a ADF pipeline. Up to how many files can I merge into one and what is the recommended maximum size of the output file?
1 Answers
Up to how many files can I merge into one?
There might not be no limit on the number of files that can be merged into a single file. The following is a demonstration where I have merged up to 10,000 files into one single file.
- The copy data activity merge:
- The output:
what is the recommended maximum size of the output file?
Since you are merging your files to store it in Azure Data Lake gen2 storage, this data lake storage allows you to store up to 5TB for a single file with few performance limitations.
According to this official Microsoft documentation for Best practices for using Azure Data Lake Storage Gen2, storing many small files reduce performance (which is not the case here). A file with size greater than 100GB also reduces efficiency. Therefore, it is recommended that the optimal output file size would be something less than 100GB.
Related