I'm currently tasked to load about 3600 small csv files from Azure Data Lake into a single table in Azure Synapse. Each file is less than 10k bytes. I used Polybase to create an external table to point to a folder on Data lake with some wildcards and inserted records from the external table into another staging table. The whole process took about 4hrs to complete. Is there anyway to see if there is any parallelism taking place? Is the time taken normal?
0 Answers
Related