Performance issues when merging large files into one single file

user_12496643 0
Viewed 249

I have a pipeline contains multiple copy activity, and the main purpose of these activities is to merge multiples files into one single file. the problem of this pipeline is, it takes about 4 hours to executes (to merge the files).is there any way to reduce the duration please.

thanks for your reply .

azure-data-factory
1 Answers

  1. If the copy operation is being performed on an Azure integration runtime, the following steps must be followed:

    For Data Integration Units (DIU) and parallel copy settings, start with the default values.

  2. If you're using a self-hosted integration runtime, you'll need to do the following:

    Would recommend that you run IR on a separate computer. The machine should be kept isolated from the data store server. Start using the default defaults for parallel copy settings and the self-hosted IR on a single node.

Else you may leverage:

  1. A Data Integration Unit (DIU)

It is a measure that represents the power of a single unit in Azure Data Factory and Synapse pipelines. Power is a combination of CPU, memory, and network resource allocation. DIU only applies to Azure integration runtime. DIU does not apply to self-hosted integration runtime.

  1. Parallel Copy

Could set the parallel Copies property to indicate the parallelism you want the copy activity to use. Think of this property as the maximum number of threads within the copy activity. The threads operate in parallel. The threads either read from your source, or write to your sink data stores.

Here, is the MSFT Document to Troubleshoot copy activity performance.

When copying data into Azure Table, default parallel copy is 4.he range of DIU setting is 2-256.However, specific behaviors of DIU in different copy scenarios are different even though you set the number as you want.

Please see the table list here,especially for the below part

enter image description here

DIU has some limitations as you seen,so you could choose the optimal setting with your custom scenario.

If you are trying to copy 1GB data, thus somehow DIU never crossed 4.

But when If you try to copy 10GB data, then you could notice DIU started scaling up beyond 4.

Here is the list of the Data Integration Units.

enter image description here

user_159695280
Related