Is there a way to compare two Get Metadata Activity Child Items in ADF?

user_18338232 0
Viewed 169

Just wondering if someone could help me with a problem I have.

I am trying to set up a Pipeline to download files from an FTP Server into ADLS2, as the Pipeline will run everyday I need it to only download new files from the FTP Server.

As I have no control over the files that are input I will need to check whether the filename is already existent in ADLS2.

I have attempted to use a ForEach Loop and pass through each filename from the FTP Server (from the relative Get Metadata Activity) to check whether it is contained in the ADLS2 Folder (Get Metadata Activity). However, this process is incredibly slow when dealing with a large number of files (10k+).

Does anyone have any ideas of how I could do a bulk comparison between the two outputs?

azure-data-factory ftp loops
1 Answers

First: to reduce runtime you can do many things to optimize your pipeline, it depends on your use-case for instance :

  1. build integration runtime closer to the db, if your database is on-premise, dont make your integration runtime in the cloud, because each time you need to access the data , it will go over the network layers,firewalls etc which will increase runtime.

  2. compress data before moving it to the cloud,when you do that , you will have less time moving/coping data to the cloud.

  3. if you have a huge volume of data (lets say terabytes) use Azure databox https://azure.microsoft.com/en-us/services/databox/

  4. the best way to copy data from on-premise to the cloud is to use the Az-Copy command. https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy

check this link to build a baseline: https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance#performance-tuning-steps

Second , copying new data is done by using LastModifiedDate On the Properties page, take the following steps:

Under Task type, select Built-in copy task,when you get to the target, make sure to use : incremental load as described in the attached screenshot.

please check this link out : https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-lastmodified-copy-data-tool

enter image description here

user_67888850
Related