ADF-how to compare two get metadata activity results and skip if the same name exists

Shaik Haji 0
Viewed 60

Just wondering if someone could help me with a problem I have I am trying to set up a Pipeline to download files from an FTP Server into Blob

I have 2 get metadata activities 1 to get the filenames from Sftp site and other one to bring the files form Blob storage. I need to compare the results of both the activities and pick only the files which are not processed.

Example:

Get Metadata Activity 1 results: {"childItems": [ {"name": "2022.txt", "type": "File" }]}

Getmetadata activity 2 results: {"childItems": [ {"name": "2022.txt", "type": "File" }]}

I need to match the childItems.name of both the activities and if the same file contains in both the activities then Skip the process, if not exists then process that file Any help?

Thanks in Advance.

azure-data-factory
1 Answers

You can use Filter activity to get the required files list.

Please go through a sample demonstration below.

These are my source files from first Get Meta Data:

enter image description here

Files from target Meta data activity:

enter image description here

Now use Filter and give Source child items as Items. @activity('files from source').output.childItems

enter image description here

For condition give the below expression.

@not(contains(activity('files from target').output.childItems,item()))

This condition will give the list of Source files which are not exists in the Target files as array.

enter image description here

Filter Result:

enter image description here

You can use this array as @activity('Filter1').output.Value as per your requirement.

Rakesh Govindula0
Related