Azure Data Factory Copy Data From Blob Storage Only New Added file(s)

Question

I would like to create a pipeline with Copy Data (mapping JSON source to MySQL columns). At this moment it is working, but the tool is copying all the files from my source container after the 'created' trigger. Is it possible to retrieve only 'new added' files in my container, so I do not have double data in MySQL?

Example current situation:

first copy -> retrieve file 1
(added new file)
second copy -> retrieve file 1 + file 2

Example expected situation:

first copy -> retrieve file 1
(added new file)
second copy -> retrieve file 2

user_6788885 · Answer

This is actually a very used scenario , you should use LastModifiedDate for copying most recently added files and to prevent duplication.

Please check this link : https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

Azure Data Factory Copy Data From Blob Storage Only New Added file(s)

1 Answers