Azure Data factory - get metadata activity - The length of execution output is over limit (around 4MB currently)

user_1983001 0
Viewed 576

I am facing some issue and hoping that some of you have solved this before and can help me. :)

I have a pipeline in ADF which uses get metadata activity to get list of files and then it iterates each items and based on certain values derived from filename, it unzips file and saves it to another folder.

Now the thing is each folder contains lots of files and I get 4 mb limitation error. Do you have experience this? what is the best approach to solve this? Should i consider to write azure function? Any other solution ?

azure-data-factory out-of-memory
1 Answers

Yes, today it is a limitation with Get Metadata activity that it cannot return results > 4MB. One possible approach is to modify source files to load into separate folders each with < 5000 files and they can be accessed by Get Metadata activities separately. Since this requires change from source, another possible approach is to use "Azure Functions" to get the list of files information and then pass it to Foreach or entirely azure function can be used to implement entire requirement.

limitation : https://docs.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity

Ref : https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-list?tabs=dotnet https://www.mssqltips.com/sqlservertip/6361/process-blob-files-automatically-using-an-azure-function-with-blob-trigger/

user_114054230
Related