I have a json file and it contains the start date and end date. I need to iterate over this start date and end date with azure data factory foreach activity. As per my knowledge, the foreach expect the items (collection/array). But in my case, I have only two items which are start and end date. I want to run the data factory for process some historic data. I don't have the collection of dates, so how can I iterate this with start date and end date? If someone can help me to figure it out, it would be great.
2 Answers
My suggestion will be to use the UNTIL activity to iterate from start date to end date.
Like FOR-EACH, UNTIL is also used for looping in ADF. While for-each iterates over a defined collection, Until iterates till a criteria is met. This is equivalent of while loop of programming languages.
You can assign both start and end dates to a variable and use a counter variable to increment dates one by one from start to end date.
You may use AddDays expression function for incrementing a day and equals function for comparing dates
The for-each loop has no intelligence built in to calculate the number of iterations it has to repeat for certain tasks added into the looping activity.
What can be done here is to use some sort of watermark inputs and run the data load using these watermarks. Tasks such as lookup activity will come handy to emulate this behavior.
Related