How to create iteration scoped variables inside ForEach activities in Azure Data Factory

user_7596217 0
Viewed 3896

I have a ForEach activity where inside each iteration, I need to set a few iteration specific variables. I can achieve this by using variables defined for the pipeline (pipeline scope), but this forces me to run the loop in Sequential mode so that multiple iterations running in parallel will not update the same variable. What I really need is the ability to define these variables within each iteration (iteration scope) so that I can run the ForEach activity in parallel mode.

enter image description here

enter image description here

I've considered creating a SQL dataset where I could do a lookup for fake values (SELECT 1 AS var1, 2 AS var2) just to get a structure where I can set and use those values, but that seems really lame. I've also considered using an array variable type with the AppendVariable option, but that introduces a lot of custom parsing.

Would be nice if I could just have an InMemory dataset that doesn't have to be tied to a data source where I could use it as a structure inside my ForEach iteration. Does anyone have any other ideas about how to set iteration specific variables inside ForEach loop?

azure-data-factory-2 azure-data-factory
3 Answers

I agree, this is very annoying and irritating.

If the first part of Jason's answer is vaiable for your situation, then that's definitely the way to go. (Define the variables outside the loop).

But assuming that the variables are dynamically calculated per-iteration, then the only solution that I know is to define the body of the Foreach loop as its own pipeline. Now you can define variable inside that inner pipeline, which are "scoped" to the separate executions of the inner-pipeline.

Quite a lot of ADF's pipeline limitations can be circumvented like this. Nested Ifs/Foreaches, Activity limits, etc.

user_16622680

About the best way to currently do this it to pull the values from an outer lookup or get metadata activity if you can. using the inner lookup wouldn't be as cost efficient or performance efficient. especially if you were iterating over 100's or thousands. Of course this is if you can determine the values for each iteration ahead of time. if you can't than. I would drably go for your lookup approach. or if you can get away from the variables entirely just set the values using an expression using dynamic properties.

user_882690

Another workaround that worked for me was to use a Filter Activity. It's not super pretty but can help.

Say you want to assign expr to a variable. Just add a filter activity and configure it like this:

Items: @array(expr)

Condition: @equals(1, 1)

Then instead of using a variable simply use the Filter Activity output like this:

@first(activity('<your filter activity name>').output.Value)

user_193553090
Related