I am looking for an ADF solution to introduce workload management for a metadata driven ingestion system. In the pipeline, I read data from some metadata table into a lookup activity and say the data looks something like this
ObjectName,Tshirtsize,TaskGroup,IncrementalLoadFlag,InitialLoadFlag
Asset1,Large,1,N,Y
Asset2,Large,1,N,Y
Asset3,Large,1,N,Y
Asset4,Small,2,N,Y
Now I have to process this data in a foreach in sequential manner based on the value of TaskGroup, as in my first batch, I need to process the 3 tables which has the same TaskGroup and copy them asynchronously after determining the load flags.
However as far as I have seen foreach it will iterate every item of the lookup output one after the another and as a result I am not able to iterate on the grouped data (based on TaskGroup) for bulk load.
Is there a solution how this scenario can be implemented?