Here is what I am trying to do.
NOTE: I am newbie to Airflow.
Currently we have a step function that uses Map functionality, so I would pass an array of values to AWS Lambdas and the step function would loop through array, and fire a lambda(Concurrency is set to 5, so 5 aws lambdas are running at any given time) with a value as payload, and keep repeating it once an Aws lambda function is done.
What I am looking to accomplish is, can something like this be done using Airflow ? Where I have a task that fires say 5 Lambdas with the values 0 till 4 from the array and once each lambda is done, another Lambda is fired with the next value as payload ? The array is calculated ahead of time in the pre-processing step.
The AWS Lambdas are pulling data from external system using APIs. Everything is done in Python.
Thanks