Working around AWS Step Function Map concurrency limit

user_718526 0
Viewed 2005

I have a Map task in an AWS Step Function which executes 100-200 lambdas in parallel, each running for a few minutes, then collects the results. However, I'm running into throttling where not all lambdas are started for some time. The relevant AWS documentation says you may experience throttling with more than ~40 items, which is what I believe I'm running into.

Does anyone have any experience with working around this concurrency limitation? Can i have nested Maps, or could I bucket my tasks into several Maps I run in parallel?

aws-step-functions amazon-web-services
2 Answers

Use nested state machine inside your map state, so you can have ~40 child state machines executing in parallel. Then inside each child state machine use a map state to process ~40 items in parallel. This way you can reach processing ~1600 items in parallel.

But before reaching that you will reach AWS Step Functions Quotas:

https://docs.aws.amazon.com/step-functions/latest/dg/limits.html

user_143028520

I ended up working around this 40 item limit by creating 10 copies of the Map task in a Parallel, and bucketing up the task information to split tasks between these 10 copies. This means I can now run ~400 tasks before running into throttling issues. My state machine looks something like this:

state machine

user_7185260
Related