How many items can AWS Step Function Map task handle?

user_2328154 0
Viewed 2002

I am thinking of using this solution to process about 3000 objects from S3.

https://docs.aws.amazon.com/step-functions/latest/dg/sample-map-state.html

My use case is that I need to loop through all items one at a time, passing each file to multiple lambda functions for processing, until all items have been processed. I haven't been able to find any information to tell me if there are limitations to this approach. All the examples I have seen use a small number of items and I was wondering if there was any reason that I shouldn't use it for 3000 (or more) items.

aws-step-functions aws-lambda amazon-web-services amazon-s3
1 Answers

AWS Step Functions has 25,000 events limit for maximum execution history size. So if you have 5 steps for processing one file you can't process more than 5000 items.

To overcome this limit:

  • Wrap all the process for the file in one step functions and call if from your parent step functions (Nested Step Functions).

  • Instead of passing 1 file to your map or nested step functions, pass a batch of them and them iterate over them in your child state machine. So for example if your batch 10 files, instead of 5000 you can process 50000.

  • The other workaround is to start another Execution when you reached the 25000 limits, but solving it with the first 2 options is easier.

user_143028520
Related