I have daily scheduled task that triggers around 10k lambda functions for 10k records that I need to maintain. I'm using SQS to queue all those messages and I want to spread execution over couple of hours. So I set up reserved concurrency to only 3 concurrent invocations. But still when that scheduled task hits concurrent invocations of that lambda functions goes over 3. Any advice on how to do it?
When I check lambda configuration it shows that reserved concurrency is 3. But on monitoring concurrent invocations shows way over 3.
Reserved concurrency on aws lambda does not prevent lambda to scale more?
Viewed 422
1 Answers
It's always tricky to use SQS with Lambda (concurrency limit configured) because in short, is not gonna work, instead, you will get some throttling records because the lambda can't process messages limited by the concurrency.
You can check this article which explains the why and a workaround solution : https://zaccharles.medium.com/lambda-concurrency-limits-and-sqs-triggers-dont-mix-well-sometimes-eb23d90122e0
check also this AWS documentation for further information about this subject https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#events-sqs-queueconfig
Related