Lambda with SQSEvent & large batch size invokes multiple instances each handling few items

user_5580830 0
Viewed 1173

Bit of background, I'm using Serverless and .Net to create a lambda with a SQS trigger. The event trigger is set with batch size of 10k and wait time (Batch Window ie MaximumBatchingWindowInSeconds) of 30 seconds. Queue's visibility timeout is set to almost 16 minutes.

Now that I've set the lambda to reserved concurrency of only 1 and ran a test where I send 100 items to the queue & was hoping to see only one lambda invocation with exactly those 100 items.

Problem was that it separated the items in the queue and invoked the lambda five times instead, causing five packages to be created as part of the lambda's functionality instead of the one package I wanted. (FYI the lambda's output creates packages in s3 of the messages. I want to have fewer packages that are large.)

Now the question: Is this the expected behavior? and if so why is it so when I've set the queue to accumulate up to 10k items and instead it settled for 15.

According to the aws docs the lambda can grab fewer messages than the batchSize if the payload is larger than 256kb but my messages are very small and 100 messages are no where near 256kb. So that can't be the cause.

Suggestions for alternatives to dealing with this issue are also welcome, right now I'm thinking of running an event bridge scheduler that calls lambda with SQS ReceiveMessage api and creates a single package but then I also have to make sure to properly delete the queue afterwards.

I'm a bit clueless here, I'd appreciate any ideas you guys have. Thanks.

amazon-sqs aws-lambda amazon-web-services concurrency
3 Answers

Problem was that it separated the items in the queue and invoked the lambda five times instead, causing five packages to be created as part of the lambda's functionality instead of the one package I wanted.

I think that this is probably because there are five SQS pooling threads that lambda uses to pool SQS. From AWS blog:

Lambda service will begin polling the SQS queue using five parallel long-polling connections.

So even though you had reserved concurrency of 1, lambda still uses the 5 threads (you can't control that), and your SQS messages were distributed into these threads. Then, each thread invoked your lambda function, one by one, resulting in observed 5 invocations.

user_2488230

I think a FIFO queue could be your solution:

"FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated."

[https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html][1]

user_39616230

You might try setting the MaximumBatchingWindowInSeconds value on your event source mapping. That allows you to collect up to 5 minutes of messages before invoking the lambda.

Keep in mind that SQS is a queuing mechanism, not a batching mechanism. The batching that is part of the event source mapping, or that is generally available with SQS, is intended to reduce the number of calls. It is not designed to provide guaranteed batch sizes. While setting the above value may achieve your current goal I don't know that you can reasonably expect it to always work that way. Also keep in mind that SQS is an "at least once" delivery mechanism (unless you use FIFO, which has a batch limit of 10).

user_28600320
Related