How to implement Amazon SQS (fifo)-lambda with message processing EXACTLY ONE BY ONE

user_14981915 0
Viewed 1056

I'm having a use case where I have an Amazon SQS fifo queue with lambda function. I need to make sure that fifo triggers the lambda only when the previous lambda execution is completed (also the events come in order). As from aws docs, fifo supports exactly once processing but it does not mention anywhere that it would not push more event on lambda untill the first message is completely processed.

I need to make sure that the next message is processed only when the previous message is completely processed by the lambda function.

Is there are way to ensure that message 2 is only processed by lambda when message 1 is completely processed by lambda?

amazon-sqs aws-lambda amazon-web-services message-queue queue
2 Answers

fifo supports exactly once processing but it does not mention anywhere that it would not push more event on lambda untill the first message is completely processed.

SQS never pushes anything anywhere. You have to poll SQS for messages. When you configure Lambda integration with SQS Lambda is actually running a process behind the scenes to poll SQS for you.

AWS FIFO queues allow you to force messages to be processed in order by specifying a Message Group ID. When you specify the same Message Group ID for multiple messages, then the FIFO queue will only make one of those messages available at a time (in first-in-first-out) order. Only after the first message is removed from the queue is the second message made available, etc...

In addition to this, you should configure AWS Lambda SQS integration with a Batch Size of 1, so that it doesn't try to wait for multiple messages to be available before processing. And you could configure the Reserved Concurrency on the Lambda function to 1, as mentioned in the other answer, so that only one instance of the Lambda function can be running at a time.

user_130700

It is actually pretty easy to do this. It is not clarified, since it will by default simply use up the available account concurrency and handle as many messages in parallel as is possible.

You can influence this by setting the reserved concurrency for the lambda function to 1. This will ensure no more than 1 lambda function will be executed at the same time.

user_9074800
Related