How to prevent AWS Lambda from deleting message from SQS queue automatically and instead delete it programmatically?

user_10977777 0
Viewed 1585

When a file is added to my S3 bucket an S3PUT Event is triggered which puts a message into SQS. I've configured a Lambda to be triggered as soon as a message is available.

In the lambda function, I'm sending an API request to run a task on an ECS Fargate container with environment variables containing the message received from SQS. In the container I'm using the message to download the file from S3, do processing and on successful processing I wish to delete the message from SQS.

However the message gets deleted from SQS automatically after my lambda executes. Is there any way that I can configure the lambda not to automatically delete the SQS message (other than raising an exception and failing the lambda purposely), so that I can programmatically delete the message from my container?

Update: Consider this scenario which I wish to achieve.

  1. Message enters SQS queue
  2. Lambda takes the message & runs ECS API and finishes without deleting the msg from queue.
  3. Msg is in-flight.
  4. ECS container runs the task and deletes msg from queue on successful processing. If container fails, after the visibility timeout the message will re-enter the queue and the lambda will be triggered again and the cycle will repeat from step 1.
  5. If container fails more than a certain number of times, only then will message go from in-flight to DLQ.

This all currently works only if I purposely raise an exception on the lambda and I'm looking for a similar solution without doing this.

amazon-sqs aws-fargate aws-lambda amazon-web-services amazon-s3
2 Answers

The behaviour is intended and as long as SQS is configured as a Lambda trigger, once the function returns (i.e. completes execution) the message is automatically deleted.

The way I see it, to achieve the behaviour you're describing you have 4 options:

  • Remove SQS as Lambda trigger and instead execute the Lambda Function on a schedule and poll the queue yourself. The lambda will read messages that are available but unless you delete them explicitly they will become available again once their visibility timeout is expired. You can achieve this with a CloudWatch schedule.
  • Remove SQS as Lambda trigger and instead execute the Lambda Function explicitly. Similar to the above but instead of executing on a schedule all the time, the Lambda function could be triggered by the producer of the message itself.
  • Keep the SQS Lambda trigger and store the message in an alternative SQS Queue (as suggested by @jarmod in a comment above).
  • Configure the producer of the message to publish a message to an SNS Topic and subscribe 2 SQS Queue to this topic. One of the two queues will trigger a Lambda Function, the other one will be used by your ECS tasks.

Update

Based on the new info provided, you have another option:

Leave the event flow as it is and let the message in the SQS be deleted by Lambda. Then, in your ECS Task, handle the failure state and put a new message in the SQS with the same payload/body. This will allow you to retry indefinitely.

There's no reason why the SQS message has to be the exact same, what you're interested is the body/payload.

You might want to consider adding a mechanism to set a limit to these retries and post a message to a DLQ.

user_23044210

One solution I can think of is: remove lambda triggered by the sqs queue, create an alarm that on sqs queue. When the alarm triggers, scale out the ecs task. When there's no item in the queue, scale down the ecs task. Let the ecs task just poll the queue and handle all the messages.

user_92288140
Related