How to retain AWS SQS messages when error occurs in aws lambda function

user_14040749 0
Viewed 1193

Recently working on a project where there is a requirement to download documents from SFTP and need a queue based implementation for the same.

So i have opted the implementation using the SQS and lambda. Whenever the documents comes in SFTP i will push document information to SQS via web api. In SQS Message body i have passed the file name and other information.

Also created a .net core lambda function trigger for the SQS queue (used FIFO queue).

public async Task FunctionHandler(SQSEvent evnt, ILambdaContext context)
{
 try
 {
     //trigger web api
     var response = GetDocumentFromSFTP(message.Body).Result;                     
 }
 catch (Exception ex)
  {
    LambdaLogger.Log("Error::" + ex.Message);
  }
}

if any error happens inside GetDocumentFromSFTP() where external web api is called and returns the flag true or false, or other places how can i retain message in SQS queue without deleting so that can poll the same message later?

Please help ?

Update: I have pushed some message to the queue and it got processed and couldn't see those messages even after visibility time over also.But the messages which are thrown error i am able to view in dead letter queue. Please refer the below image. Is this a known behavior?

SQS messages

amazon-sqs aws-lambda asp.net-core-webapi .net-core c#
3 Answers

AWS SQS keeps the message in the queue (though hidden) until it is explicitly deleted. This applies to AWS Lambda as well. The message will be kept hidden until the visibility timeout is reached.

Quoting what is found here.

When Lambda reads a batch, the messages stay in the queue but become hidden for the length of the queue's visibility timeout. If your function successfully processes the batch, Lambda deletes the messages from the queue. If your function is throttled, returns an error, or doesn't respond, the message becomes visible again. All messages in a failed batch return to the queue, so your function code must be able to process the same message multiple times without side effects.

You might want to also read about DLQ where the failed message go to. Here's a good article about it: https://dev.to/piczmar_0/error-handling-in-aws-lambda-trigger-by-sqs-events-3amp

user_64751560

Here I have provided the link to my answer to a similar question. If you go through the code, there is "amazonSqs.deleteMessage" method which gets executed only when there is no exception in your message processing. You can implement in a similar way or get little creative.

https://stackoverflow.com/a/66540960/6159869

user_61598690

When a message is sent to a LAMBDA via a SQSEvent, if the message is processed normally by the LAMBDA function, then SQS will automatically deletes it as soon as the LAMBDA completes. But what if there is an issue processing the message and you don't want SQS deleting it ? throw an exception that's all.

More details please go through the AWS documentation

user_21293200
Related