When using SQS w/ Dead letter and Lambda , why do messages remain "in flight" for 5 min after lambda fails w/ runtime exception?

user_3256123 0
Viewed 1641

When using SQS w/ Dead letter and Lambda , why do messages remain "in flight" for 5 min after lambda fails w/ runtime exception?

I've created 3 resources

MyQueue (configured to send undeliverable messages to MyQueueDLQ.Default visibility timeout:30 sec )

MyQueueDLQ

Lambda (retry attempts set to 0, timeout 30 seconds)

I for some reason expect (perhaps because of lack of understanding) that upon my lambda failing, the dead letter queue to receive the message shortly after failure. (instead of minutes after).

How exactly can I ensure the dead letter queue gets the message in the fastest way possible so that anything responding to the dead letter queue messages doesn't wait minutes unnecessarily?

Note: I am intentionally throwing a runtime exception in the lambda to test this so that I understand how this all works.

My goal would be to ensure that messages go to the dead letter queue as fast as possible. Is 5 min the best I can do?

Update 1: Ive set the timeout on the lambda to 5 seconds, and the timeout on the queue to 25 seconds, now it takes about 1min and 40 seconds for the message to arrive on the DLQ. Which still doesnt match my expectations. SHouldnt the message arrive on the DLQ in 25 seconds?

Update 2: So today I discovered a little info icon on the AWS Explorer SQS queue on the bottom window. This may very well describe what I am seeing enter image description here

amazon-sqs aws-lambda amazon-web-services
1 Answers

When your lambda receives messages from MyQueue, the messages will go into invisibility mode, where it is not visible by others who also read the same queue.

Normally, when your function successful processes the msg, the lambda service will automatically remove the message from the queue. However, if this does not happen, the message remains invisible for the remainder of the invisibility time. Then when it becomes visible again, lambda service may again re-try the processing of the same message. If the re-tries have been exhausted, the message will go to the DLQ.

More about this is here:

If a message fails to be processed multiple times, Amazon SQS can send it to a dead-letter queue. When your function returns an error, Lambda leaves it in the queue. After the visibility timeout occurs, Lambda receives the message again. To send messages to a second queue after a number of receives, configure a dead-letter queue on your source queue.

user_2488230
Related