Configuring AWS SQS/Lambda trigger to honor both 1:1 policy as well as max concurrent instances

user_5235665 0
Viewed 2461

Java 8 here using the AWS Java SDK to write a Java lambda that should execute in response to a message being sent to an SQS queue.

Ideally, one-and-only-one lambda instance will be invoked/executed for each record sent to the SQS queue. So if 5 messages are sent to the queue, 5 lambdas will fire (or - depending on my lambda configuration - I may set a max # of concurrent lambdas in which case my expectation is that pending/unconsumed SQS messages would wait for the next available lambda).

This is not a hard requirement, just ideal.

I noticed that in the com.amazonaws.services.lambda.runtime.events.SqsEvent class, there is a getRecords() : List<SQSMessage> method which has me a bit concerned. To me, this implies that a single lambda instance may be fed more than 1 SQS message per execution, which again, goes against my desired behavior.

So I'm wondering if there is a way to configure the Lambda trigger such that it only ever fires once per SQS queue message, and also honors a "max # of concurrent Lambda instances" setting, making messages wait in SQS until a Lambda is ready. So as another example, say I have the max # of concurrent Lambdas set to three (3), and 5 messages get sent onto the queue at the same time. In this case, I would want 3 Lambdas to fire, each processing one of the 5 queued messages, and 2 of the 5 messages would be waiting for one of those 3 Lambdas to finish so another one could fire and pick them up.

Is this possible to do? Or does Lambda just "decide" (?) somehow on its own how many messages to submit to a given Lambda execution? If so, anybody know how this is decided?

amazon-sqs aws-lambda amazon-web-services java
2 Answers

TL;DR

As @joseph already pointed out correctly, you could use an event source mapping with BatchSize set to 1. That will make getRecords() return maximally 1 SQSMessage. In order to process maximally 1 message at a time, you have to set the Lambda function's reserved concurrency to 1. However, as also stated correctly, this is not optimal for a standard SQS queue. The event source mapping will run into some TooManyRequestsException: Rate Exceeded errors which are logged into CloudWatch Logs.

To use a proper sequential-one-message-at-a-time processing pattern without relying on Lambda function throttling, use an SQS FIFO queue as described in the AWS blog post [1]. It says: "Total concurrency is equal to or less than the number of unique MessageGroupIds in the SQS FIFO queue". That is, you could configure exactly one MessageGroupId for you SQS FIFO queue in order to:

  • only ever fire Lambda once per SQS queue message (because batchSize = 1)
  • while also honoring a "max # of concurrent Lambda instances" of exactly 1 (because concurrency count = #unique message group ids = 1)

The number of unique message group ids thus is the max. number of concurrent Lambda invocations by an event source mapping for an SQS FIFO queue.

Some more Information

Java Libraries for Lambda

As far as I can see, AWS provided a set of POJOs (e.g. SQSEvent in library aws-lambda-java-events) [2] in order to process the incoming SQS event [3]. The SQS event is delivered by the Lambda event source mapping and deserialized into the given POJO. The docs for the POJO SQSEvent are also available at JavaDoc.io [4] and the source code is available at GitHub [5]. The method getRecords() returns a list of SQSMessage objects because an AWS Lambda event source mapping can indeed provide between 1 and 10 SQS messages.

Lambda Event Source Mapping

The event source mapping is created and configured with attributes which are specific to the source type. As we are looking at the SQS integration, we must take into account SQS-specific attributes only. These are mainly: BatchSize and EventSourceArn. For a full list, see [6]. If an attribute is not applicable to the SQS source type, its description starts with the keyword (Streams).

You must set the BatchSize if you want to limit the number of SQS messages which are retrieved using getRecords(). The default value is 10.

Lambda Scaling

As described in the docs [7], a Lambda concurrency limit could be used to limit the number of SQS message batches which are processed concurrently by a Lambda function. However, this does not prevent the event source mapping from calling the Lambda function. At least, I could not find any official source which states the opposite - please correct me if I am wrong.

That is, a lot of throttling errors are thrown (code 429) if the SQS queue is heavily used. It is possible to overcome this issue by instructing the event source to process messages sequentially. This is achieved by using an Amazon SQS FIFO event source. It is a fairly new feature. [8]

Summary

All in all, I would recommend to:

  • use an SQS queue with the FIFO type instead of the standard type
  • use an event source mapping with BatchSize set to 1
  • use the same value for the MessageGroupId attribute across all SQS SendMessage API calls [9]
  • be familiar with the differences between SQS FIFO queues and standard queues [10][11] - including pricing differences [12]
  • do not necessarily set a reserved concurrency as it is handled by the event source mapping for FIFO queues

References

[1] https://aws.amazon.com/blogs/compute/new-for-aws-lambda-sqs-fifo-as-an-event-source/
[2] https://docs.aws.amazon.com/lambda/latest/dg/with-sqs-create-package.html#with-sqs-example-deployment-pkg-java
[3] https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
[4] https://javadoc.io/static/com.amazonaws/aws-lambda-java-events/2.2.2/com/amazonaws/services/lambda/runtime/events/SQSEvent.html
[5] https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-events/src/main/java/com/amazonaws/services/lambda/runtime/events/SQSEvent.java
[6] https://docs.aws.amazon.com/lambda/latest/dg/API_CreateEventSourceMapping.html#API_CreateEventSourceMapping_RequestBody
[7] https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html
[8] https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-amazon-sqs-fifo-event-source/?nc1=h_ls
[9] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html
[10] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html
[11] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html#FIFO-queues-moving
[12] https://aws.amazon.com/sqs/pricing/?nc1=h_ls

user_104734690

Getrecords is the function to get records from 1 to max possible for the source. The batch size is controlled by the lambda event source mapping. If you set that to 1, your lambda will always receive a records array with just one element.

The number of lambdas processing the message depends on the concurrency limit you set for the lambda. Just remember if the number of concurrent lambdas which you have allowed is less than the number of sqs messages you have at any time, you may see a lot of throttling exceptions in your cloudwatch metrics. You can ignore them if thats the desired behaviour.

Also you can increase the visibility timeout of your sqs configuration to make sure the same message is not delivered to another lambda while its already being process by one.

user_29409660
Related