Is there an idiomatic way to aggregate events before processing them in AWS Lambda?

user_320537 0
Viewed 1131

I have an AWS Lambda function which processes events from S3. I'd like to aggregate them before processing and let lambda process the batch.

This is depicted below: enter image description here

Ideally, I'd like to be able to specify a batch size, and a timeout (say a single even, and then nothing for 5 sec, I'd like to send an 1-event batch).

Is there an idiomatic way to do it using Lambda or other AWS services?

aws-lambda amazon-web-services
2 Answers

There are a few things you can do:

1. Make upstream do the aggregation:

Make publishing the publisher's responsibility, and get the publisher to give you one event per group of objects to process. This works well if the publisher is already working in batches.

2. Insert your own aggregation step:

  1. Trigger on each event.
  2. Store the event somewhere.
  3. If enough events have been stored, empty the store and pass all the contents to the processing step.

This works well if your processing step is much more expensive per event than just handling the event. Often, this can take the form of {aggregating lambda} -> {processing batch job}, since Lambda isn't great for very expensive processing.

3. Do aggregation on a time basis:

  1. Send your events to an SQS queue.
  2. Trigger on a timer (e.g. Cloudwatch events).
  3. When triggered, empty the queue and process everything in it. If it's too much to process in a single invocation, immediately trigger an additional lambda.

This works well if processing is fairly cheap, and you want to minimize your number of Lambda invocations. The trigger schedule (how long you wait in between invocations) is determined by weighing how long you're willing to wait to process an event against how many invocations you're willing to pay for. Things to watch out for: 1. if you get no events at all, you will still be invoking your Lambda, and 2. if you get events faster than they can be processed, your queue will grow more and more and your processing will fall further and further behind.

user_78595150

I think you can achieve the batch operation by setting SQS queue as destination for S3 notification. Let's say you want to specify a batch size of 20, all your S3 events are going to SQS. You would create a CloudWatch rule to trigger a Lambda when your SQS have 20 items. Your Lambda would poll SQS for the batch of 20 items and process them.

You can also set SQS triggers but it has a limit of max batch size 10.

user_14059420
Related