Triggering an AWS Lambda Function based on multiple (1k-10k) schedules

Question

We have an AWS Lambda function which queries some data for a client from our DB and sends a report to the client. Some clients want daily reports, some might need weekly or monthly reports. The number of clients can go up ~1000 and each client might have ~10 such reports.

So we are looking for a way to trigger the Lambda function with different parameters based on schedules set by each client.

For Example:

Client A wants daily report of their data to be sent to abc@clienta.com and Client B wants a weekly report of their data to be sent to xyz@clientb.com. So the Lambda function will be invoked twice on Sunday 12 AM (for both clients) and once on Monday-Saturday 12 AM (for Client A).

We found the following solutions on AWS, but both have some limitations.

Approach 1: Use CloudWatch Events

We can create a CloudWatch Events Rule for each client and each report that could trigger our Lambda function on each schedule.

Pros: Simple setup, easy to implement.

Cons:

There is a limitation of 100 Event Rules per AWS Account. It's mentioned that we can contact AWS to get it increased, but we are not sure if it can be increased to the number we are looking for (Currently it is ~10k, but we would prefer a solution in which there is no such limit). Also, a limit of 100 per account gives an indication that this is not a suitable solution for such a use case.

Approach 2: Using Step Functions

For each client and each report, we can create one AWS State Machine. We can use the Iterator pattern in Step Functions to wait for a day/week/month and then re-invoke the Lambda Function.

Pros: No limitations on number of State Machines, so this enables us to scale easily.

Cons:

Step Functions have a limitation that they can run for a year, at maximum. This will be a problem in our case because the users will need to get the reports for a much longer period. There is a way to overcome this in Step Functions. Just before it's about to reach the 1-year limit, we can cancel the execution and start a fresh execution. So overall, this solution looks complex.

Can someone suggest a better solution for this on AWS?

user_1806763 · Answer

Do you really need a CloudWatch for each client? Why not do something like the following architecture.

Have cloudwatch kick off a lambda that checks schedules for all clients each day (or whatever the most frequent report schedule you allow). You don't want this to take a long time so you just have this check a database (i.e. DynamoDB) of schedules and drop metadata about any reports that need to be generated onto an SQS queue (i.e. type of report, client information, destination email). Worst case, this execute and finds nothing to schedule but this should only takes seconds so the cost is very low to just run this everyday.

Then you have a lambda that actually does the report generator and email that consumes the queue. This report generator lambda will scale and spin up as many instances it needs to handle the messages on the queue. You can set the concurrency limit for the report generator lambda to ensure it doesn't spin up too many at a time if that is a concern once you are having 1000s of clients.

The definition and deployment of all these components can easily be automated via an AWS SAM.

Hope this alternate approach gives you a few more ideas.

user_15937460 · Answer

You can combine both approach, to get the best result.

step 1: Use stepfunction to run your lambdas.

step 2: Trigger your stepfunction from cloudwatch, based on stepfunction event(SUCCESS,FAILED ETC).

In this way when step 1 fails or completes 1 year run. Cloudwatch event can trigger it back on, based on the json input you pass.

Triggering an AWS Lambda Function based on multiple (1k-10k) schedules

2 Answers