Scheduling a process in AWS

Question

I have a simple serverless architecture in AWS that processes incoming requests. An API Gateway receives a request and sends it to a lambda. The lambda saves the request to a Dynamodb and then moves the request to an SQS for further processing (other lambdas will listen to the SQS).

Some of the requests received by the API GW include a time stamp that indicates when it should be processed (can be up to 24 hours).

I’m looking for a safe and efficient way to schedule those requests to be processed.

What did I look into:

Cloudwatch/ event scheduler are not a good fit as I see as it will pass the schedule responsibility to the Lambda. Meaning, the lambda will have many specific schedules. It doesn’t sound a great fit

Delayed SQS Messages has limited delay time. It doesn’t fit my 24 hours requirement.

I could manage the schedule myself by running a query on DynamoDB every minute and check if there are requests ready to be processed. I don’t like this solution as it sounds very wasteful.

How would you suggest to schedule the incoming requests to be processed?

user_158668 · Answer

You can use AWS Step Functions to schedule a delayed execution using a Wait state. The maximum execution time of a Step Function workflow is 1 year. Please have a look at this example of a task timer for implementation details.

user_5483526 · Answer

For times < 15 minutes, you can use SQS message timers.

For times greater than 15 minutes, you can poll DynamoDB every 15 minutes, and then forward messages to SQS with a message timer when they're supposed to be sent before the next poll.

Polling every 15 minutes is only 3000 polls per month -- very cheap.

Scheduling a process in AWS

2 Answers