Scheduled AWS Lambda Task at less than 1 minute frequency

user_2314001 0
Viewed 15910

We are trying to develop a true lambda-based application in which certain tasks need to be performed at schedules of variable frequencies. They are actually polling for data, and at certain times of the day, this polling can be as slow as once every hour, while at other times, it has to be once every second. I have looked at the options for scheduling (e.g. Using AWS Lambda with Scheduled Events and AWS re:Invent 2015 | (CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda), but it seems that short of spinning up an EC2 instance or a long-running lambda, there's no built-in way of firing up lambdas at a frequency of less than one minute. The lambda rate expression doesn't have a place for seconds. Is there a way to do this without an EC2 instance or long-running lambda? Ideally, something that can be done without incurring additional cost for scheduling.

aws-lambda amazon-web-services amazon-ec2 scheduled-tasks
5 Answers

You theoretically can wire up a high-frequency task-trigger without an EC2 instance or long-running lambda using an AWS Step Function executed by a CloudWatch Events scheduled event (see Emanuele Menga's blog post for an example), but that approach would actually be more expensive than the other two options, according to my current (as of May 2019) estimates:

(assuming 1 year = 31536000 seconds)

Step Function:

  • $0.0250 per 1000 state transitions
  • 2 state transitions per tick (Wait, Task) + at least 1 per minute (for setup/configuration)
  • 31536000 * (2 + 1/60) * 0.0250 / 1000 = $1589.94/year, or as low as $65.70/year for lower frequency trigger (2 ticks per minute)

Lambda Function:

  • $0.000000208 per 100ms (for smallest 128mb function)
  • 31536000 * 0.000000208 * 10 = $65.595488/year

EC2 Instance:

  • t3a.nano is $0.0047 per hour on-demand, or as low as $0.0014 using Spot instances
  • 31536000 * 0.0047 / 3600 = $41.172/year, or $12.264/year using Spot instances

So there will be some scheduling cost, but as you can see the cost is not that much.

user_25183550

There is a simple hack though where you could use a setTimeout or setInterval.

i.e:

    'use strict';
    async function Task() { 
      console.log('Ran at'+new Date())
    }
    const TIMEOUT = 30000;
    exports.handler = (event, context) => {
       return Promise.all([
          Task(), 
          new Promise(function(resolve, reject) {
             let timeout = setTimeout(function() {
               Task().then(() => {
                 clearTimeout(timeout)
               })
             }, TIMEOUT);
          })
          
    ])
        
    }
user_11694680
Related