Is AWS Lambda the proper way of running a batch job?

user_9647022 0
Viewed 26947

I have a batch job that I need to run on AWS. I'm wondering what's the best service to use. The job needs to run once a day, so I think that naturally AWS Lambda with a CloudWatch Rule triggering it would do it. However, I'm starting to think that AWS Lambda is thought to be used as a service to handle requests. This AWS official library to integrate Spring-Boot is very oriented to handle HTTP requests, and when creating a lambda via AWS Console, only test cases that send an input to the lambda can be written.

Then, is this a use case for AWS Lambda? Also, these functions can run up to 15 minutes. What should I use if my job needs to run longer?

batch-processing aws-lambda amazon-web-services
2 Answers

The purpose of Lambda, as compared to AWS EC2, is to simplify building smaller, on-demand applications that are responsive to events and new information.

If your batch is running within a limit of 15 minutes then you can go with a lambda function.

But if you want batch processing to be done, you should check AWS batch.

Here is nice article which demonstrates the usage of AWS batch.

If you are already using some batch framework like spring-batch, you can also take a look at ECS scheduled task with Fargate.

With ECS Fargate you can launch and stop container services that you need to run only at certain times. Here are some related articles on Fargate event and scheduled task and Scheduled Tasks.

user_30862890

If you're confident that your function will only run at maximum of 15mins, AWS Lambda could be the solution. Here are the AWS Lambda limits that could help you decide on that.

Also note that lambda has cold start, it's when it will run slower at first but will eventually pick up the pace. Here are some good reads about it that could help you decide on the lambda direction, but feel free to check on any articles that could better explain at your disposal.

  • This one shows a brief lists that you would like to consider and the factors affecting it.
  • This one might have a deeper explanation of the cold start with regards to how it works internally.

What should I use if my job needs to run longer?

Depending on your infrastructure, you could maybe explore Scheduled Tasks

user_52593570
Related