Need to run a aws lambda function which takes more than 15 minutes to complete?

There is no way around this. If you're doing some sort of long running processing then your other option may be to run this task on an EC2 instance. If this long running process can be broken down in to multiple steps then you could look in to Lambda Step Functions.

15 Minutes is the max and this max can not be extended.

EDIT:

Recently I started running some long running tasks that are variable in length (anywhere from a couple minutes to several hours). To accomplish this I've been using AWS Fargate and my task is node.js script that is stored as a Docker container in ECR. Doing this was fairly easy and also is fairly cheap (I think we spent a little over $1 for running this task daily in a month). This may be something worth looking in to for others who may come across this answer.

https://docs.aws.amazon.com/AmazonECS/latest/userguide/scheduled_tasks.html

Typically use a Fat Lambda strategy or Step Function

Fat Lambda Strategy

A Fat Lambda strategy is used when your task is singular but has a long-running execution time and/or you have heavy hardware requirements. The idea is that you would create a script that executes your long process and put it into a docker container that's hosted in Fargate. Meaning no limits to execution time and access to powerful hardware (How to create a Fat Lambda https://youtu.be/XUp9SHIHU8w)

Step Function strategy

A Step Function strategy is used to break down your entire process into smaller steps. Usually, a step function strategy would work for you if your process could have lots of miniature stages linked together instead of a big colossal job attempting to do everything simultaneously. Bear in mind that a "Fat Lambda" can also be triggered by a Step Function (How to create a Step Function https://www.youtube.com/watch?v=s0XFX3WHg0w)

Also, another note, remember lambdas can also trigger other lambdas. So you might even be able to have different lambdas run bits of your lambda code. For example, a FOR loop sends off a lot of mini lambdas to run small tasks. You might not even need a Step Function or a Fat Lambda.

If you're stuck on what to choose, follow the below. It will help you reason with your problem.

Singular Lambda >> Lambda invoke another Lambda? >> Step Functions? >> Fargate (Fat Lambda)?

If you can checkpoint the task then you can check the getRemainingTimeInMillis (docs) and if the time is running out then invoke the same lambda with a parameter where to continue.

Something like this flow:

• start working (0% done)
• time is running low (40% done) => start a new lambda telling it to start from 40%
• old lambda is terminated, new lambda starts working (40%)
• when its time is running low, start a new lambda again (80%)
• the third lambda finishes the job

But it requires a very specific type of task to support this. If your require a single execution from start to finish then lambda is not a good choice for this.

What do you think about using a lambda to trigger an ECS task? An ECS task just runs a containerized application for as long as it needs to run.

This blog post is relevant: https://www.gravitywell.co.uk/insights/using-ecs-tasks-on-aws-fargate-to-replace-lambda-functions/.

Aws lambda is meant to be used for quick processing. if your task is this long then better choose some other way to develop that functionality. Although you can define the timeout property for AWS lambda, but that can not exceed 15 minutes.

As per you use case better to use EC2 for deploying you application and then terminate the EC2 instance when the processing is done or it remains idle more than the threshold time.

Refer the AW Lambda documentation - https://docs.aws.amazon.com/lambda/index.html

To add to the Step function answer - here's a very simple playbook:

1. Work for 10 minutes
2. Write progress to S3
3. kick off another lambda to consume your progress
4. terminate

Once you're done, output. Viola, infinite runtime lambda with very little effective overhead.