I want to create a REST API that lets a caller submit input for a job which will run in the background for potentially a minute or so. Hence, the API should spawn a background job and immediately return a job ID. The client may then query another endpoint for the job status and result.
The first thought that pops up in my head is something like this:
- API "submit job lambda" creates a job record in the database and submits the input to a SQS queue
- "Worker lambda" polls SQS, performs the work and updates the job in the database
- API "query job lambda" returns job record from the database
For this to work, the worker lambda has to have a timeout sufficient to finish a large job. The API lambdas will always be quick since they just make a database operation and a submission to SQS.
Still it kind of bugs me that I need to have 2 lambdas. But if I understood it correctly, it's not possible to continue executing a lambda function after it has returned a response to the caller. I also considered perhaps using Kinesis rather than SQS for the events, as order of execution is not critical, every caller just wants his job finished in a reasonable amount of time. My trafffic pattern could be bursts of 100.000 jobs and then nothing for days. I'm also considering supporting multiple sub-jobs in each job, so that each job would actually be say 1-100 units of work.
I don't want to host a instances for the workers, e.g. containers on ECS; I want to rely only on serverless concepts.
Is this a decent setup or is there a better one?