Execute only one Glue job at a time / sequential Glue job execution

Sapnesh Naik 0
Viewed 29

Currently, we have the following AWS setup for executing Glue jobs. An S3 event triggers a lambda function execution whose python logic triggers 10 AWS Glue jobs.

S3 -> Trigger -> Lambda -> 1 or more Glue Jobs.

With this setup, we see that at a time, multiple different Glue jobs run in parallel. How can I make it so that at any point in time, only one Glue job runs? And any Glue jobs sent for execution wait in a queue until the currently running Glue job is finished?

aws-step-functions aws-glue aws-lambda amazon-web-services
2 Answers

You can use step function and in each steps specify job you want to run so you will have control to run jobs and once step one complete then call step 2 jobs etc

vaquar khan0

If you are looking for having some job queues to have the Glue jobs trigger in sequence, you may consider using a combination of SQS->lambda->Glue jobs? Please refer this SO for details

AWS Step function is also another option as suggested by Vaquar Khan

Yuva0
Related