AWS Glue Boto3 - Where can I find the list of the waiters?

user_1244173 0
Viewed 3204

I'm trying to kick off an AWS Glue ETL job in my Python script and check the status of it until the job finishes.

Initially I just did a simple while loop, which waits for 1 minute and checks the job status. Once the job finishes successfully, break the loop. While researching more about this approach, I found out about this thing called 'waiter'.

But the thing is that even though the 'get_waiter' method is available for the Glue client, I do see any waiters.

So when I run this...

s3 = boto3.client('s3')
glue = boto3.client('glue')

print('s3 waiters')
print(s3.waiter_names)
print('glue waiters')
print(glue.waiter_names)

I get this...

s3 waiters
[u'bucket_exists', u'bucket_not_exists', u'object_exists', u'object_not_exists']
glue waiters
[]

So I'm wondering if there are actually waiters for AWS Glue. If so, were can I find them?

Some references I found: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_waiter

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/clients.html#waiters

aws-glue amazon-web-services python
1 Answers

If you want to know more about a service's waiters there are three main ways to find out more details:

  1. Check the boto3 documentation. If a service has a waiter it will be listed there and how to use it.
  2. Print the waiter names in the code just as you mentioned in your question.
  3. Go to the botocore data folder in github and check your specific services for information about the waiters available.

Edit: for those that need a custom waiter for glue jobs

import boto3
from botocore.waiter import WaiterModel
from botocore.waiter import create_waiter_with_client

client = boto3.client('glue')
waiter_name = "JobRunCompleted"
waiter_config = {
    "version": 2,
    "waiters": {
        "JobRunCompleted": {
            "operation": "GetJobRun",
            "delay": 60, # Number of seconds to delay
            "maxAttempts": 30, # Max attempts before failure
            "acceptors": [
                {
                    "matcher": "path",
                    "expected": "SUCCEEDED",
                    "argument": "JobRun.JobRunState",
                    "state": "success"
                },
                {
                    "matcher": "path",
                    "expected": "FAILED",
                    "argument": "JobRun.JobRunState",
                    "state": "failure"
                }
            ]
        }
    }
}
waiter_model = WaiterModel(waiter_config)
glue_job_run_waiter = create_waiter_with_client(waiter_name, waiter_model, client)

glue_job_run_waiter.wait(JobName='my_job_name', RunId='my_run_id')
user_153228870
Related