Clean Boto3 Pagination

user_1390558 0
Viewed 1849

I am trying to find a very nice python idiom to use aws boto3 paginators in the most "pythonic" way. Below is the best I have been able to come up with and I'm still not happy with it. Any ideas on how to make pagination simpler, possibly not using while True:?

import boto3

client = boto3.client('acm', region_name='ap-southeast-2')

paginator = client.get_paginator('list_certificates')
response_iterator = paginator.paginate()

while True:
    for certificates in response_iterator:
        for certificate in certificates['CertificateSummaryList']:
            print(certificate)

    if response_iterator.resume_token:
        response_iterator = paginator.paginate(
            PaginationConfig={
                'StartingToken': response_iterator.resume_token
            })
    else:
        break
boto3 amazon-web-services pagination python
2 Answers

Woudn't the following form work?:

client = boto3.client('acm', region_name='ap-southeast-2')

paginator = client.get_paginator('list_certificates')

for page in paginator.paginate():
    print(page)
user_2488230

It’s not really documented but you can do something like this with paginators

client = boto3.client('acm')
results = (
    client.get_paginator('list_certificates')
    .paginate()
    .build_full_result()
)
print(results)

edit

a similar approach for ec2 which has one of the more annoying response structures:


import jmespath

client = boto3.client('ec2')
results = (
   client.get_paginator('describe_instances')
   .paginate()
   .build_full_result()
)

# jmespath could be replaced with some nested for loops if you prefer, but I find this "cleaner" and easier to read.

instances = jmespath.search('Reservations[].Instances[]')

for i in instances:
  print(i['InstanceId'])

You can generate an example output response which is super handy for most, if not all, the services which has probably saved me days in trying to figure.

Unfortunately, I think there's a bug in the aws-cli skeleton output for acm list-certificates

but the equivalent for ec2.describe_instances just for fun.

$ aws ec2 describe-instances --generate-cli-skeleton output
{
    "Reservations": [
        {
            "Groups": [
                {
                    "GroupName": "GroupName",
                    "GroupId": "GroupId"
                }
            ],
            "Instances": [
                {
                    "AmiLaunchIndex": 0,
                    "ImageId": "ImageId",
                    "InstanceId": "InstanceId",
                    "InstanceType": "InstanceType",
                    "KernelId": "KernelId",
                    "KeyName": "KeyName",
                    "LaunchTime": "1970-01-01T00:00:00",
                    "Monitoring": {
                        "State": "State"
                    },
  ...
user_45260190
Related