reading file from s3 (boto3) in multithread fashion

Viewed 915

I would like to process an huge folder of XML files locate in s3 buckets. I would like to do this in a distributed way in Python.

So I first define a paginator with boto3 library, setting up (for example) a page size of 100 elements (refer to doc)

profile = boto3.Session()
client = profile.client('s3')
paginator = client.get_paginator('list_objects')

page_iterator = paginator.paginate(
    Bucket='my-bucket',Prefix='my-prefix',
    PaginationConfig={'PageSize': 100}
)

after that with concurrent.futures package I create a pool of 10 threads each of which calling my_process_method:

def my_process_method(pages):
    for page in pages['Contents']:
        # ...process...

with concurrent.futures.ThreadPoolExecutor(10) as executor:
    executor.map(my_process_method, page_iterator)

I would like to know if there are some drawbacks in this example, eg. concurrent boto3 API calls can cause some sort of problems

0 Answers
Related