I would like to process an huge folder of XML files locate in s3 buckets. I would like to do this in a distributed way in Python.
So I first define a paginator with boto3 library, setting up (for example) a page size of 100 elements (refer to doc)
profile = boto3.Session()
client = profile.client('s3')
paginator = client.get_paginator('list_objects')
page_iterator = paginator.paginate(
Bucket='my-bucket',Prefix='my-prefix',
PaginationConfig={'PageSize': 100}
)
after that with concurrent.futures package I create a pool of 10 threads each of which calling my_process_method:
def my_process_method(pages):
for page in pages['Contents']:
# ...process...
with concurrent.futures.ThreadPoolExecutor(10) as executor:
executor.map(my_process_method, page_iterator)
I would like to know if there are some drawbacks in this example, eg. concurrent boto3 API calls can cause some sort of problems