read only particular json files from s3 buckets from multiple folders

user_7166834 0
Viewed 811

am trying to scroll over all buckets in s3 and see if there is a prefix that matches and get into those folders and read the json files.

I have tried to get the folders that contain a prefix, but failing to enter them.

Code: 

import boto3
bucket = ['test-eob', 'test-eob-images']
client = boto3.client('s3')
for i in bucket:
    result = client.list_objects(Bucket=i,Prefix = 'PROCESSED_BY/FILE_JSON', Delimiter='/')
    print(result)

Using this am getting the ones with prefix and fails when bucket doesnt have that prefix.

structure of test-eob , test-eob/PROCESSED_BY/FILE_JSON/*.json I have to read the json if only my prefix matches, else come out of the bucket.

Can anyone help me out here.

amazon-web-services amazon-s3 python
1 Answers

Try to catch the error(is it a KeyError?) when the bucket does not contain the prefix.

For example:

for i in bucket:
    try:
          result = client.list_objects(Bucket=i,Prefix = 'PROCESSED_BY/FILE_JSON', Delimiter='/')
          print(result)
    except KeyError:
          pass

To read the json, there are several ways. For example with json.loads() from the json module.

So for each object in the bucket:

content_object = s3.Object(bucket_name, file_name)
file_content = content_object.get()['Body'].read().decode('utf-8')
json_content = json.loads(file_content)
user_14696630
Related