Read files with only specific names from Amazon S3

user_7166834 0
Viewed 1210

I have connected to Amazon S3 and am trying to retrieve data from the JSON content from multiple buckets using the below code.

But I have to read only specific JSON files, but not all. How do I do it?

Code:

for i in bucket:
    try:
          result = client.list_objects(Bucket=i,Prefix = 'PROCESSED_BY/FILE_JSON', Delimiter='/')
          content_object = s3.Object(i, "PROCESSED_BY/FILE_JSON/?Account.json")
          file_content = content_object.get()['Body'].read().decode('utf-8')
          json_content = json.loads(file_content)
    except KeyError:
          pass

Bucket structure example.

test-eob/PROCESSED_BY/FILE_JSON/222-Account.json
test-eob/PROCESSED_BY/FILE_JSON/1212121-Account.json
test-eob/PROCESSED_BY/FILE_JSON/122-multi.json
test-eob/PROCESSED_BY/FILE_JSON/qwqwq-Account.json
test-eob/PROCESSED_BY/FILE_JSON/wqwqw-multi.json

From the above list, I want to only read *-Account.json files.

How can I achieve this?

amazon-web-services amazon-s3 python
2 Answers

There are several ways to do this in Python. For example, checking if 'stringA' is in 'stringB':

list1=['test-eob/PROCESSED_BY/FILE_JSON/222-Account.json',
'test-eob/PROCESSED_BY/FILE_JSON/1212121-Account.json',
'test-eob/PROCESSED_BY/FILE_JSON/122-multi.json',
'test-eob/PROCESSED_BY/FILE_JSON/qwqwq-Account.json',
'test-eob/PROCESSED_BY/FILE_JSON/wqwqw-multi.json',]

for i in list1:
    if 'Account' in i:
        print (i)
    else:
        pass
user_14696630

You can make use of a regex that matches your pattern from the list of objects.

import re

MATCH = "FILE_JSON/.*?Account.json"

full_list = [
  "test-eob/PROCESSED_BY/FILE_JSON/222-Account.json",
  "test-eob/PROCESSED_BY/FILE_JSON/1212121-Account.json",
  "test-eob/PROCESSED_BY/FILE_JSON/122-multi.json",
  "test-eob/PROCESSED_BY/FILE_JSON/qwqwq-Account.json",
  "test-eob/PROCESSED_BY/FILE_JSON/wqwqw-multi.json"
]

for item in full_list:
  if re.search(MATCH, item):
    print(item)
user_59032760
Related