get s3 files with prefix using python

user_9516260 0
Viewed 7121

My s3 filename is 'folder/filename.xml'. i want to take the files end with 'name.xml'

import boto3
s3 = boto3.resource('s3')
try:
fileobj = s3.Object('lcu-matillion',''folder/.*name.xml'').get()['Body']

data=fileobj.read()
except Exception:
  print('not found')

Any one please help with accurate code? Thanks

amazon-web-services amazon-s3 python
2 Answers

Don't forget that there could be multiple files that match that wildcard.

You would use something like:

import boto3

s3 = boto3.resource('s3', region_name='ap-southeast-2')

bucket = s3.Bucket('my-bucket')

objects = bucket.objects.filter(Prefix='folder-name/')

for object in objects:
  if object.key.endswith('.txt'):
    object.download_file('/tmp/' + object.key)
user_1747770

This is a pretty OLD one and I am at loss that the main answer which has been accepted is a very poor and potentially dangerous one.

This essentially lists ALL objects and brings searching to the client side. On a bucket with thousands of objects (I guess most buckets) that is terrible.

What you need to do is to use .filter() instead of .all():

s3 = boto3.resource('s3')
buc = s3.Bucket("twtalyser")
for s in buc.objects.filter(Prefix='my/desired/prefix'):
    print(s)

UPDATE

The main answer was updated to reflect the point I was making.

user_4405020
Related