Can't get total size of a bucket with Boto3

user_3080315 0
Viewed 9848

I'm trying to get the total size of a bucket. However total_size returns 0. Of course there are a couple of files in the bucket. If I have five files in my bucket the following function prints five zeros. What am I doing wrong?

bucket = boto3.resource('s3', config=Config(signature_version="s3", s3={'addressing_style': 'path'})).Bucket(name)
for object in bucket.objects.all():
    total_size += object.size
    print(object.size)
boto boto3 amazon-web-services amazon-s3 python
5 Answers

I see few issues:

  • Not sure about your call to boto3.resource(). Is that correct?
  • total_size not initialized

Try this:

total_size = 0
bucket = boto3.resource('s3').Bucket('mybucket')
for object in bucket.objects.all():
  total_size += object.size
  print(object.size)
print(total_size)

Or a one liner:

sum([object.size for object in boto3.resource('s3').Bucket('mybucket').objects.all()])
user_42377010

I am using this:

s3client = boto3.client('s3', region_name=region,
                            aws_access_key_id=access_key,
                            aws_secret_access_key=secret_key)
response = s3client.list_objects(Bucket=bucket_name)['Contents']
bucket_size = sum(obj['Size'] for obj in response)
user_43356130

Change signature_version="s3" to signature_version="s3v4".

I also like helloV's answer.

Also specify the region for the bucket instead of relying on the default configuration.

user_80167200

You can use this to get the size in GB:

import boto3
s3 = boto3.resource('s3')
bytes = sum([object.size for object in s3.Bucket('myBucket').objects.all()])
print(f'total bucket size: {bytes//1000/1024/1024} GB')
user_94207170

A simpler alternative is to use Amazon S3 Inventory to dump a list of objects on a daily basis, then calculate the totals from that.

user_1747770
Related