I have a single bucket which has very different Total Size and Number of Objects metrics between the S3 Management Console / CloudWatch, and the S3 command line tool.
S3 Management Console Metrics screen
- BucketSizeBytes: StandardStorage: 3.2TB
- NumberOfObjects: 172,000
S3 Management Console - Get Size Dialog
- Total Size: 264.0 GiB
- Total Objects: 127,030
Cloudwatch:
- BucketSizeBytes: 3.55TB
- NumberOfObjects: 172,000
Command Line Tool
aws s3 ls --summarize --human-readable --recursive s3://mybucket.com
- Total Size: 264.0 GiB
- Total Objects: 127,030
The command line tool and S3 Management Console - Get Size Dialog is returning what I'd expect.
We are being billed based on the much higher CloudWatch stats.
I wouldn't be bothered if they are close, but Cloudwatch is reporting close to 15 x as much StorageSize than the CLI tool.
There is no replication on the S3 bucket.
There are lifecycle policies to:
- Clean up incomplete multipart uploads
- Send files +1 years to Glacier
- Permenantly delete previous versions after a year
Versioning is turned on but I am pretty sure it's not used; i.e. files have only ever been uploaded once. I have tested a few random files to confirm, and they all only have one, the latest, version, available. I am happy to turn this off at the bucket settings, but I understand this won't 'delete' old versions (if they exist) anyway?
I don't believe it's because of Glacier stored files either, as I only recently set this lifecycle policy up, and the size mismatch was present before that.
And I'm extra confused given the S3 Management Console UI > Overview > Select all folders > Actions > Get Size reports the same as the CLI tool, 264GB.
So, seleting all filers/folders in the UI returns 264GB, but the Metrics return 3.2TB.
Can anyone please shed some light on what's going on here, and how I can clean it up! Thanks Rob