What is the best/fastest approach to check if multiple files exist in AWS S3 bucket?
For example I have 100k files metadata in my local DB. I would like to make sure all of them exist in S3 bucket. I can do 'aws s3 ls' for particular file, but that would mean 100k aws requests. Is there a better approach to this?
If you are just doing a general audit, you could use Amazon S3 Inventory to obtain a complete daily dump of all object keys and associated metadata.
You could then write some code to compare the contents of the Inventory file against the DB entries.
If you want to retrieve all keys in a specific bucket in one command then you can use this.
aws s3api list-objects --bucket <bucket-name> --no-paginate
Once you have that list, you can process it by a custom code.
If you would like to make sure your local files are on S3 you can try the s3 sync command.
You can also check out which files are there currently with Commandeer, which supports S3 file browsing in a nice tree view.
