Exclude objects from AWS S3 Inventory report

user_14605524 0
Viewed 286

I have an S3 bucket that contains millions of objects. I want to set up an AWS S3 inventory for that bucket. However, I want to exclude all objects inside the test/ folder present at the bucket's root from the inventory report (CSV). There is an option in the console while configuring inventory which is: Prefix which I think is a filter to include objects and not to exclude them.

How can I generate an inventory report of all bucket objects except the objects present inside the test/ folder? Is there any way I can do this from the S3 Inventory console?

amazon-web-services amazon-s3
1 Answers

Unfortunately, there is no option to exclude a prefix. You can use prefix to indicate inclusion, not exclusion. From docs:

Amazon S3 inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet (Parquet) output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or a shared prefix (that is, objects that have names that begin with a common string).

However, you could post process the inventory file with Athena to get only records you require:

Athena can query Amazon S3 inventory files in ORC, Parquet, or CSV format. When you use Athena to query inventory, we recommend that you use ORC-formatted or Parquet-formatted inventory files.

user_2488230
Related