AWS S3: Deleting all parquet files in multiple S3 subfolders

P.Max 0
Viewed 44

We have very many folder and subfolders in each S3 bucket where parquet files are stored, issue is when data error occurs, we have to manually go in to each folder/subfolder/sub-subfolder etc to delete all the parquet files.

Is there a way either via a) AWS cli or b) using a py-script and running it in a AWS Glue jobs that it could delete all parquet files in multiple sub and sub-sub folders?

Option b) would be great, and ideally a script that runs without having to specify the subfolder. Our folder structure is set, so if script needed paths to all subfolders, we could live with that solution also.

Thanks in advance!

amazon-s3 parquet amazon-web-services
1 Answers

Yes, you can delete all objects under a specific partition/prefix as follows using the awscli:

aws s3 rm s3://folder/2022/09/ --recursive

The prefix could be YYYY/, YYYY/MM/, YYYY/MM/DD/ etc. as needed.

Note that this will delete all objects below the specific prefix.

You indicated a preference to retain the folder structure while deleting the objects but in the general case this is not possible because there are no actual folders. They're virtual and only appear to be present when objects with a given key prefix exist. If you want to create a given folder structure then you can do that by simply writing additional objects to the relevant key prefixes e.g. folder/2022/09/dummy.txt or even a zero-sized object at folder/2022/09/ to represent the folder.

jarmod0
Related