I have a Spark job which writes data to AWS S3, sometimes the application fails and this leaves the spark staging directory behind on S3 as shown below.
My question is what is the best practices to automate the cleanup of these directories and how to achieve that? Is there a Spark config to do so or there should be an independent step to perform the cleanup?