I'm using delete from org.apache.hadoop.fs package to exclude some HDFS files. But I don't want to move them to my user's trash folder, I want to permanently delete all. How can I do this?
Below is an example of my code:
fs = spark.sparkContext._jvm.org.apache.hadoop.fs.FileSystem.get(self.spark.sparkContext._jsc.hadoopConfiguration())
fs.delete(self.fileSystem.Path('my/dir/'))