Pyspark remove local/hdfs file/folder

user_11151394 0
Viewed 666

I'm doing rdd.saveAsTextFiles(path), but sometimes, it will report error:

py4j.protocol.Py4JJavaError: An error occurred while calling o41.saveAsTextFile.
: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/D:/code/use_spark/output already exists or hdfs://user/output exists

Every time I need remove existed file/folder mannally, it is not really smart.

Is there a way to delete the file/folder in hdfs/local by using pyspark API. Dont use command line in code.

pyspark
1 Answers

Use java spark api in pyspark:

fs = sc._jvm.org.apache.hadoop.fs.FileSystem.get(sc._jsc.hadoopConfiguration())
if fs.exists(sc._jvm.org.apache.hadoop.fs.Path(args.output_dir)):
    fs.delete(sc._jvm.org.apache.hadoop.fs.Path(args.output_dir))
user_111513940
Related