I am currently creating a single csv file with the following code in pyspark:
df.coalesce(1).write.format("csv").option("header", "true").mode("append").save("s3:path/")
It creates a file named part0000.csv. I used the following code to rename the file:
hdfs_dir = "s3://path"
new_filename="results.csv" #new filename
df.coalesce(1).write.format("csv").option("header", "true").mode("append").save(hdfs_dir)
fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration())
list_status = fs.listStatus(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dir))
file_name = [file.getPath().getName() for file in list_status if file.getPath().getName().startswith('part-')][0]
fs.rename(Path(hdfs_dir+''+file_name),Path(hdfs_dir+''+new_filename))
This code is working only for HDFS and not for s3. Please let me know if there is a way to rename.