I am trying to write dataframe in parquet format to NFS (Network File System) by command:
df.write.mode('overwrite').parquet(OUTPUT_DIR)
It works fine with HDFS but with NFS, only _temporary is created and no actual parquet files are written. Job is successfule without any errors. Jobs writitn same df to HDFS and NFS, have same logs for OutputCommitter:
FileOutputCommitter: Saved output of task 'attempt_XXXX_0002_m_000000_0' to path_to_DF/_temporary/0/task_XXXX_0002_m_000000
SparkHadoopMapRedUtil: attempt_XXXX_0002_m_000000_0: Committed
Executor: Finished task 0.0 in stage 2.0 (TID X). 2707 bytes result sent to driver
Thanks.