Write parquet files concurrently with pyspark

user_12173580 0
Viewed 977

In Azure Databricks I would like to write to the same set of parquet files concurrently from multiple notebooks using python / pyspark. I partitioned the target files so the partitions are disjoint / written independently which is supported according to databricks docs.

However I keep getting an error in my cluster logs and one of the concurrent write operations fails:

Py4JJavaError: An error occurred while calling o1033.save.
: org.apache.spark.SparkException: Job aborted.
...
Caused by: org.apache.hadoop.fs.PathIOException: `<filePath>/_SUCCESS': Input/output error: Parallel access to the create path detected. Failing request to honor single writer semantics

Here is the base path of where the parquet files are written to.

Why is this happening? What are the _SUCCESS files even for? Can I disable them somehow to avoid this issue?

parquet pyspark azure-databricks databricks
1 Answers

_SUCCESS is an empty file which is written at the very end of the process to confirm that everything went fine.

The link you provided is about delta only, which is a special format. Appently, you are trying to write a parquet format file, not a delta format. This is the reason why you are having conflicts.

user_50137520
Related