What are the STARTED, COMMITTED , and SUCCESS files that are created in the underlying storage folder when writing/creating a Spark Parquet table? Can there be multiple of those file? If so, what does it mean to have more than one of them?
Thanks.
1 Answers
Those files are stored there by the DBIO transactional protocol.
With DBIO transactional commit, metadata files starting with
_started_<id>and_committed_<id>accompany data files created by Spark jobs. Generally you shouldn’t alter these files directly. Rather, you should use the VACUUM command to clean them up.
https://docs.databricks.com/spark/latest/spark-sql/dbio-commit.html
Related