Writing to Google Cloud Storage with v2 algorithm safe?

aah. I wrote that bit of the docs. And one of the papers you cite.

1. GCP implements rename() non-atomically, so v1 isn't really any more robust than v2. And v2 can be a lot faster.
2. Azure "abfs" connector has O(1) Atomic renames, all good.
3. S3 has suffered from both performance and safety. As it is now consistent, there's less risk, but its still horribly slow on production datasets. Use a higher-perfomance committer (EMR spark commtter, S3A committer)
4. Or look at cloud-first formats like: Iceberg, Hudi, Delta Lake. This is where the focus is these days.

https://databricks.com/blog/2017/05/31/transactional-writes-cloud-storage.html

We see empirically that while v2 is faster, it also leaves behind partial results on job failures, breaking transactionality requirements. In practice, this means that with chained ETL jobs, a job failure — even if retried successfully — could duplicate some of the input data for downstream jobs. This requires careful management when using chained ETL jobs.

It's safe as long as you manage partial writes on failure. And to elaborate, they mean safe in regard to rename safety in the part you quote. Of Azure, AWS and GCP only AWS S3 is eventual consistent and unsafe to use with the V2 algorithm even when no job failures happen. But GCP (nor Azure or AWS) is not safe in regards to partial writes.

FileOutputCommitter V1 vs V2

1. mapreduce.fileoutputcommitter.algorithm.version=1

AM will do mergePaths() in the end after all reducers complete. If this MR job has many reducers, AM will firstly wait for all reducers to finish and then use a single thread to merge the outout files. So this algorithm has some performance concern for large jobs.

2. mapreduce.fileoutputcommitter.algorithm.version=2

Each Reducer will do mergePaths() to move their output files into the final output directory concurrently. So this algorithm saves a lot of time for AM when job is commiting.

• http://www.openkb.info/2019/04/what-is-difference-between.html

If you can see Apache Spark documentation Google cloud marked safe in v1 version so its same in v2

What, exactly, does it mean for the algorithm to be "not safe"?

S3 there is no concept of renaming, so once the data is written to s3 temp location it again copied that data to new s3 location but Azure and google cloud stores do have directory renames

AWS S3 has eventual consistent meaning If you delete a bucket and immediately list all buckets, the deleted bucket might still appear in the list ,eventual consistency causes file not found expectation during partial writes and its not safe.

What are the concrete set of criteria to use to decide if I am in a situation where v2 is not safe?

• What is the best practice writing massive amount of files to s3 using Spark

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel

• https://spark.apache.org/docs/3.1.1/cloud-integration.html#recommended-settings-for-writing-to-object-stores

• https://databricks.com/blog/2017/05/31/transactional-writes-cloud-storage.html

• https://github.com/steveloughran/zero-rename-committer/files/1604894/a_zero_rename_committer.pdf