For reference, I'm having the exact same issue reported here without answers: Databricks spark job writing/updating _SUCCESS file twice on job completion
I have a lambda that is triggered by s3 events on the _SUCCESS file. When the dataframe is written via Databricks/Spark with df.write.parquet(path) the _SUCCESS file is written twice, as you can see from the s3 file versioning:
This messes up with my lambda as I'm expecting just one _SUCCESS file (and as reported above, this happens only when using databricks.