From the latest runs, we observed a strange behaviour in writing the file to output S3. We generally overwrite the latest output to s3 and then it would overwrite all the old files in the folder. But, from past 2 successful runs we see that the files are not overwriting in s3 output but it is appending the data (no changes made in the code and it has mode=overwrite). The only change we made relates to Databricks jobs is to modify the cluster configuration. We are not sure whether this change is related to that as we followed this as a standard in all the other jobs.
0 Answers
Related