We have some gluejobs which uses Spark dataframes. We write the dataframe to a parquet file in the job like this.
matched.write.mode("overwrite").parquet(output)
We then set the bookmark, which I have now learned has nothing to do with persisting data, rather tracking data about the job.
job.commit()
Now, if I throw any exception after matched.write.mode("overwrite").parquet(output) then my data does not get persisted to a parquet file and Athena cannot read it.
If I just let the script contiues, it persists the data.
My question is then where and how does the gluejob actually persist the data that was written here -> matched.write.mode("overwrite").parquet(output) To me it seems as if the data is only persisted to parquet if we allow the gluejob to finish executing normally? Is this assumption correct? This is what all of my tests confirm but I may be missing something as I used to think that matched.write does the writing and persisting already.