There is a well-known issue in Apache Spark: if you are reading a table which is being updated at the same time - you can face error like this:
Caused by: java.io.FileNotFoundException: No such file or directory '<snipped for posting>.snappy.parquet'
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
Recommended solution is to call REFRESH TABLE tableName.
In my case the issue is that the table I want to read is being updated NOT BY ME at some random moments. Actually the update can start right after I have started my job, or may be it was already going on when I started my job.
What can I do to prevent the error? The update of the source table may be ongoing, it doesn't make sense to call REFRESH TABLE tableName in a cycle, and I cannot wait for that update to be done (actually I dont know how to do it). I just want to ignore it and read some consistent version of the table.
Is there a way to tell spark that I dont care if it reads outdated, but consistent version of the table?