Spark failing because S3 files are updated. How to eliminate this error?

user_1955720 0
Viewed 3679

My Spark script is failing because the S3 bucket from which the df is drawn gets updated with new files while the script is running. I don't care about the newly arriving files, but apparently Spark does.

I've tried adding the REFRESH TABLE command per the error msg, but that doesn't work because it is impossible to know at execution time when the new files will arrive, and so no way to know where to put that command. I have tried putting that REFRESH command in 4 different places in the script (in other words, invoking it 4 times at different points in the script) - all with the same failure message

Caused by: java.io.FileNotFoundException: No such file or directory '<snipped for posting>.snappy.parquet'
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.

I create the df with: df = spark.table('data_base.bal_daily_posts')

So what can I do to make sure that S3 files arriving at S3 post-script-kickoff are ignored and do not error out the script?

apache-spark-sql apache-spark amazon-s3
2 Answers

Move files you're going to process to a different folder(key) and point spark to work with this folder only

user_39000850

This can happen if the same (e.g. if reading and writing directories are the same, Spark will first delete the files in destination) or other process has updated the files while they are being read.

user_39225170
Related