we use spark structured streaming with Spark 2.2. at some point the streaming crashes and when it starts it tries reading from checkpoint and fails:
java.lang.IllegalStateException: Error reading delta file /XYZ/Checkpoint/myApp_V13-enrichment/state/0/7/132792.delta of HDFSStateStoreProvider[id = (op=0, part=7), dir = /XYZ/Checkpoint/myApp_V13/state/0/7]: /myApp/Checkpoint/myApp-V13/state/0/7/132792.delta does not exist
Caused by: java.io.FileNotFoundException: File does not exist: /mcspace/myApp/Checkpoint/myApp/state/0/7/132792.delta
two bugs were opened:
it seems this bug wasn't resolved even in spark 2.3
We search for a way to find out the missing delta file in order to search the previous delta file when the streaming app starts.
the only bypass we found is a remark written for the 2nd post: removing the checkpoint location along with the _spark_metadata folder in the affected writeStream output folder helped to get rid of the issue
is there a way for the streaming app to start from the last delta file that exists? or is the only way to delete all the delta files in the checkpoint? because this causes a data loss of all the delta files in that directory