state.HDFSBackedStateStoreProvider: The state for version 1780 doesn't exist in loadedMaps

Viewed 260

I am using Spark 2.4.0, kafka 2.10. I am currently working on a Spark Structured Streaming window job and running on YARN. it seems that on every batch interval I get the warning then the job is down:

WARN state.HDFSBackedStateStoreProvider: The state for version 1780 doesn't exist in loadedMaps. Reading snapshot file and delta files if needed...Note that this is normal for the first batch of starting query.

the warn log picture

the stdout log picture2

Here is my code:

df.writeStream\
    .foreachBatch(handle_kafka.__execute)\
    .option("checkpointLocation", "/user/hive/warehouse/tempdb.db/kafka_silver_gamelog_streaming") \
    .outputMode("update")
    .start()

Please help me out for the same. There may be workarounds for this issue, please suggest me if any.

0 Answers
Related