I have built a spark structured streaming application which reads the data from kafka topics,I have specified the starting offsets as latest and what happens if there is any failure from spark side, from which point/offset the data will continue to read after restarting and is it good idea to have checkpoint specified in the write stream to make sure we are reading from the point where the application/spark has failed? Please let me know.
Specifying checkpoint location when structured streaming the data from kafka topics
swetha k
0
Viewed 19
1 Answers
You can use checkpoints, yes, or you can set kafka.group.id (in Spark 3+, at least).
Otherwise, it may start back at the end of the topic
Related