Azure Event Hubs Streaming: Does Checkpointing override setStartingPosition?

user_959788 0
Viewed 344

If we specify the starting position in EventHub conf like so:

EventHubsConf(ConnectionStringBuilder(eventHubConnectionString).build)
  .setStartingPosition(EventPosition.fromStartOfStream)
or
  .setStartingPosition(EventPosition.fromEndOfStream)

And also sepecify the checkpoint location in the StreamWriter

streamingInputDF
  .writeStream
  .option("checkpointLocation", checkpointLocation)
  ...

After a restart, does the setStartingPosition become irrelevant because the checkpoint is always used as the point from where to begin reading?

Thanks.

checkpointing azure-eventhub spark-structured-streaming event-driven-design
1 Answers

The information on offsets stored in the checkpoint files will be used when restarting the streamimg query.

Interestingly, this is not specifically mentioned in the structured streaming eventhubs integration guide, however, in the DStreams guide it is:

"The connector fully integrates with the Structured Streaming checkpointing mechanism. You can recover the progress and state of you query on failures by setting a checkpoint location in your query. This checkpoint location has to be a path in an HDFS compatible file system, and can be set as an option in the DataStreamWriter when starting a query."

Make sure to follow the general guidance on checkpoint recovery.

user_122089100
Related