Data loss scenarios while restarting a streaming query with Kafka as source

user_3910099 0
Viewed 670

In the book "Spark Streaming with Apache Spark", the authors mention that the field "failOnDataLoss (default: true)" should be set to false while testing with Kafka as the source. They say,

This flag indicates whether to fail the restart of a streaming query in case data might be lost. This is usually when offsets are out of range, topics are deleted, or topics are rebalanced. We recommend setting this option to false during the develop/test cycle because stop/restart of the query side with a continuous producer will often trigger a failure. Set this back to true for production deployment.

I want to understand why would there be failures in stopping/restarting the query. How would offsets get out of range? Doesn't Spark maintain what offsets to read as checkpointing?

spark-streaming spark-structured-streaming apache-spark
1 Answers

I aggree that the reason "because stop/restart of the query side with a continuous producer will often trigger a failure" does not really make sense. As you said, Spark maintains the current offsets. In addition, stopping/restarting the query while having a continuous producer is exactly what we aim for on a productive system.

Therefore, I could only guess what is meant by that statement and my best guess is that there is a typo and it should have been written "without a continuous producer".

You usually do not have a continuous producer on development environments and, to save some disk space, you usually have a short retention period in your topic on a development environment.

During the development phase you produce a few messages manually to your input topic and run your Spark Structured Streaming job to see if it behaves as expected. Let's say you have produces messages with offsets 0 to 10. Then, someone in your team want to test another consumer on that topic and produces messages with offsets 11 to 20. After a while, in the next development phase (assume it is 3 weeks later) you want to test another feature for which you will again manually produce some messages. However, the cleanup policy of your Kafka topic already removed all old offsets 0 to 20 and when restarting your streaming query it tries to read offset 11 which does not exist anymore.

Another common scenario is when you remove all data on development environment from a topic to be able to "start from scratch".

These scenarios often show up on development environments and it is therefore recommended to set the option failOnDataLoss to false.

user_122089100
Related