How to configure backpreasure in Spark 3 Structure Stream Kafka/Files source with Trigger.Once option

user_2870704 0
Viewed 150

In Spark 3 Behave of backpressure option on Kafka and File Source for trigger.once scenario was changed.

But I have a question. How can I configure backpressure to my job when I want to use TriggerOnce?

In spark 2.4 I have a use case, to backfill some data and then start the stream. So I use trigger once, but my backfill scenario can be very very big and sometimes create too big a load on my disks because of shuffles and to driver memory because FileIndex cached there. SO I use max maxOffsetsPerTrigger and maxFilesPerTrigger to control how much data my spark can process. that's how I configure backpressure.

And now you remove this ability, so assume someone can suggest a new way to go?

spark-kafka-integration spark-structured-streaming apache-spark
1 Answers

Trigger.Once ignores these options right now (in Spark 3), so it always will read everything on the first load.

You can workaround that - for example, you can start stream with trigger set to periodic, with some value like, 1 hour, and don't execute .awaitTermination, but have a parallel loop that will check if first batch is done, and stop the stream. Or you can set it to continuous mode, and then check if batches having 0 rows, and then terminate the stream. After that initial load you can switch stream back to Trigger.Once

Alex_Ott_186270
Related