We have a use case where we read from a Kafka once every hour and save the events to HDFS after a couple of small transformations. We don't want to keep this job running but only schedule it once an hour. This job is part of a workflow of jobs so it needs to run after a pre-requisite job has finished it's hourly run.
So the very first run of this job would read entire topic and save. And every subsequent run after that will be from previously saved offsets.
With spark 2.4.8, this used to be fine. The first run takes about 2 hours and every subsequent run after that took less than 10 minutes. With 2.4.8 we don't use a streaming query, we run a batch query and manage the offsets ourselves.
So we use
spark.read
AND NOT
spark.readStream
We are now migrating to Spark 3.2.0 so I have tried the below two options
I tried using Spark Structured Streaming spark.readStream with Trigger.Once so spark can do all the offset management itself and I schedule it in airflow once an hour.
I also tried the batch way with spark.read and do the offset management myself, like we do today.
But either approaches are way too slow for some reason. The first run took about 14 hours to finish in either case and the subsequent runs are also in the hours range.
I have been trying to debug and understand what I am missing but all I could gather from the logs are a ton of messages like this below
Seeking EARLIEST
Resetting Offset
Seeking Latest
Resetting Offset
The Kafka we use is 2.7 Can you help me what I'm missing here. Or is there any other option to try and understand this slowness.