I was going through the structured streaming docs on trigger interval: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers
I used the trigger interval as 30 seconds while publishing data to kafka
val query = dfToWriteInKafka.writeStream
.queryName("kafkaWriter")
.format("kafka")
.outputMode("append")
.trigger(Trigger.ProcessingTime(30000))
.option("kafka.bootstrap.servers", bootstrapServers)
.option("topic", topicName)
.option("checkpointLocation", config.getString("job.checkpoint"))
.start()
I see that for the first couple of runs, the micro-batch interval is not 30 seconds. For the later runs, it does trigger the batch at 30 seconds.
I want to understand how does it decide how many records to pick in the 1st run. I know I have specified checkpoint location, but why is the trigger interval different for the first 2 runs.
Also, how does spark decide how many records to pick in the first run ?
