How does Trigger work in Spark Structured Streaming?

Viewed 1035

I was going through the structured streaming docs on trigger interval: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers

I used the trigger interval as 30 seconds while publishing data to kafka

val query = dfToWriteInKafka.writeStream

  .queryName("kafkaWriter")
  .format("kafka")
  .outputMode("append")
  .trigger(Trigger.ProcessingTime(30000))
  .option("kafka.bootstrap.servers", bootstrapServers)
  .option("topic", topicName)
  .option("checkpointLocation", config.getString("job.checkpoint"))
  .start()

I see that for the first couple of runs, the micro-batch interval is not 30 seconds. For the later runs, it does trigger the batch at 30 seconds.

I want to understand how does it decide how many records to pick in the 1st run. I know I have specified checkpoint location, but why is the trigger interval different for the first 2 runs.

Also, how does spark decide how many records to pick in the first run ?

Structured Streaming trigger intervals

0 Answers
Related