How to set batch size in one micro-batch of spark structured streaming

user_7486621 0
Viewed 3359

I am reading streaming data from Kafka source, but all the data from kafka is read in a single micro batch.

spark.readStream.format("kafka").option("kafka.bootstrap.servers",bootstrap_servers).option("subscribe", topics).option("startingOffsets", "earliest").load()

Which parameter or option is used to set max batch size in one micro-batch of spark structured streaming?

spark-structured-streaming pyspark apache-kafka apache-spark
1 Answers

use maxOffsetsPerTrigger to limit the no of messages.

as per spark doc "maxOffsetsPerTrigger - Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume."

user_30120930
Related