We are using spark Structured Streaming with foreachbatch to update records in delta table. The number of records part of each batch are random. We have 10000 record in kinesis stream but while creating micro batch it picks random number of records that is some time 800, 500 and some time . This takes long time in processing because ideally it should create 2 batch of 5000 each.
0 Answers
Related