When Kafka sends multiple batches in the same request?

user_4685486 0
Viewed 125

Kafka sends a Batch as soon as batch.size of bytes is collected or linger.ms time has passed.

Then how come there can be multiple Batches in a single request? Does it mean that when Kafka is ready to send a Batch for one partition, Batches for other partitions would be closed and sent in the same request?

Doesn't it affect compression efficiency for the Batches that were not closed by batch.size or linger.ms ?

apache-kafka
1 Answers

After investigating the source code it looks like when there is data ready to be sent to a node (at least one batch for partition of this node is ready), then Sender will take for each partition of the node the ready batch that fits into request and sends them all. So if the batch is not yet "ready" (by bytes or by time) it is not included in the batch. So the compression rates are not affected.

user_46854860
Related