How kafka producer send batches?

user_12824504 0
Viewed 647

I know that kafka producer will batch messages into batches. Each batch belongs to a particular partition.

My questiones are

  1. Does the producer know which partition each batch belonges to ?
  2. Does the producer know the broker address of each partition ?
  3. When producer send requestes, does each request contains multiple batches or contains just one batch which belongs to the target partition.
  4. If the producer send multiple batches, Does the receiving kafka server retransmit the batches to the target partition.
apache-kafka
1 Answers

  1. Producers assign the target partition to each record (if it's not provided by the application) so yes they know the partition for batches.

  2. When starting up, producers will retrieve the cluster metadata. This contains the current leader of partitions. Producers also refresh metadata periodically so they are able to discover updates or new partitions.

  3. A Produce request can contain batches for multiple partitions. You can see there's a RECORDS field for each partition in the protocol.

  4. Producers directly send records to the broker that is the partition leader.

user_17651890
Related