Kafka Topic Partitioner function - configuration question

user_10563020 0
Viewed 1267

I am trying to write a Kafka producer using librdkafka (c code library). I do not want my application to have to manually round-robin through the number of partitions on a topic. In the Java version of the Kafka Producer, this is possible by not specifying the partition or key. The documentation in librdkafka states the following:

 * \p partition is the target partition, either:
 *   - RD_KAFKA_PARTITION_UA (unassigned) for
 *     automatic partitioning using the topic's partitioner function, or
 *   - a fixed partition (0..N)
RD_EXPORT
int rd_kafka_produce(rd_kafka_topic_t *rkt, int32_t partition,
              int msgflags,
              void *payload, size_t len,
              const void *key, size_t keylen,
              void *msg_opaque);

So what is the topic's partitioner function? I cannot find documentation on this anywhere. Is this round-robin by default? Is this something I need to enable in Zookeeper?

apache-kafka
1 Answers

If you don't explicitly specify the partition when producing, by using RD_KAFKA_PARTITION_UA, librdkafka will automatically determine the partition using a partitioner. The strategy is defined by a setting named partitioner, the default is called consistent_random and uses

CRC32 hash of key (Empty and NULL keys are randomly partitioned)

See https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md for the full details. Also note that if you want exactly the same behaviors as Java you need to set partitioner to murmur2_random.

Finally you can also write your own partitioner logic. In that case, you need to set partitioner_cb or rd_kafka_topic_conf_set_partitioner_cb()

user_17651890
Related