How do I plug custom partition logic in WriteKafkaP?

Question

I'm using the WriteKafaP vertex in a Jet job. Is there a way to plug in custom partition logic when publishing to the Kafka topic?

user_531098 · Answer

In 0.4, you can use a custom Kafka partitioner configuration. In 0.5, we added the ability to map the item to a ProducerRecord directly.

For example:

Vertex v = new Vertex("kafkaSink", KafkaProcessors.writeKafkaP(properties, o -> new ProducerRecord(topic, partition, key, value));