custom partition in kafka using .net (using hash based algorithm )

user_2247651 0
Viewed 510

I want to use custom partition in kafka using .net application.I have created a topic , which is divided in 15 partition. I have multiple message , that comes to our web services. I want to use partition in a way that each message goes to its partition when comes to our service. each message has unique identifier and always comes with that identifier. So suppose the message came for 2345 id and first time it is stored in partition 1 , then always this message should go on this partition.

In java i found some code that was using hash based algorithm to rotate the message in the partition. can we implement these functionality in .net?. I found some thing in librdkafka, but this feature in under development. I have to implement custom partitioner in kafka using .net.

Please help me or give me some suggestion.

Thanks

kafka-producer-api apache-kafka .net
1 Answers

If I properly understand your usecase you want a message with a particular identifier to be stored on a particular partition.

Kafka is smart enough to handle this, you must use the identifier as the Key of the message , Kafka will determine the partition using that Key which I presume will not be changing for a particular message type. Kafka will always store the message with same Key on a particular partition.

user_50389810
Related