How can I get a list of all `PartitionTopic`s for a topic in Confluent.Kafka?

user_13927996 0
Viewed 1604

I am using confluent-kafka-dotnet (Confluent.Kafka) to produce messages for Kafka.

I would like to manually manage which partitions messages go to in order to preserve order for certain groups of messages.

How is it possible to get a list of PartitionTopic for a Kafka topic?

I have looked at this solution in Java. But I don't understand how to achieve the same functionality with Confluent.Kafka.

Alternatively, it would be acceptable to send messages with keys, because the same keys are guaranteed to be on the same partitions. But again, I could not find a way to create a new Message with any key type other than Null. So, an example of sending a message with a non-null key would be helpful.

apache-kafka c#
2 Answers

To get a list of TopicPartitions for a single topic you could use AdminClient class:

using (var adminClient = new AdminClientBuilder(new AdminClientConfig { BootstrapServers = "bootstrap-servers" }).Build())
{
    var meta = adminClient.GetMetadata(TimeSpan.FromSeconds(20));

    var topic = meta.Topics.SingleOrDefault(t => t.Topic == "topic-name");

    var topicPartitions = topic.Partitions;
}

You can find more AdminClient examples here. But notice the warnings, saying that: "The API for this functionality is subject to change."

user_68275080

I have worked out that the reason I could not create messages with a non-null key was because I had specified my Producer with <Null, ...>.

Thank you to mjwills for asking me for minimal reproducible example, which prompted me to work this out.

user_139279960
Related