Can single consumer read from multiple partitions of a kafka topic?

user_3317808 0
Viewed 4080

There is a kafka topic with 16 partitions

With a given consumer group name, we are currently launching single consumer to read from the topic.

  1. Does single consumer read from partition 0(only) of that topic? If partition 0 has gone empty with messages, does consumer start reading from next partition(partiton 1... and so on)?

  2. we have option to launch more than consumer(with same consumer group name) to read from same topic(having 16 partitions). How many consumers can be maintained, to read from multiple partitions parallely?

kafka-topic apache-kafka
2 Answers

The consumers in a group divide the topic partitions as fairly amongst themselves as possible by establishing that each partition is only consumed by a single consumer from the group. When the number of consumers is lower than partitions, same consumers are going to read messages from more than one partition.

In your scenario, a single consumer is going to read from all your partitions. This type of consumer is known as exclusive consumer. This happens when consumer groups have only one consumer. Such a consumer must be connected to all partitions it requires.

Ideally, the number of partitions should be equal to the number of consumers. Should the number of consumers be greater, the excess consumers were to be idle, wasting client resources. If the number of partitions is greater, some consumers will read from multiple partitions, which should not be an issue unless the ordering of messages is important.

Ordering

Kafka does not guarantee ordering of messages between partitions. It does provide ordering within a partition. Therefore, Kafka can maintain message ordering for a consumer if it is subscribed to only a single partition.

If message ordering is required in your use case, the messages should be ordered using a key to be grouped by during processing.

user_70666470

you have a topic with 16 partitions

With a given consumer group name, you are currently launching single consumer to read from the topic This single consumer will read from all partitions.

Since you can launch multiple consumer (within same consumer group).

For 16 partitions you should have max 16 consumer listening to each partitions. In Kafka within a consumer group you can have max 1 consumer per partition.

user_11154220
Related