consume kafka topics with different partition numbers

Dushan 0
Viewed 25

Hi i have a kafka consumer (using spring kafka dependency) that listens to multiple topics. Lets say i have 3 topics which are topicA, topicB and topicC. In my application i consume all three topics in one consumer like below.

@KafkaListener(topics = "topicA,topicB,topicC", groupId = "myGroup", concurrency="3")

My topics have partitions and those number of partitions are deferent from each. Lets say my topicA has 3 partitions. topicB have 6 partitions and topicC has 9 partitions. How should i determine a number for "concurrency" option in @KafkaListener. (I'm confused since topicB and topicC contain 6 and 9 partitions respectively. So should i change the concurrency to 6 or 9 ? or should i change it to 18 which is total number of partitions from 3 topics)

I know that on the consumer side, Kafka always gives a single partition’s data to one consumer thread and the degree of parallelism in the consumer (within a consumer group) is bounded by the number of partitions being consumed. My main goal is to consume parallelly by using concurrency option in @kafkalistener

kafka-consumer-api spring-kafka apache-kafka
1 Answers

If you set the concurrency to 18, with the default partition assignor, if the concurrency is greater than the number of partitions, you will have idle consumers. The partitions from different topics have no bearing on how the partitions are distributed.

You can use a custom partition assignor (in the consumer configuration) to distribute the partitions differently.

See https://kafka.apache.org/documentation/#consumerconfigs_partition.assignment.strategy

Also see the discussion about RoundRobinAssignor here https://docs.spring.io/spring-kafka/docs/current/reference/html/#using-ConcurrentMessageListenerContainer

Or, simply add 3 separate @KafkaListener annotations to the method, one for each topic, with different concurrencies.

Gary Russell0
Related