Apache Kafka: 3 partitions, 3 consumers in the consumer group, each consumer should be multithreaded

user_3405101 0
Viewed 484

Lets say there is a topic in Apache Kafka with 3 partitions. I need to run 3 consumers inside a consumer group and, according to the documentation, it means each consumer will read data from 1 partition.

Consumers are implemented using Spring Kafka. As we all know, by default all messages are received in a single thread, but using ConcurrentMessageListenerContainer should allow us to set up concurrency.

What I want? I want to use server CPU resources efficiently and make each consumer to receive and process messsages in separate threads (3 threads in our case, which is equal to the number of partitions).

As a result - 3 consumers (3 servers) in the consumer group and each consumer receives messages from all 3 partitions.

Is it possible? If yes, will it be enough if I just use ConcurrentMessageListenerContainerand specify 3 listeners for each partition?

spring-kafka apache-kafka multithreading java
1 Answers

I was little confused by your statement. Just to clarify, in Kafka only one consumer can read from one partition within a consumer group. It is not possible for two consumers in same consumer group to read from same partition.

Within a consumer group,

  • if no of consumers is greater than number of partition, then extra consumer threads will be idle.
  • if no of consumers is less than number of partition, then same consumer thread will read from multiple partitions

this code snippet will read from topic named "mytopic" and it will use 3 thread to read from 3 partitions @KafkaListener(topics = "mytopic", concurrency = "3", groupId = "myconsumergroup")

user_38172060
Related