I would like to understand how to have scale an application (more than a few instances) consuming messages from Kafka on the same topic. My doubt is if I have an A topic with 4 partitions and I create an @KafkaListener with the attribute concurrency = 4 and a groupId = "FixedGroup" so spring will create 4 workers and these 4 workers will be chosen by kafka to consume the messages, one worker for each partition, if I scale up a new instance of the same application (scale), these 4 workers have already been assigned to work for each partition and the new application like this in the same group as the previous one (since it was just an autoscale) will be idle and the autoscale will not improve the message processing performance with horizontal scaling, how should the horizontal scaling and manage to distribute the message processing between the applications in the same topic/partition?
1 Answers
will not improve the message processing performance
That is correct; only one consumer in the same group can consume from a partition.
You need at least as many partitions as consumers.
In general, you should over-provision partitions. With a smaller number of consumers, the partitions will be distributed so each consumer is assigned multiple partitions.
There is no practical solution other than adding more partitions.
Related