Spring Boot Kafka Listener concurrency

user_884444 0
Viewed 1357

I am using Spring Boot @kafkaListener in my application. Lets assume I use below configuration -

 Topic Partitions : 2

 spring.kafka.listener.concurrency : 2

 group-id : TEST_GRP_ID

 Acknowledgement : Manual

My question is ,

  1. As per my knowledge Concurrency will create parallel thread to consume message. So, thread 1 consumed the batch of records and thread 2 consumed the batch of records in this case processing of the messages will sequential and then commit the offset?

  2. If I have two instances of the micro service in my cloud environment (in production more partition and more instances), then how concurrency will work? In each instance will create two parallel thread for my Kafka consumer?

How can I improve performance of my consumer or how can I make fast consumption and processing of the messages?

spring-kafka spring-boot
2 Answers

Your understanding is not too far from the truth. In fact only one consumer per partition can exist for the given group. The concurrency number gives us an approximate number of target consumers. And independently of microservice instances only two maximum consumers can exist if you have only two partitions in your topic.

So, to increase a performance you need to have more than 2 partition or more topics to consume, then they all can be distributed between your instances and their consumers evenly.

See more info in Apache Kafka docs: https://docs.confluent.io/platform/current/clients/consumer.html

user_27565470

✓you are having concurrency as 2 , which means 2 containers will be created to your listener.

✓As you are having 2 partitions in topic , messages from both the partitions will be consumed and processed parallelly.

✓When you spin up one more instance with same group name , the first thing that will happen is Group Rebalance . ✓Despite this event , as at any point of time only one consumer from a specific consumer group can be there for a partition , In the end , only 2 containers will be listening to messages and other 2 containers just remain idle.

✓In order to achieve more scalability , we need to add more number of partitions to the topic there by we can have more number of active listener containers

user_84603130
Related