KafkaConsumer visibility across threads

user_6841669 0
Viewed 118

I have an application which has two threads running. These threads share a KafkaConsumer instance between them.

KafkaConsumer is created by disabling the auto commit config. One thread will assign topic-partitions to the KafkaConsumer instance based on some external input. The other thread constantly polls the topic, process the records and commit the offsets at the end.

Now, let's say the topic-partitions assignment was changed in Thread-1. What can we do to make sure that Thread-2 can see these changes? i.e Thread-2's KafkaConsumer will know that topic-partitions have been updated in Thread-1.

Does defining the variable as volatile solve this issue? Or should we do anything more? Please provide your explanantions with code examples as I am new to multi-threading in Java.

kafka-consumer-api apache-kafka java
1 Answers

One thread will assign topic-partitions to the KafkaConsumer instance based on some external input. The other thread constantly polls the topic, process the records and commit the offsets at the end.

According to the javadoc (Section Multi-threaded Processing):

The Kafka consumer is NOT thread-safe.

With that I'm not so sure you can change partition assignment in one thread and see the changes in the other.

user_13053440
Related