How to safely unsubscribe to a topic in Kafka

user_2324703 0
Viewed 3191

I have a simple java program (dockerized) and deployed in kubernetes (pod) This java program is just a normal java project that listens and consumes to a specific topic. eg. SAMPLE-SAFE-TOPIC

I have to unsubscribe to this topic safely, meaning no data will be lost even I deleted this pod (java consumer).

This is the code that I saw from searching:

 public static void unsubscribeSafelyFromKafka() {  

  logger.debug("Safely unsubscribe to topic..");

  if (myKakfaConsumer != null) {
        myKafkaConsumer.unsubscribe();
        myKafkaConsumer.close();
     }
}

I need to run this via command line wherein the Java program has already an existing static main method.

My questions are:

  1. Is the code above guarantees no records will be lost?
  2. How can I trigger the code above via command line when there is already an existing static main()

Note: I am running the java project via command line. E.g. java -jar MyKafkaConsumer.jar as this is the requirement.

Please help

spring-kafka apache-kafka java
1 Answers

If I understand question 1 right you are concerned that after unsubscribing via one thread triggered by a console command there is a risk that the polling consumer is processing a batch of records that might be lost if the pod is killed?

If you have other pods that are consuming as part of the same consumer group, or if this or any pod subscribes again with the same group ID then the last committed offset will ensure that no records are lost (though some could be processed more than once) as that is where the consumer that takes over will start from.

If you use auto-commit that is safest as each commit happens in a subsequent poll so you cannot possibly commit records that haven't been processed (as long as you don't spawn additional threads to do the processing). Manual commit leaves it to you to decide when records have been dealt with and hence when it is safe to commit.

However, calling close after unsubscribe is a good idea and should ensure a clean completion of the current polled batch and commit of the final offsets as long as that all happens within a timeout period.

Re question 2, if you need to manually unsubscribe then I think you'd need JMX or expose an API or similar to call a method on the running JVM. However if you are just trying to ensure safe shutdown when the pod terminates, you could unsubscribe in a shutdown hook, or just not worry, given the safety provided by offset commits.

user_25686490
Related