Spring @Kafkalistener auto commit offset or manual: Which is recommended?

user_9652698 0
Viewed 3833

As per what I read on internet, method annotated with Spring @KafkaListener will commit the offset in 5 sec by default. Suppose after 5 seconds, the offset is committed but the processing is still going on and in between consumer crashes because of some issue, in that case after rebalancing, the partition will be assigned to other consumer and it will start processing from next message because previous message offset was committed. This will result in loss of the message.

So, do I need to commit the offset manually after processing completes? What would be the recommended approach?

Again, if processing is done, and just before commit, the consumer crashed, then how to avoid the message duplication in this case.

Please suggest the way which will avoid message loss and duplication. I am using Spring KafkaListener with default configuration.

spring-kafka apache-kafka spring-boot
2 Answers

As usual this depends on your use case and how you would like to deal with issues during your processing. The usage of auto-commit will change the delivery semantics of your application.

Enabling the auto commits is more an "at-most-once" semantics as you would read the data and commit it before you have actually processed the data. In case your processing fails the message was already committed and you will not read it again, it is therefore "lost" for your application (for your particular consumerGroup to be more precise).

Disabling the auto commit is more a "at-least-once" semantics as you are committing the data only after the processing of the data. Imagine you fetch 100 messages from the topic. 50 of them were processed sucessfullay and your application fails during the processing of the 51st message. Now, as you disabled auto commit and only commit all or none messages at the end of the processing, you have not committed any of the 100 messages, the next time your application reads the same 100 messages again. However, you have now created 50 duplicate messages as they were already processed successfully previously.

To conclude, you need to figure out if your use case can rather handle data loss or deal with duplicates. Dealing with duplicates can be ensured if your application is idempotent.

You are asking about "how to prevent data loss and duplicates" which means you are referring to "exactly-once-scemantics". This is a big topic in distributed streaming systems and you could check the spring-kafka docs if this is supported under which configuration and dependent on the output operation of your application.

Please also check the comment of GaryRussell on this post:

"the Spring team does not recommend using auto commit; the listener container Ackmode (BATCH or RECORD) will commit the offsets in a deterministic manner; recent versions of the framework disable auto commit (unless specifically enabled)"

user_122089100
  1. If the consumer takes 5+ seconds to process the message then you have a problem in the code that needs to be fixed.
  2. Auto-commit is risky in Production as can lead to problem scenarios (message loss etc.)
  3. Better to go with manual commit to have better control.
  4. Make the consumer idempotent so that duplicate message and WIP state of consumer is not a problem. May be, maintain processing status in consumer's DB so that if processing is half done then on consumer restart it can clear the WIP state and process afresh. Similarly, if processing status is Complete state then on restart it will see the Complete status and simply commit the duplicate message to Kafka.
user_39182500
Related