Not clear about the meaning of auto.offset.reset and enable.auto.commit in Kafka

user_7117816 0
Viewed 69557

I am new to Kafka,and I don't really understand the meaning of Kafka configuration, can anyone explain more understandable to me !

Here is my code:

 val kafkaParams = Map[String, Object](
  "bootstrap.servers" -> "master:9092,slave1:9092",
  "key.deserializer" -> classOf[StringDeserializer],
  "value.deserializer" -> classOf[StringDeserializer],
  "group.id" -> "GROUP_2017",
  "auto.offset.reset" -> "latest", //earliest or latest
  "enable.auto.commit" -> (true: java.lang.Boolean)
)

what does it mean in my code?

kafka-consumer-api apache-kafka
3 Answers

Adding more details on configurations mentioned in the title: "Not clear about the meaning of auto.offset.reset and enable.auto.commit in Kafka"

auto.offset.reset

With the auto.offset.reset configuration you can steer the behavior of your consumer (as part of a consumer group) in situations when your Consumer Group has never consumed and committed from a particular topic or the last committed offset from that Consumer Group was deleted (e.g. through cleanup policy).

Each message in a partition of a Kafka topic has a unique identifier which is the offset. Offsets are unique per Kafka partition. A consumer usually commits back the offsets on each partition of the topic it consumed. That way, the consumer is able to avoid duplicate readings.

Imagine you have a consumer reading from a topic for the first time (or if you change the consumer group name). The consumer group has therefore never committed any offsets. According to the Config Docs you can choose between the following behaviors with the configuration auto.offset.reset:

  • earliest: automatically reset the offset to the earliest offset

  • latest: automatically reset the offset to the latest offset

  • none: throw exception to the consumer if no previous offset is found for the consumer's group

  • anything else: throw exception to the consumer.

The default setting is latest.

enable.auto.commit

As mentioned above it is critical to think about your offsets and their commits when consuming messages from Kafka. When setting the configuration enable.auto.commit to true the consumer offsets will be committed automatically in the background.

In the JavaDocs of KafkaConsumer you will find a nice example on how to manually commit the offsets in a Consumer Client using

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.commitSync();

To emphasize the importance of the Offset Management in you consumer client again it is worth reading the whole Java Docs description or the confluent Kafka Documentation on Offset Management.

user_122089100
Related