When an application using Kafka is restarted, the offset is often initialized

Viewed 66

I am currently using kafka 0.9, and set the property as follows.

    setProperty("auto.offset.reset", "latest")
    setProperty("enable.auto.commit", "true")
    setProperty("session.timeout.ms", "30000")
    setProperty("max.partition.fetch.bytes", "2048576")
    setProperty("heartbeat.interval.ms", "10000")
    setProperty("request.timeout.ms", "40000")

It is using the same consumer group on the same topic in the application. And by using Spring's @PreDestory, close the Kafka Consumer when the application ends, and close the ThreadPool that runs the consumer.

@PostConstruct
fun start() {
    threadPool.execute {
        // start consumer
        ...
    }
}

@PreDestroy
fun stop() {
    consumer.close()
    threadPool.shutdown()
}

When the application is closed and restarted, it was expected that data would be fetched from the last committed offset, but it is often fetched from latest.

Maybe I've missed something?

0 Answers
Related