Kafka offset is reset after restart

Han Chris 0
Viewed 28

I'm using Apache Kafka 2.11-0 and deploy into container in GKE. I put 3 replicas for the HA and the message is consumed by Flink Job.

I'm using persistent volume in /var/lib/kafka
and this is my base docker image: kubernetes-kafka/kubernetes-kafka:1.0-10.2.1

I have 1 topic using 4 partitions
I notice that once there's 1 kafka node is restarted, the offset is reset to 0. I didn't overwrite the value of auto.offset.reset , which default should be latest.

Is there any reasons why the offset is reset ?

apache-flink apache-kafka google-kubernetes-engine kubernetes
1 Answers

Your topic only has one ReplicationFactor, so if Broker ID=1 goes down, then it's data will all be unavailable.

If you want to persist that topic data, increasing its replication factor is a good idea, but also sounds like your persistent volumes aren't working as expected

You'll also want to double check your offsets.topic.replication.factor value to ensure it's also not set to 1, otherwise, your consumers such as Flink will get reset when the broker hosting that topic loses its data.

OneCricketeer0
Related