One of my Kafka brokers (ID: 2011 below) is failing to initialize due to the following error:
FATAL [ReplicaFetcherThread-0-2006]: Exiting because log truncation is not allowed for partition REDACTED, current leader 2006's latest offset 0 is less than replica 2011's latest offset 170 (kafka.server.ReplicaFetcherThread)
It seems as though broker 2006 got the idea that it should become the leader of this partition with offset 0 and now 2011 doesn't know what to do with its data up to offset 170.
What are possible scenarios where this could have happened? "Unclean leader election" is disabled (by default), but is this what I would expect to see in the case of an unclean leader election?
One way to get the broker back online is to manually delete the partition from the affected broker. But, is there a way to recover the data from that partition?