We have an issue with one of our kafka clusters (Kafka version 2.2.1). Some __consumer_offsets partitions are not getting cleaned and are in the 10s of GB in size (normal ones are 10s MBs in size).
The LogCleaner mentions this:
[2019-06-17 12:27:26,147] DEBUG Finding range of cleanable offsets for log=__consumer_offsets-13 topicPartition=__consumer_offsets-13. Last clean offset=Some(4546182967) now=1560774446127 => firstDirtyOffset=4546182967 firstUncleanableOffset=4737493473 activeSegment.baseOffset=5008571978 (kafka.log.LogCleanerManager$)
So according to the LogCleaner, offset 4737493473 should not be cleaned. This is a really old offset though so we looked further and found this in the latest *.snapshot files for that partition:
/usr/kafka/bin/kafka-dump-log.sh --files /data/__consumer_offsets-13/00000000005012457913.snapshot | grep 4737493473
producerId: 574024 producerEpoch: 24 coordinatorEpoch: 114 currentTxnFirstOffset: Some(4737493473) firstSequence: 0 lastSequence: 0 lastOffset: 4737493473 offsetDelta: 0 timestamp: 1558689306384
So that offset seems to be part of an ongoing transaction. My question is, what can we do to make the log cleaner move past this offset? Can we somehow manually close that transaction, delete it or something else?
Best Regards