For some time we are having an issue with the ever-growing partition of __consumer_offsets topic (currently, this single partition is 800 GB, while other ones are a magnitude smaller). It keeps all of the segments and the compaction doesn't seem to be working for this single partition.
The issue with a growing partition of __consumer_offsets topic is mentioned in a couple of questions over the internet, however, most of the solution mention updating Kafka. Unfortunately, that is not an option for us at the moment.
In logs we see:
INFO The cleaning for partition __consumer_offsets-12 is aborted and paused (kafka.log.LogCleaner)
INFO The cleaning for partition __consumer_offsets-12 is aborted (kafka.log.LogCleaner)
and since this moment segments are being kept. We tried to restart brokers which are handling the problematic partition, unfortunately after
INFO Starting the log cleaner (kafka.log.LogCleaner)
log entry we see that log cleaner starts for all of the partitions except the __consumer_offsets-12.
My question is:
- Is it possible to somehow restart log cleaner for a particular partition?
- Is it possible and safe to manually remove segment log files from ".../kafka-logs/__consumer_offsets-12" directory?
Those may be useful:
Kafka version: 2.5.1
log.retention.check.interval.ms=300000
offsets.retention.minutes=43200
log.cleanup.interval.mins=15
log.retention.hours=168
log.retention.bytes=-1
log.segment.bytes=1073741824