Is it possible to force the cleaner to compact a partition log for partitions with very low traffic?

user_2155948 0
Viewed 282

Is it possible to force the cleaner to compact a partition log for partitions with very low traffic?

For a topic with retention.policy set to "compact, delete" it is understood that compaction (and for null records - deletion) takes place when a cleaner thread decides to clean a log. This decision is based on a few things. Relevant to this question is the segment roll functionality; unless a new segment is created compaction won't run.

Segment roll may be configured via segment.ms and segment.bytes.

Now, for the question. Since active segments won't be cleaned and a new segment doesn't become active until a record has been written to it, is it possible to force the cleaner to compact a partition log for topics that no longer receive any write-traffic?

Example log:

$ kcat -b kafka:9092 -t foo -C -K:
1:hello
2:world
1:

Regardless of configuration, unless a records is written after 1:null compaction does not run. However:

$ echo "3:compact" | kcat -b kafka:9092 -t foo -P -K:
# `segment.ms` time passes
$ kcat -b kafka:9092 -t foo -C -K:
2:world
1:
3:compact
apache-kafka
2 Answers

I've investigated further over at the Confluent Community and got it verified that a segment rolls when the timestamp of a written record differs more than segment.ms from the first record in the currently active segment.

In other words, a segment never rolls unless a record is written. So, the answer to the question is "no, it is not possible".

user_21559480

If log.cleaner.backoff.ms has the default value (15000) the cleanup thread(s) should run every 15 seconds. Of course if there is nothing to cleanup nothing is done. The tombstone messages (null messages) are included in the cleanup process. If compact and delete policies are both enabled, the delete policy should follow the log.retention.* parameters. Of course no active segment can be deleted. Anyway pay attention, because the cleanup thread(s) work on the full partition segment by segment, and when cleanup is completed for one segment the same one is replaced by the cleaned one. If the cleanup thread(s) does not have enough memory to cleanup a segment it is skipped, but you should see an error on kafka log. If traffic is low you should perhaps tune your segment closure policy using log.segment.bytes and log.segment.ms.

user_70568470
Related