At first glance, the Kafka rules for data retention periods are simple, but on closer inspection they can be configured in a complex way. I think I now have a certain overview, but the details leave open questions. I think I have understood that:
- The
log.retention.*broker properties are only considered in case thelog.cleanup.policyincludes 'delete' (this condition is mention in the corresponding topic properties but not in the broker property section so I wonder if it is also valid for the general broker setting)? - In a topic with "
cleanup.policy=compact" a record with a unique key that never gets "updated" will never be removed from the log.
I now try to understand the files in the file system for a given topic configuration:
Topic: MyTopic PartitionCount:5 ReplicationFactor:2
Configs:
cleanup.policy=compact,
segment.bytes=1073741824,
retention.ms=86400000,
max.message.bytes=5242880,
message.timestamp.type=LogAppendTime,
unclean.leader.election.enable=false,
segment.ms=3600000
Topic: MyTopic Partition: 0 Leader: 9 Replicas: 9,7 Isr: 9,7
Topic: MyTopic Partition: 1 Leader: 7 Replicas: 7,8 Isr: 8,7
Topic: MyTopic Partition: 2 Leader: 8 Replicas: 8,9 Isr: 8,9
Topic: MyTopic Partition: 3 Leader: 9 Replicas: 9,8 Isr: 8,9
Topic: MyTopic Partition: 4 Leader: 9 Replicas: 7,9 Isr: 9,7
As the 'cleanup.policy' is only 'compact' I guess the 'retention.ms' setting is useless because the 'cleanup.policy' does not include 'delete'. The 'segment.ms' setting causes the log file to roll every hour.
At the moment I have difficulties to understand the log segment files in the filesystem
-rw-r--r--. 1 kafka kafka 2123 29. Jun 17:04 00000000000000000000.log
-rw-r--r--. 1 kafka kafka 57813 29. Jun 18:00 00000000000000017703.log
I think these are regular log segment files and '00000000000000017703.log' is the active segment file because it has the highest offset. The records in the 00000000000000000000.log file should be eligible for log compaction. And the size of this file should only shrink, right? Instead I see a quite constant file size (probably nothing to clean) which also once increased. Why?
-rw-r--r--. 1 kafka kafka 2123 29. Jun 15:03 00000000000000000000.log
-rw-r--r--. 1 kafka kafka 2123 29. Jun 17:04 00000000000000000000.log
-rw-r--r--. 1 kafka kafka 2433 30. Jun 15:10 00000000000000000000.log
-rw-r--r--. 1 kafka kafka 2123 30. Jun 16:10 00000000000000000000.log
And then there are numerous "snapshot" files between the first log and the active log segment file. All of them 10 bytes large.
-rw-r--r--. 1 kafka kafka 2123 29. Jun 17:04 00000000000000000000.log
-rw-r--r--. 1 kafka kafka 10 23. Jun 14:29 00000000000000000126.snapshot
-rw-r--r--. 1 kafka kafka 10 23. Jun 15:30 00000000000000000247.snapshot
-rw-r--r--. 1 kafka kafka 10 23. Jun 16:30 00000000000000000368.snapshot
-rw-r--r--. 1 kafka kafka 10 23. Jun 17:31 00000000000000000489.snapshot
-rw-r--r--. 1 kafka kafka 10 23. Jun 18:31 00000000000000000609.snapshot
...
-rw-r--r--. 1 kafka kafka 10 29. Jun 15:03 00000000000000017461.snapshot
-rw-r--r--. 1 kafka kafka 10 29. Jun 16:04 00000000000000017582.snapshot
-rw-r--r--. 1 kafka kafka 10485760 29. Jun 17:59 00000000000000017703.index
-rw-r--r--. 1 kafka kafka 57813 29. Jun 18:00 00000000000000017703.log
Obviously they represent the hourly roll over, but why 'snapshot'?
And if records in the '00000000000000000000.log' file will never be cleand up, will there be the full roll over history in snapshot files up to the time of the active file?
My questions in short:
- What are snapshot files? When are they created and when are they deleted?
- Do the offsets/filenames change during cleanup? Are segment files merged in a cleanup?
- What happens if records are not cleaned up? Will the subsequent segment files (which may already be empty) not be deleted?
- Do I understand correctly that records in topics with
retention.policy=compact(no delete) are never deleted until a new version of the record is written?
Thanks for helping me understand Kafka better