Can I break anything if I set log.segment.delete.delay.ms to 0

user_401059 0
Viewed 362

I'm trying to put an absolute limit to the size of a topic so it won't fill up disks in case of unexpectedly high write speeds.

Mostly, this amounts to some calculation like segment=min(1GB, 0.1 * $max_space / $partitions) and setting retention.bytes=($max_space / $partitions - $segment),segment.bytes=$segment and reducing log.retention.check.interval.ms. (You can even pull stunts like determining your maximum write rate (disk or network bound) and then subtract a safety margin based on that and log.retention.check.interval.ms, and maybe some absolute offset of the 10MB head index file.) That's all been discussed in other questions, I'm not asking about any of that.

I'm wondering what log.segment.delete.delay.ms is for, and if I can safely set it to 0 (or some very small value - 5?). The documentation just states "The amount of time to wait before deleting a file from the filesystem". I'd like to know why you would do that, i.e. what or who benefits from this amount of time being non-0.

kafka-consumer-api apache-kafka
1 Answers

This configuration can be used to ensure that a Consumer of your topic has actually the time to consume the message before it gets deleted.

This configuration declares the delay that is applied when the LogCleaner is triggered by the time-based or size-based cleanup policies set through log.retention.bytes or log.retention.ms.

Especially when your have calculated the retention based on the size of data, this configuration can help you set a minimal time before a message gets deleted and to allow the consumer to actually consume the message during that time.

user_122089100
Related