log retention to implement message TTL

user_1804802 0
Viewed 251

we are considering to implement a timeout as part of a Kafka-based API by utilising its time based retention capabilities.

Basically, setting log.retention.ms = 10000 to make messages expire from a command topic if not processed within 10seconds.

I am wondering though whether this would provide a message level guarantee (i.e. every message is available the same amount of time) given that retention policies operate at the log segment level (based on largest timestamp per segment).

Of course, we can reduce log.segment.bytes to achieve more granular retention control, not sure though about the implications on performance.

any advice?

Nick

apache-kafka messaging
1 Answers

In Kafka, the retention settings are lower bounds, ie Kafka guarantees it will not delete a message before its retention limits are reached.

In practice, that means messages can stay in the log for longer than their retention limits.

Also as you said, Kafka operate at the log segment level. For time retention, only once the latest message in a segment gets older than the limit, this segment becomes eligible for deletion. And that does not apply to the active segment. So retention can't be used to provide per message time to live.

I don't know about your use case but maybe have a look at the offsetsForTimes() and seek() APIs in the consumer. These allow to select what the consumer will read based on time.

Finally, if you really need strong per message TTL, maybe Kafka is not the best tool.

user_17651890
Related