How to implement Time To Live (TTL) for a Kafka topic

user_15459894 0
Viewed 1204

I want to have a topic deleted after some predefined time of inactivity.

To give you some context, there's a microservice that has many replicas, and each replica has its own topic to communicate, identified by its replica Id (e.g. topic_microservice-name_<random_id>). If for any reason, a replica crashes, K8s will start another Pod, with a completely different replica Id, therefore the previous topic will not be used anymore. For this reason, after some time there could be many useless topics.

Does kafka have a built-in Time To Live for the whole topic?

Another idea I have is to have a Quartz Job iterating all topics somehow getting the last modified/written date and checking if the TTL expired.

apache-kafka
2 Answers

There currently isn't a way to give a topic a TTL, where once the TTL expires Kafka automatically deletes the topic.

One can configure retention on the topic level (retention.ms - how long messages should be retained for this topic or retention.bytes - the amount of messages to retain in bytes). With this, you could have a separate service leveraging the AdminClient to execute scheduled operations on your topics. The logic could simply be iterating over the topics, filtering out the active topics, and deleting each topic that has been inactive long enough for the retention strategy to take effect.

user_144750340

The original question as to whether kafka topic actually has a TTL has already been answered (which is NO as of writing this answer). This answer deals with several ways to handle deletion of topics w.r.t your scenario.

  1. Write a container preStop hook

    Where you can execute the topic's deletion code upon a pod termination. This could be simple approach. The hook implementations include exec command (or) a HTTP call.

    You can for example, include a small wrapper script on top of kafka-topics.sh (or) a simple python script that could connect to the broker and delete the topic.

    You might also want to make a note of terminationGracePeriodSeconds and increase it accordingly if your topic deletion script takes longer than this value.

  1. Get notified using Kubernetes Watch APIs

    You may need to write a client that listens to the events and use the AdminClient to delete the topics corresponding to the terminated pod. This typically needs to be separated from the terminated pod.

  1. Find out what topics needs to be deleted by getting list of active pods.
    1. Retrieve the pod replicas available in the Kubernetes cluster using Kubernetes API.
    2. Iterate over all the topics and delete those which do not conform to the above retrieved list.

P.S:

Note that the deletion of topics is an administrative task and it is typically done manually after some verification checks.

Creation of a lot of topics isn't recommended as maintenance would be difficult. If your applications are creating a lot of topics, for eg, as many as the number of workload instances running, then it might be the time to rethink your application design.

user_25340900
Related