Changing the retention period of kafka during run time

user_15272074 0
Viewed 400

HI I want to reduce the retention period of the Kafka.

so , How to reduce the retention during the run time and so as to not required to restart the Kafka service.

Note: I want to do retention at global level of Kafka and not topic level.

retention apache-kafka
1 Answers

As Kafka documentation described in Broker configs section under the following sub topic

Updating Default Topic Configuration

Default topic configuration options used by brokers may be updated without broker restart. The configs are applied to topics without a topic config override for the equivalent per-topic config. One or more of these configs may be overridden at cluster-default level used by all brokers.

You can change dynamically Kafka topic default configs in cluster level. For retention period you can change below configs.

  • log.retention.ms
  • log.retention.minutes
  • log.retention.hours

You can see other config list in the documentation.

But again according to documentation log.retention.hours and enter link description here configurations' update mode in read-only and log.retention.ms is cluster-wide

So as stated in 3.1.1 Updating Broker Configs

From Kafka version 1.1 onwards, some of the broker configs can be updated without restarting the broker. See the Dynamic Update Mode column in Broker Configs for the update mode of each broker config.

  • read-only: Requires a broker restart for update
  • per-broker: May be updated dynamically for each broker
  • cluster-wide: May be updated dynamically as a cluster-wide default. May also be updated as a per-broker value for testing.

So only you can change log.retention.ms

config updating command for all brokers

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config log.retention.ms=3600000

output

Completed updating default config for brokers in the cluster.

to verify if the config upodated in the cluster level, run following describe command

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --describe

output

Default configs for brokers in the cluster are:
  log.retention.ms=3600000 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.retention.ms=3600000}

If you need to remove or reset the config again, run

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --delete-config log.retention.ms
user_108667980
Related