How to view and set offsets.retention.minutes using kafka-configs

user_2727270 0
Viewed 3584

From the documentation, this is a broker config, but details are thin.

1) What is the correct way to set this to indefinite? If I do, will I still be able to delete consumer groups and their associated offsets manually?

2) Can I set individual consumer groups to have differing retention?

3) How do I view the retention globally, or for a specific consumer group? Can I view this and get a number even if I have not set this before?

apache-kafka
1 Answers

1) Kafka Docs mention that the valid values for offsets.retention.minutes are

[1,...]

which means that you cannot set offsets.retention.minutes=-1 but you can set it to a fairly large integer in order to keep the offsets for a long period.

2) I would assume that you are asking whether you can set different values for offsets.retention.minutes on a topic level; As far as I know, this is not possible. You can only modify this configuration on a broker level, i.e. inside server.properties file.

EDIT: I am afraid that you cannot avoid a rolling restart as you cannot change offsets.retention.minutes using kafka-configs.sh; 

> bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type brokers --entity-name 0 --alter --add-config offsets.retention.minutes=200800

Error while executing config command requirement failed: Unknown Dynamic Configuration 'retention.minutes'.
java.lang.IllegalArgumentException: requirement failed: Unknown Dynamic Configuration 'retention.minutes'.
        at scala.Predef$.require(Predef.scala:224)
        at kafka.server.DynamicConfig$$anonfun$kafka$server$DynamicConfig$$validate$1.apply(DynamicConfig.scala:101)
        at kafka.server.DynamicConfig$$anonfun$kafka$server$DynamicConfig$$validate$1.apply(DynamicConfig.scala:100)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at kafka.server.DynamicConfig$.kafka$server$DynamicConfig$$validate(DynamicConfig.scala:100)
        at kafka.server.DynamicConfig$Broker$.validate(DynamicConfig.scala:59)
        at kafka.admin.AdminUtils$.changeBrokerConfig(AdminUtils.scala:555)
        at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:105)
        at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:68)
        at kafka.admin.ConfigCommand.main(ConfigCommand.scala)

3) Since this is a configuration on the broker level, you can view the value in server.properties file or from your broker's log files. For example, 

 grep offsets.retention.minutes /path/to/your/kafka-broker.log

Note that even if you haven't configured offsets.retention.minutes, its value defaults to 10080.

user_71317570
Related