Recommended settings for Kafka Internal Topics after upgrade to 1.0

user_785523 0
Viewed 6830

I am migrating from Kafka 0.10 to the latest 1.0 . I did not set any value for these fields in Kafka 0.10. Can some one let me know what is the recommended settings for a 3 node broker cluster & for a single node broker cluster for the below internal topic settings in server.properties -

  • offsets.topic.replication.factor
  • transaction.state.log.replication.factor
  • transaction.state.log.min.isr

If I am not changing this value after upgrade to 1.0 in a single node kafka broker it will take the default value of 3 . So what will be the behavior in this case?

apache-kafka
2 Answers

  • For clusters with 3 or more brokers, the default values for these settings are good for most use cases:

    • offsets.topic.replication.factor=3
    • transaction.state.log.replication.factor=3
    • transaction.state.log.min.isr=2

    These default values provide strong availability and durability guarantees as data will be replicated across 3 brokers. They also allow to have 1 broker down and still be able to produce/consume with and without transactions.

  • For clusters with a single node you don't have the choice, these settings can't be higher than the number of brokers in the cluster, so they must all be set to 1.

user_17651890

All three parameters enforce cluster high availability and redundancy of transactions and topic replications.

offsets.topic.replication.factor parameter were also in 0.10.X. For 3 node cluster I suggest to leave the default, but for the single node (I suppose you use it for dev and test purposes) this have to be minimized to 1.

For the transaction.log parameters they are not exactly the same in 2 versions of Kafka. See this. But the concept is the same... minimize the default if you want to test and dev on the single node cluster. otherwise on 3 node cluster, dependtly on how much traffic and consumer/producers connected select. Transaction logs obviously could occupy a lot of space, proportionally to how many request (to add a message or read a message) you have on the cluster.

See this. On the 0.10X kafka server has this paramater optional when you create a topic, in the new version, maybe something is changed for the 0.11 In the upgrading notes of the 1.0X: "config/consumer.properties file updated to use new consumer config properties."

user_89329980
Related