Kafka - set min ISR based on topics' prefix

user_8447910 0
Viewed 1015

I have following problem to solve. Do You Guys know mechanism that would allow set different min ISR value based on topic prefix? For example topics created with prefix app-* would have min ISR factor value set to 2, logs-* equal to 1 but value for core-* would be 4.

Solution proposal

min.insync.replicas.per.topic=topic_name_1:value,topic_name_2:value looks almost as a solution but unfortunately it is shadowed when min.insync.replicas is also present in kafka.properties. So I will explore if there is solution to combine those two props. source

Verifying min ISR value using

./kafka-configs.sh --bootstrap-server localhost:9092 \
 --entity-type topics --entity-name $TOPIC --describe --all
apache-kafka
2 Answers

Edit: In order to avoid 80's style solutions (such as shown in this answer), please take a look at Dirk's answer.

I just love bash/sh

For already created topics, you'd have to execute a partition reassignment operation.

If you wish to control the ISR for newly created topics, you could wrap the kafka-topics.sh script in a new script, lets call it createTopics.sh:

#!/bin/bash
isr=0

if [[ $1 = \logs* ]]; then
        isr=1
elif  [[ $1 = \app* ]]; then
        isr=2
elif  [[ $1 = \core* ]]; then
        isr=4
fi

/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                     --partitions $3  --bootstrap-server $2
  • $1 - topic name
  • $2 - bootstrap-server
  • $3 - partitions

For example:

  ./createTopics.sh apps-topic localhost:9092 1 && \
  ./createTopics.sh core-topic localhost:9092 1

This would create two new topics with their respective replication-factor:

  • apps-topic > ISR = 2.
  • core-topic > ISR = 4.

(max ISR value, assuming all replicas are in-sync)

The script would also validate the creation of new topics, as it wouldn't create any topic which name doesn't start with one of these: logs / app / core. Kafka would show an error, Replication factor must be larger than 0.

You could change this behaviour in order to allow different topic prefixes, by setting the default isr value at the start of the script different than 0: isr=1, for example. This would lead to:

#!/bin/bash
isr=1

if  [[ $1 = \app* ]]; then
       isr=2
elif  [[ $1 = \core* ]]; then
       isr=4
fi

/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                     --partitions $3  --bootstrap-server $2
user_21489530

You can also use kafkactl for this:

# first run with --validate-only to see what kafkactl will do
kafkactl get topics -o compact | grep '^app.*'  | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 2 --validate-only" | sh
kafkactl get topics -o compact | grep '^log.*'  | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 1 --validate-only" | sh
kafkactl get topics -o compact | grep '^core.*' | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 4 --validate-only" | sh

# then do the replica reassignment
kafkactl get topics -o compact | grep '^app.*'  | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 2" | sh
kafkactl get topics -o compact | grep '^log.*'  | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 1" | sh
kafkactl get topics -o compact | grep '^core.*' | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 4" | sh

Note that the Kafka API that kafkactl is using for this is only available for Kafka ≥ 2.4.0.

Disclaimer: I am contributor to this project

user_11152790
Related