Kafka Consumer RegExp: stale consumers

user_9167731 0
Viewed 189

I subscribe to Kafka Topics by RegExp. There are 5 topics matching the given pattern. Four of them has 10 partitions. And the last topic has 12 partitions. So, there are 52 partitions at all.

I have 5 running instances and each of them starts 10 consumers. So, there are 50 consumers at all. I expect that the load is spread horizontally across all the consumers. So, each consumer reads 1 partition except two of them because total number of consumers is less than the whole partitions count.

Though the reality is a bit different.

A single consumer reads multiple partitions

Most consumers remains stale

In short, here is what happens

  1. There are 50 consumers subscribing to multiple topics by RegExp. All found topics has 52 partitions totally.
  2. Each consumer tries to subscribe to 1 partition from each found topic. There are also two consumer subscribed to 1 partition from the single topic because the latter has 12 but not 10 partitions.
  3. 12 consumers are working whilst 38 remains stale due to unavailable partitions to read.

Is there any way to force Kafka consumer to read 1 partition maximum with RegExp subscription? In this case, I can make all consumers start work. Or maybe there is a different approach that allows to read multiple topics respecting the number of partitions and running consumers?

kafka-consumer-api apache-kafka architecture java performance
1 Answers

Well, it's about partition.assignment.strategy Kafka property. The default value is RangeAssignor which leads to assigning partitions on topic basis. That leads to spreading load between consumers unfairly.

We set the property to RoundRobinAssignor and it helped.

enter image description here

Though you should be careful when you deploy new version with different partition.assignment.strategy. Suppose you have 5 running consumers with RangeAssignor strategy. Then you redeploy one with RoundRobinAssignor strategy. In this case, you get an exception on consumer replacement. Because all consumers in one consumer group should provide the same partition.assignment.strategy. The problem is described in this StackOverflow question

So, if you want to change the partition.assignment.strategy, you have several options:

  1. Redeploy all consumers and then deploy new ones.
  2. Specify new group.id for the new consumers.

Both of these ways have pros and cons.

user_91677310
Related