Scaling out with 200+ Kafka topics

user_3929917 0
Viewed 708

I'm trying to understand how to dynamically scale out application which consumes a huge number of topics (unfortunately I can't reduce their number - by design each topic is for particular type of data).

enter image description here

I want my application cluster to share the load from all 200+ topics. E.g when a new app node added to the cluster, it should "steal" some topics subscriptions from old nodes, so the load become evenly distributed again.

As far as I understand, Kafka partinions/consumer groups help to parallelize a topic, not to share a load between multiple topics.

apache-kafka distributed-system message-queue
2 Answers

You need to make sure that all your App instances use the same Kafka Consumer Group (via group.id). In this case you actually have an even distribution you want. When a new App instance is added, consumer group is going to rebalance and make sure the load is distributed.

Also, when a new topic/partition is created it'll take consumer up to "metadata.max.age.ms" (default is 5 minutes) to start consuming from it. Make sure to set "auto.offset.reset" to "earliest" to not miss any data.

Finally, you might want to use a regex to subscribe to all those topics (if possible).

user_8999370
Related