I have 3 kafka topics containing fairly similar types of data. The data is in 3 topics because it is generated by 3 different sources, and some consumers prefer to process only a subset of the 3 topics. In addition, the topics have fairly different scale: one has 200 partitions, another has 10, and the last has 5.
I then have a stateless kafka streams application that is consuming from all 3 topics, and more-or-less performing identical processing on all 3. For almost all messages, it ends up making a network call to an external service, so it is mostly network-bound. I'm running this application on 25 hosts.
When tuning the performance of this application, we found that increasing thread count was the most effective way to improve resource utilization and throghput, because it allowed parallel processing of all partitions assign to a host. We tried 8, so each host would get assigned ~25 partitions from the biggest topic, and that worked fairly well. But of course, there were still those extra 15 partitions from the other two topics, which caused some hosts to have 9 partitions assigned to it. In fact, in some cases, a single host would get assigned 2 or more of those 15 partitions (on different threads), meaning that a host could have 10+ partitions and only 8 threads, leaving some parallelism on the table.
If we increase the number of threads (e.g. to 12), things get worse. In fact, more often than not, one host would get ALL 15 (10 + 5) of those smaller topics, plus 12 partitions from the larger topic (for a total 27 partitions). Several other hosts would get 0 partitions in this scenario.
So... What's the best way to evenly distribute partitions across hosts, but ensure that each partition is getting its own thread (and therefore is able to run/process in parallel with other partitions, which each is doing slow, network-bound work)? Or maybe threads aren't it, and I'm missing something fundamental about how to maximize throughput in this scenario...