I need a Kafka topic that has multiple java consumers all in the same group. One requirement is that we must be able to reduce the throughput of a specific consumers. Now, due to the facts of how kafka works as I understand it, simply just introducing sleeping between reads would cause messages partitioned to the consumer to pile up, which is something we do not want. Throttling of consumers should preferably be able to be done at will without restarting neither consumers (apart from the one being throttled) or producers.
After some research I have only been able to find two methods for doing this:
- Simply disconnect the consumer for some time after some amount of consumption to retain static throughput. This would cause the broker to reassign its partitions to other consumers.
- Create a custom partitioner for the producers, which would look at partition metadata and prioritize those that are not throttled. Looking at the Java Kafka API for this, it does not seem to support custom metadata, so this data would have to come from elsewhere.
Is this all there is? I feel that throttling consumers would be a more common usecase, but I have been unable to find anything that satisfy our requirements. Please answer if you know of better solutions. Thank you.