We are in the process of designing a Kafka Cluster (at least 3 nodes) that will process events from an array of web servers. Since the logs are largely identical, we are planning to create a single Topic only (say - webevents)
We expect a lot of traffic from the servers. Since there is a single topic, there will be a single leader broker. In such a case how will the cluster balance the high traffic? All write requests will always be routed to the leader broker at all times and other nodes might be underutilized.
Does a external hardware balancer help solve this problem? Alternately, can a Kafka configuration help distribute write requests evenly on a 1-topic cluster?
Thanks, Sharod