Does Kafka work with load balancers using reverse proxies?

user_231934 0
Viewed 786

Recently I found some articles/projects using reverse proxy load balancers in front of Kafka (ie. https://github.com/banzaicloud/kafka-operator).

Until today, I though I understood Kafka principles and architecture well, because I thought that Kafka uses client-side balancing and consistent hashing so every client knows which partition is mastered on which broker and communicated with appropriate broker directly. Kafka even has protocol for notifying producers and consumers about repartitioning and other topology changes, so I really though that Kafka is supposed to be used without any load balancer. This has been even proven to me by receiving exceptions about producing to invalid broker/partition when we experienced issues during operations.

So what's the meaning of those proxies and load balancers in front of Kafka?

apache-kafka load-balancing
1 Answers

You are not wrong! All the things that you've mentioned in there are correct.

If we are talking about a "classic" Load Balancer you would need to meet the following 2 conditions in order to use it with Kafka:

  1. Load Balancers are at the TCP level ( can't use L6 or L7 Load Balancers with Kafka )
  2. One Load Balancer per Kafka Broker ( just as you've mentioned, clients connect directly to the Broker that have business with )

The articles that you are mentioning are probably somehow related to Envoy Kafka Filter ( including Banzai ).

I don't know the internal details but I think I can make a summary. The main challenge with dynamic routing is the Kafka Protocol which is a TCP level protocol. So, we don't have access to metadata as we would have in the case of a higher-level protocol ( e.g: HTTP ) so that we can properly route the communication.

Envoy developed a Kafka Filter which allows just that. When a client connects to the proxy, the proxy can decode the Kafka protocol and it knows "ok, so you want to connect to x broker, let me do that for you".

user_114147450
Related