Accessing Kafka broker from outside my LAN

Question

I start-up a plain Zookeeper/Kafka broker using the default commands below as these are described in the Kafka documentation in one machine (let's call it Machine A)

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

These start a broker at localhost:9092 in Machine A.

I then go to another computer (let's name it Machine B) which is on the same network and create a default consumer provided by Kafka by calling this

bin/kafka-console-consumer.sh --bootstrap-server IP_ADDRESS_HERE:9092 --topic test --from-beginning

Where IP_ADDRESS_HERE is the IP address in the local network of Machine A that hosts the broker (e.g., 192.168.1.10)

Everything works fine. Then I try to access the broker from a machine outside of my local network (let's call it Machine C). I go to my router configuration and I do a port forwarding for the machine that hosts the broker (Machine A). For example I do the following

192.168.1.10:9092 --> 9094

Meaning that I forward the internal 9092 port of the 192.168.1.10 device (Machine A) to port 9094 of my router. I then go to a service like the YouGetSignal and check if port 9094 of my public IP address (e.g., 97.190.92.128) is open. And I get a message that indeed the port is open.

When I try to consume the borker in Machine A from Machine C using the following command

bin/kafka-console-consumer.sh --bootstrap-server PUBLIC_IP_ADDRESS_HERE:9094 --topic test --from-beginning

Where the PUBLIC_IP_ADDRESS_HERE is the public IP address of the network where Machine A is in (e.g., 97.190.92.128). However, I get an error that cannot connect to 192.168.1.10:9092 - broker might not be available (notice that the internal IP:port is returned in the error which means that my router provides the info correctly)

What am I doing wrong?

user_350613 · Answer

You need to update advertised.listeners in your server.properties to give a host/IP and port combination that your client can resolve and connect to.

As you've observed, the broker gives out 192.168.1.10:9092 in response to a client connecting to it, which will work fine for any machine on the same network (including Machine B).

For Machine C the broker needs to tell it how to connect to the broker, which if I have followed correctly is PUBLIC_IP_ADDRESS_HERE:9094.

If you need to connect from both your LAN and WAN, you'll need two listeners; one for LAN and one for WAN. In server.properties put:

listeners=LISTENER_LAN://0.0.0.0:9092,LISTENER_WAN://0.0.0.0:9094
advertised.listeners=LISTENER_LAN://192.168.1.10:9092,LISTENER_WAN://PUBLIC_IP_ADDRESS_HERE:9094
listener.security.protocol.map: LISTENER_LAN:PLAINTEXT,LISTENER_WAN:PLAINTEXT
inter.broker.listener.name=LISTENER_LAN

Note that you need to change your port forwarding, so that the endpoint on your broker is the port as defined for LISTENER_WAN (9094). If a WAN connection tries to connect to the broker on 9092 then it will be given the LISTEN_LAN details (which is what's happening at the moment, and won't work).

Ref: https://rmoff.net/2018/08/02/kafka-listeners-explained/

Accessing Kafka broker from outside my LAN

1 Answers