Docker listening inside the docker host for RabbitMQ but not from outside, why?

Viewed 288

This is how I run the rabbitMQ image:

docker run -d --restart always --hostname host-rabbit --name cg-rabbit -p 5029:5672 -p 5020:15672 -e RABBITMQ_DEFAULT_VHOST=sample_vhost -e RABBITMQ_DEFAULT_USER=sampleuser -e RABBITMQ_DEFAULT_PASS=samplepass rabbitmq:3-management

Now in netstat -nltp:

ubuntu@infra:~$ netstat -nltp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 :::5020                 :::*                    LISTEN      -
tcp6       0      0 :::5029                 :::*                    LISTEN      -

I'm not sure why I see tcp6 when docker exposes ports to host and if it makes issues!

Now when I telnet from within the server I can see that port is open:

ubuntu@infra:~$ telnet MY-SERVER-IP-ADDRESS 5029
Trying MY-SERVER-IP-ADDRESS...
Connected to MY-SERVER-IP-ADDRESS.
Escape character is '^]'.
^]

telnet> Connection closed.

But in my machine when I try to telnet (or from another server):

$ telnet MY-SERVER-IP-ADDRESS 5020
Trying MY-SERVER-IP-ADDRESS...
^C

iptables -L reports:

sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:5020
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:5029
ACCEPT     tcp  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:amqp
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:15672

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

It is good to note that I have installed a redis server in server (non-docker) and I am able to telnet to it form outside.


EDIT-1:

sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !localhost/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        anywhere
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:15672
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:amqp

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
DNAT       tcp  --  anywhere             anywhere             tcp dpt:15672 to:172.17.0.2:15672
DNAT       tcp  --  anywhere             anywhere             tcp dpt:amqp to:172.17.0.2:5672

EDIT-2: Docker configuration:

ubuntu@infra:~$ sudo cat /var/snap/docker/796/config/daemon.json
{
    "log-level":        "error",
    "storage-driver":   "overlay2"
}
1 Answers

This is really odd. By flushing the NAT in iptables everything works as expected:

iptables -t nat -F

My nat before flushing:

ubuntu@infra:~$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !localhost/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        anywhere
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:15672
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:amqp

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
DNAT       tcp  --  anywhere             anywhere             tcp dpt:15672 to:172.17.0.2:15672
DNAT       tcp  --  anywhere             anywhere             tcp dpt:amqp to:172.17.0.2:5672

And now after flushing everything is gone:

ubuntu@infra:~$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (0 references)
target     prot opt source               destination

NOTE: by restarting docker via sudo snap restart docker net rules are back again and I had to flush NATs again!

Related