Docker on AWS ElasticBeanstalk: "host.docker.internal: host-gateway": Connection refused. Iptables issue?

Viewed 1858

So I have a multicontainer project running on AWS ElasticBeanstalk (latest Docker platform).

The containers share the same bridge network since they need to be able to communicate with each other (ports 3000, 8080, 9090, 9100). But one of those containers needs to be able to communicate with the host too (ports 55113, 56113).

This is normally possible by just adding extra_hosts: "host.docker.internal:host-gateway" to that container and then using host.docker.internal instead of localhost inside the container, i.e. host.docker.internal:55113. I've successfully used it before on a Debian.

But this is not working on AWS ElasticBeanstalk, the connection is refused.

Am I doing something wrong? Is there any workaround?


AWS ElasticBeanstalk Platform:

Docker running on 64bit Amazon Linux 2/3.4.2

Host's Docker & Docker-Compose versions:

$ sudo docker --version

Docker version 20.10.4, build d3cb89e

$ sudo docker-compose --version

docker-compose version 1.29.2, build unknown

Project's docker-compose.yml: (showing only the relevant parts)

version: "3.9"

services:
  prometheus:
    image: prom/prometheus:v2.28.1
    container_name: prometheus
    ports:
      - 9090:9090
    extra_hosts:
      - host.docker.internal:host-gateway # <-------- EXTRA HOST

networks:
  default:
    name: monitoring

Container debug:

$ sudo docker exec -it prometheus cat /etc/hosts

127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.1  host.docker.internal # <-------- the mapping is present
172.25.0.4  f01ffc6b2a8a

$ sudo docker exec -it prometheus wget host.docker.internal

Connecting to host.docker.internal (172.17.0.1:80)
wget: can't connect to remote host (172.17.0.1): Connection refused

$ sudo docker exec -it prometheus wget host.docker.internal:55113  # <-------- the port I want to reach

Connecting to host.docker.internal:55113 (172.17.0.1:55113)
wget: can't connect to remote host (172.17.0.1): Connection refused

Host's iptables (configured by AWS and/or Docker, I didn't edit it):

$ sudo iptables -S

-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-a8a58f19c627 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a8a58f19c627 -j DOCKER
-A FORWARD -i br-a8a58f19c627 ! -o br-a8a58f19c627 -j ACCEPT
-A FORWARD -i br-a8a58f19c627 -o br-a8a58f19c627 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.25.0.2/32 ! -i br-a8a58f19c627 -o br-a8a58f19c627 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.25.0.3/32 ! -i br-a8a58f19c627 -o br-a8a58f19c627 -p tcp -m tcp --dport 9100 -j ACCEPT
-A DOCKER -d 172.25.0.4/32 ! -i br-a8a58f19c627 -o br-a8a58f19c627 -p tcp -m tcp --dport 9090 -j ACCEPT
-A DOCKER -d 172.25.0.5/32 ! -i br-a8a58f19c627 -o br-a8a58f19c627 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-a8a58f19c627 ! -o br-a8a58f19c627 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-a8a58f19c627 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

Host's open ports:

$ netstat -tuplen

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 0.0.0.0:9090            0.0.0.0:*               LISTEN      0          200315     -
tcp        0      0 127.0.0.1:55113         0.0.0.0:*               LISTEN      1000       119886     -
tcp        0      0 0.0.0.0:9100            0.0.0.0:*               LISTEN      0          200280     -
tcp        0      0 127.0.0.1:22221         0.0.0.0:*               LISTEN      1001       197452     -
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      0          14121      -
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      0          201081     -
tcp        0      0 127.0.0.1:56113         0.0.0.0:*               LISTEN      1000       119890     -
tcp        0      0 127.0.0.1:38581         0.0.0.0:*               LISTEN      0          20325      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          18585      -
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      0          201164     -
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          16235      -
tcp6       0      0 ::1:55113               :::*                    LISTEN      1000       119885     -
tcp6       0      0 :::111                  :::*                    LISTEN      0          14124      -
tcp6       0      0 ::1:56113               :::*                    LISTEN      1000       119889     -
tcp6       0      0 :::22                   :::*                    LISTEN      0          18594      -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           0          15762      -
udp        0      0 0.0.0.0:111             0.0.0.0:*                           0          14119      -
udp        0      0 127.0.0.1:323           0.0.0.0:*                           0          14990      -
udp        0      0 0.0.0.0:979             0.0.0.0:*                           0          14120      -
udp6       0      0 fe80::ed:19ff:fe84::546 :::*                                0          16539      -
udp6       0      0 :::111                  :::*                                0          14122      -
udp6       0      0 ::1:323                 :::*                                0          14991      -
udp6       0      0 :::979                  :::*                                0          14123      -
1 Answers

I'm new in the field.

Your logs shows that your port is listening on localhost (127.0.0.1:56113) instead of all IPs (eg 0.0.0.0:8080).

In my case the port forward rule on the host machine has to be modified to -L0.0.0.0:<lport>:<ip>:<rport> .

Related