Can't connect to a container in the same host even though the DNS is being resolved

user_11482776 0
Viewed 697

I'm trying to make a request from a container to another container on the same host machine. It turns out, the request is declined and I am unable to get the information I would like. To illustrate, I made a small diagram:

Diagram1

On "machine 1" I have 2 services running on docker-compose: App1 has 2 services and needs to make the request for App2. App2 is the service that will receive the request, has only 1 service and listens on port 4202.

docker-compose.yaml for app1

version: '3.9'
    services:
            main_app:
                    build: .
                    depends_on:
                            - db_app
            db_app:
                    restart: always
                    image: redis

docker-compose.yaml for app2

version: '3.9'
    services:
            main_app:
                    build: .
                    ports:
                       - "4202:5000"

Connection doesn't work. Which is very strange, as other connections work, such as:

Diagram2

In this image, on another machine machine 2 I use curl to try to reach the container that is running inside machine 1. On port 4202, I make a request for a domain name.domain.example that is on the internal network and it is resolved by internal DNS and after that, it does successfully requested the container on machine 1.

Diagram3

In this other case, I create a container inside machine 2 and use curl to make a request to name.domain.example and the result is the same: success. I can reach the container inside the machine 1.

Diagram4

Here, I use curl on machine 1 (the same machine that is running the service that will be requested). The request is made to name.domain.example and the service is successfully reached.

Diagram5

In this case, this is where the request fails. This time, I try to make the request from within the app1 container, the request is made to name.domain.example and the DNS is able to resolve it, but when the request is made, it ends up being refused.

Inside app1, the request I try to make is through python, from the requests library:

import requests

r = requests.get("name.domain.example:4202")
# But, Exception...:
requests.exceptions.ConnectionError: HTTPConnectionPool(host='name.domain.example', port=4202): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f74fbcb1100>: Failed to establish a new connection: [Errno 111] Connection refused'))

I tried to put app1 on the network "host" --network=host or on the docker-composenetwork_mode="host"and I can reach the container of app2. However, this is not useful for my case since I have other services from the compose stack that I need, in addition to the fact that as the services increase and depending more on each other, this ends up becoming a spider web.

docker-network microservices docker-compose docker
1 Answers

The trick here is that docker compose is creating docker networks (or network namespaces) for each of your applications. Try running docker network ls and I bet you'll see something like this (the exact names will differ):

❯ docker network ls
NETWORK ID     NAME                        DRIVER    SCOPE
c6a3be245bd1   bridge                      bridge    local
f8e91845fd46   myapp2_default              bridge    local
677668cbb02b   myapp1_default              bridge    local

Neither of these 'bridge' networks is aware of the other, by default. When you specify --network=host or network_mode="host" you tell the containers to skip creating a docker/namespace network and to live within your host's main namespace. From here they can see everything that you can normally.

So let's teach these two compose networks that they should be aware of each other.

First, we create a docker network outside of the scope of docker-compose:

❯ docker network create myglobalnetwork
6adbd64eed5dd12a2e6d60dffe01b5c222ec4701535322fc17dfeb51aff3b801

We create it outside of the scope of docker-compose because we need to use it outside of any one docker-compose scope.

We can now update our docker-compose configurations to reference this network. I've used different images to simplify the example, but the idea remains the same:

# docker-compose.yml
version: '3.9'
services:
  main_app:
    image: busybox
    command: ping main_app
    depends_on:
      - db_app
    networks:
      - default
      - global
  db_app:
    restart: always
    image: redis
networks:
  global:
    external:
      name: myglobalnetwork

# docker-compose-other.yml
version: '3.9'
services:
  other_app:
    image: busybox
    command: ping main_app
    networks:
      - default
      - global
networks:
  global:
    external:
      name: myglobalnetwork

Then we can start the primary docker-compose project:

✖2 ❯ docker-compose -f docker-compose.yml up
Starting stackoverflow_db_app_1 ... done
Starting stackoverflow_main_app_1 ... done
Attaching to stackoverflow_db_app_1, stackoverflow_main_app_1
db_app_1    | 1:C 18 Mar 2021 00:00:41.103 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
db_app_1    | 1:C 18 Mar 2021 00:00:41.103 # Redis version=6.2.1, bits=64, commit=00000000, modified=0, pid=1, just started
db_app_1    | 1:C 18 Mar 2021 00:00:41.103 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
main_app_1  | PING main_app (172.28.0.2): 56 data bytes
main_app_1  | 64 bytes from 172.28.0.2: seq=0 ttl=64 time=0.028 ms
db_app_1    | 1:M 18 Mar 2021 00:00:41.104 * monotonic clock: POSIX clock_gettime
<snip some redis logs>
db_app_1    | 1:M 18 Mar 2021 00:00:41.105 * Ready to accept connections
main_app_1  | 64 bytes from 172.28.0.2: seq=1 ttl=64 time=0.067 ms
main_app_1  | 64 bytes from 172.28.0.2: seq=2 ttl=64 time=0.059 ms
main_app_1  | 64 bytes from 172.28.0.2: seq=3 ttl=64 time=0.120 ms

This one is able to ping itself, which is great but not that exciting or different.

The real magic is when we start the second one, and it tries to reference the first one.

❯ docker-compose -f docker-compose-other.yml up
Starting other_other_app_1 ... done
Attaching to other_other_app_1
other_app_1  | PING main_app (172.28.0.2): 56 data bytes
other_app_1  | 64 bytes from 172.28.0.2: seq=0 ttl=64 time=0.112 ms
other_app_1  | 64 bytes from 172.28.0.2: seq=1 ttl=64 time=0.112 ms

And that's how we teach docker containers about each other! There's some further reading you can do here, if you're interested: https://docs.docker.com/compose/networking/

user_26187640
Related