Docker swarm can't reach worker nodes

Question

So, i have set up a docker swarm and connected a worker on it and deployed a stack with 4 services:

generator, which will be located at the worker node
API & scheduler, both will be located on the manager node
proxy, that will be located on the manager node, accepting requests and redirecting it though the other 3

This is my stack file:

version: "3.7"
services:
  generator:
    image: musicorum/generator:latest
    restart: always
    environment:
      - 'XXXX=XXXX'
    deploy: 
      resources:
        reservations:
          memory: 860M
      placement: 
        constraints: 
          - "node.labels.generator==yes"
    ports:
      - 5000:5000
    networks:
      - proxy_ext
      - netg
    volumes:
      - type: bind
        source: /home/musicorum/cache
        target: /usr/src/app/cache
  api:
    image: musicorum/api:latest
    restart: always
    environment:
      - 'XXXX=XXXX'
    networks:
      - proxy_ext
    ports:
      - 4500:4500
    deploy: 
      placement: 
        constraints: 
          - "node.labels.generator!=yes"
  scheduler:
    image: musicorum/scheduler:latest
    restart: always
    environment:
      - 'XXXX=XXXX'
    ports:
      - 6500:6500
    networks:
      - proxy_ext
    deploy: 
      placement: 
        constraints: 
          - "node.labels.generator!=yes"
  proxy:
    image: nginx:latest
    restart: always
    networks:
      - proxy_ext
      - netg
    ports:
      - 80:80
      - 443:443
    configs:
      - source: nginx_4
        target: /etc/nginx/conf.d/default.conf
      - source: sslcrt
        target: /etc/ssl/musicorumapp/ssl.crt
      - source: sslkey
        target: /etc/ssl/musicorumapp/ssl.key
    depends_on:
      - scheduler
      - api
      - generator
    deploy: 
      placement: 
        constraints: 
          - "node.labels.generator!=yes"
configs:
  nginx_4:
    external: true
  sslcrt:
    external: true
  sslkey:
    external: true
networks:
  proxy_ext:
    external: true
  netg:
    driver: overlay
    attachable: true

As you can see, the they are connected on the same network, i even created proxy_ext and netg to double-check the connection, but Nginx give this message when start up:

/docker-entrypoint.sh: Configuration complete; ready for start up


2020/07/07 13:32:17 [emerg] 1#1: host not found in upstream "musicorum_generator" in /etc/nginx/conf.d/default.conf:30


nginx: [emerg] host not found in upstream "musicorum_generator" in /etc/nginx/conf.d/default.conf:30

I don't know why the Nginx, at the manager node, can't reach out to the generator container, at the worker node. If it helps, here's my default.conf:

server {
    listen 80;
    listen 443 ssl;
    ssl_certificate /etc/ssl/musicorumapp/ssl.crt;
    ssl_certificate_key /etc/ssl/musicorumapp/ssl.key;
    server_name api.musicorumapp.com;
    location / {
        proxy_pass http://musicorum_api:4500/;
    }
}

server {
    listen 80;
    listen 443 ssl;
    ssl_certificate /etc/ssl/musicorumapp/ssl.crt;
    ssl_certificate_key /etc/ssl/musicorumapp/ssl.key;
    server_name scheduler.musicorumapp.com;
    location / {
        proxy_pass http://musicorum_scheduler:6500/;
    }
}

server {
    listen 80;
    listen 443 ssl;
    ssl_certificate /etc/ssl/musicorumapp/ssl.crt;
    ssl_certificate_key /etc/ssl/musicorumapp/ssl.key;
    server_name generator.musicorumapp.com;
    location / {
        proxy_pass http://musicorum_generator:5000/;
    }
}

user_5171751 · Answer

In your default.conf you need to reference the services by their service name. This is the name that the internal DNS will resolve.

server {
    listen 80;
    listen 443 ssl;
    ssl_certificate /etc/ssl/musicorumapp/ssl.crt;
    ssl_certificate_key /etc/ssl/musicorumapp/ssl.key;
    server_name api.musicorumapp.com;
    location / {
        proxy_pass http://api:4500/;     <------ 'api' is the service name
    }
}

You only would need to prefix the name of your stack if the reverse proxy server was running external to your stack's network, but since they are all on the same network, the DNS will resolve the service name alone.

You may also remove the ports: 8000:8000 on all of your apps (except reverse proxy) in your stack yaml file because you want to route traffic through your reverse proxy, not bind the port to the host. This could lead to security vulnerabilities as well. There are no port restrictions within a docker network. If an application is listening on 8000, your reverse proxy can contact it with http://service-name:8000 within the stack's overlay network.

Docker swarm can't reach worker nodes

1 Answers