Using Docker Swarm as a reverse proxy using overlay network routing mesh

user_6755250 0
Viewed 231

I have a service stack that I am deploying to my Docker swarm which has 1 manager node and 1 worker node. Its services are constrained to be placed on only one of these nodes (in this case, the manager). The worker node is intended to function only as a separate ingress point.

The manager node has the label minecraft=main set on it via docker node update --label-add minecraft=main

The swarm-scoped overlay network named minecraft-net is created separately by a docker-compose stack.

That docker-compose.yml on the manager (in host mode, not swarm) contains:

...
networks:
    minecraft-net:
        name: minecraft-net
        driver: overlay
        attachable: true
        driver_opts:
            encrypted: "true"

For accessing the Minecraft server, players should be able to connect to the worker node hostname, and the routing mesh should redirect traffic to the manager.

To deploy this stack, I use

docker stack deploy --compose-file minecraft.yml minecraft

where minecraft.yml is

version: '3.7'

services:
    crafty-controller:
        image: crafty-controller  # This image is only available on the manager node
        ports:
          - "25500-25600"
          - "13121:13121"
        volumes:
          - ./minecraft/docker/minecraft_servers:/minecraft_servers
          - ./minecraft/docker/db:/crafty_db
          - /mnt/minecraft-backups:/crafty_web/backups
        networks:
            - minecraft-net
        deploy:
            placement:
                constraints:
                  - node.labels.minecraft == main

networks:
    minecraft-net:
        external: true

I have used a similar setup in the past for an event, but now I'm running into a problem. Even though I am able to connect to the Minecraft server directly using the manager node's address, the worker node is not redirecting any traffic to or from the manager. This means I cannot connect to the Minecraft server via the worker node address.

root@debvm:/opt/app/my.site# docker stack deploy --compose-file minecraft.yml minecraft
Updating service minecraft_crafty-controller (id: aug42i46efu9bc7wv39jflxdc)
image crafty-controller:latest could not be accessed on a registry to record
its digest. Each node will access crafty-controller:latest independently,
possibly leading to different nodes running different
versions of the image.

root@debvm:/opt/app/my.site# docker stack ls
NAME                SERVICES            ORCHESTRATOR
minecraft           1                   Swarm
root@debvm:/opt/app/my.site# docker node ps
ID                  NAME                            IMAGE                      NODE                DESIRED STATE       CURRENT STATE                ERROR               PORTS
to2781e7jtjm        minecraft_crafty-controller.1   crafty-controller:latest   debvm               Running             Running about a minute ago
root@debvm:/opt/app/my.site# docker service ls
ID                  NAME                          MODE                REPLICAS            IMAGE                      PORTS
aug42i46efu9        minecraft_crafty-controller   replicated          1/1                 crafty-controller:latest   *:13121->13121/tcp, *:30000-30076->25500-25576/tcp, *:30077->25600/tcp, *:30078-30099->25578-25599/tcp, *:30100->25577/tcp
root@debvm:/opt/app/my.site# docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
6tnekpdbkktl8h7puqeba06i8 *   debvm               Ready               Active              Leader              19.03.15
qiedc8wfogv25ezlasmhe52co     workernode          Ready               Active                                  19.03.15

I can see that the worker node does not have the crafty-controller image to create its own instance of the service. This is intentional, because the image is large, and there should only be one instance anyway. What I would like to know is if it is possible to have the worker node forward traffic (requests) through the ingress overlay network to the manager node, even if the worker node does not have an image needed for that stack.

Somehow, I was able to do this a year ago, but I forgot how I got it working.

Currently, I am unable to connect to the Minecraft server via the worker node's address (but it can be connected to by using the manager node's address). Is there something that I can change in my configuration to allow players to connect to the worker node and have it redirect traffic to the manager node?

docker-swarm minecraft docker
1 Answers

I see a misunderstanding about the networking within a Swarm here.

The overlay networks manage communications among the Docker daemons participating in the swarm.

The network you're looking for is the ingress network, a special overlay network which is in charge of the load balancing among service's nodes. When any swarm node receives a request on a published port, it hands the request off to a module called IPVS, and this one will select one of the IP addresses participating in the service and route the request to it, over the ingress network.

This ingress network should have been created already when you initiated the Docker Swarm, you can check by docker network inspect ingress.

So basically, in order to be able to access to a service through the published ports in any of the nodes, that service needs to be in the ingress network.

By default all the published ports will be in ingress mode. For example from the following Compose:

version: '3.7'

services:
    crafty-controller:
        image: crafty-controller  # This image is only available on the manager node
        ports:
          - "25500-25600"
          - "13121:13121"
        volumes:
          - ./minecraft/docker/minecraft_servers:/minecraft_servers
          - ./minecraft/docker/db:/crafty_db
          - /mnt/minecraft-backups:/crafty_web/backups
        deploy:
            placement:
                constraints:
                  - node.labels.minecraft == main

We will have the following service:

...
{
   "Protocol": "tcp",
   "TargetPort": 25598,
   "PublishedPort": 30098,
   "PublishMode": "ingress"
},
{
   "Protocol": "tcp",
   "TargetPort": 25599,
   "PublishedPort": 30099,
   "PublishMode": "ingress"
}
...

You can only have one ingress network in the cluster, so if you want to actually call it minecraft-net, you'd have to inspect the already existing one, remove all the existing services whose containers are connected to the ingress network, remove the existing ingress network and create a new overlay network providing the --ingress flag.

user_89515710
Related