Docker Swarm: bypass load balancer and make direct request to specific containers

user_10735374 0
Viewed 324

I have two containers running in a swarm. Each exposes a /stats endpoint which I am trying to scrape.

However, using the swarm port obviously results in the queries being load balanced and therefore the stats are all intermingled:

+--------------------------------------------------+
|                       Server                     |
|    +-------------+             +-------------+   |
|    |             |             |             |   |
|    | Container A |             | Container B |   |
|    |             |             |             |   |
|    +-------------+             +-------------+   |
|                 \              /                 |
|                  \            /                  |
|                 +--------------+                 |
|                 |              |                 |
|                 | Swarm Router |                 |
|                 |              |                 |
|                 +--------------+                 |
|                         v                        |
+-------------------------|------------------------+
                          |                         
                       A Stats                      
                       B Stats                      
                       A Stats                      
                       B Stats                      
                          |                         
                          v

I want to keep the load balancer for application requests, but also need a direct way to make requests to each container to scrape the stats.

+--------------------------------------------------+
|                       Server                     |
|    +-------------+             +-------------+   |
|    |             |             |             |   |
|    | Container A |             | Container B |   |
|    |             |             |             |   |
|    +-------------+             +-------------+   |
|        |        \              /         |       |
|        |         \            /          |       |
|        |        +--------------+         |       |
|        |        |              |         |       |
|        |        | Swarm Router |         |       |
|        v        |              |         v       |
|        |        +--------------+         |       |
|        |                |                |       |
+--------|----------------|----------------|-------+
         |                |                |
      A Stats             |             B Stats
      A Stats       Normal Traffic      B Stats
      A Stats             |             B Stats
         |                |                |
         |                |                |
         v                |                v

A dynamic solution would be ideal, but since I don't intend to do any dynamic scaling something like hardcoded ports for each container would be fine:

::8080  Both containers via load balancer
::8081  Direct access to container A
::8082  Direct access to container B

Can this be done with swarm?

docker-swarm docker load-balancing
3 Answers

From inside an overlay network you can get IP-addresses of all replicas with tasks.<service_name> DNS query:

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> -tA tasks.foo_test
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19860
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tasks.foo_test.            IN  A

;; ANSWER SECTION:
tasks.foo_test.     600 IN  A   10.0.1.3
tasks.foo_test.     600 IN  A   10.0.1.5
tasks.foo_test.     600 IN  A   10.0.1.6

This is mentioned in the documentation.

Also, if you use Prometheus to scrape those endpoints for metrics, you can combine the above with dns_sd_configs to set the targets to scrape (here is an article how). This is easy to get running but somewhat limited in features (especially in large environments).

A more advanced way to achieve the same is to use dockerswarm_sd_config (docs, example configuration). This way the list of endpoints will be gathered by querying Docker daemon, along with some useful labels (i.e. node name, service name, custom labels).

user_113445020

While less than ideal, you can introduce a microservice that acts as an intermediary to the other containers that are exposing /stats. This microservice would have to be configured with the individual endpoints and operate in the same network as said endpoints.

This doesn't bypass the load balancer, but instead makes it so it does not matter.

The intermediary could roll-up the information or you could make it more sophisticated by passing a list of opaque identifiers which the caller can then use to individually query the intermediary.

It is slightly "anti-pattern" in the sense that you have a highly coupled "stats" proxy that must be configured to be able to hit each endpoint.

That said, it is good in the sense that you don't have to expose individual containers outside of the proxy. From a security perspective, this may be better because you're not leaking additional information out of your swarm.

user_643480

You can try to publish a specific container port on a host machine ,add to your services:

    ports:
      - target: 8081
        published: 8081
        protocol: tcp
        mode: host
user_86341950
Related