Using internal: true on default docker-compose network leads to slow DNS timeouts

user_198840 0
Viewed 2873

Background

We're trying to run a test suite against a fairly complex application stack in a docker-compose setup. So far, everything is working nicely. However, we'd like to make sure we're not missing any external dependencies or accidentally poking anything we shouldn't (such as production systems).

This seems like what the internal network option is basically for.

Problem

When using internal: true on the compose network, the containers are not able to access any outside hosts as expected. So far, so good. The problem is that attemps to resolve hostnames that don't exist inside the composed network take a long time to timeout. A gethostbyname() call can take up to 40 seconds to fail which breaks the application stack in all kinds of ways.

I'd like to be able to configure the DNS setup in the network to simply give me an immediate NXDOMAIN for any name that isn't a container in the network, but so far I haven't found any indication that this is even possible. This seems like an obvious use-case and I have a hard time believing I'm the first to hit this problem...

I've googled high and low and gone over whatever documentation I could find but so far, no luck.

Minimal Test Setup

Given a docker-compose.yml:

version: '3'

networks:
  default:
    internal: true

services:
  foo:
    image: debian:latest
    command: /bin/sleep 1000000000

  bar:
    build:
      context: ./bar
    command: /bin/true

And bar/Dockerfile:

FROM debian:latest
RUN apt-get update && apt-get install -y dnsutils bind9-host netcat-openbsd

Then:

$ docker-compose up -d
Creating network "foo_default" with the default driver
Creating foo_bar_1 ... done
Creating foo_foo_1 ... done
$ docker-compose run bar bash
root@f7f6bf6b65d4:/# time nslookup foo
Server:     127.0.0.11
Address:    127.0.0.11#53

Non-authoritative answer:
Name:   foo
Address: 192.168.96.3

real    0m0.010s
user    0m0.004s
sys 0m0.000s
root@f7f6bf6b65d4:/#

Instant answer. So far, so good.

But:

root@f7f6bf6b65d4:/# time nslookup foo.example.com 127.0.0.11
;; connection timed out; no servers could be reached

real    0m15.009s
user    0m0.004s
sys 0m0.004s
root@f7f6bf6b65d4:/#

Bad. Was hoping for an instant NXDOMAIN response.

Even worse:

root@f7f6bf6b65d4:/# time getent hosts foo.example.com

real    0m40.034s
user    0m0.000s
sys 0m0.000s
root@f7f6bf6b65d4:/#

As mentioned, this breaks our stack in a bunch of ways and slows down test runs a lot.

For reference, the container's /etc/resolv.conf is:

nameserver 127.0.0.11
options ndots:0

And /etc/nssswitch.conf is:

passwd:         compat
group:          compat
shadow:         compat
gshadow:        files
hosts:          files dns
networks:       files
protocols:      db files
services:       db files
ethers:         db files
rpc:            db files
netgroup:       nis
docker-compose docker
2 Answers

I have tested this setup on my Ubuntu 18.04 LTS and I'm unable to reproduce your issue.

I get:

root@cedfc68cfa30:/# time nslookup foo.example.com 127.0.0.11
Server:     127.0.0.11
Address:    127.0.0.11#53

** server can't find foo.example.com: NXDOMAIN


real    0m0.025s
user    0m0.008s
sys     0m0.016s

After some testing it turns out that systemd-resolved does the required magic to make this work.

If you stop systemd-resolved and use an external DNS resolver, docker will send DNS queries with a broken source IP because there is no MASQUERADE rule in iptables for the docker network, and then you have to wait for the timeout.

user_60348820

I was able to reproduce the symptoms. The fix for me was two things.

  1. Upgrade Docker Engine from 19.x to 20.10.3. After upgrading to this version my celery container was able to connect to other containers on the same internal network. But, it could not connect to external servers like google.com for example.

  2. Added an open port to my problematic container, so in docker-compose.yaml I added the following to my celery container (aka service):

ports:
    - "65123:65123"

After adding this seemingly uneeded port to my celery container it can now connect to external servers.

Thoughts

I understand this is a hacky workaround, but in my case it works and seems more secure to open a port where nothing is listening than to make the entire network external. I would value any ideas on how secure this workaround might or might not be.

user_16159780
Related