Docker multiple networks, unable to connect outside world

user_1706545 0
Viewed 186

When deploying docker-compose with multiple networks, only the first interface have an access to the outside world

version: "3.9"
services:
  speedtest:
    build:
      context: .
      dockerfile: speedtest.Dockerfile
    tty: true
    networks:
      - eth0
      - eth1

networks:
  eth0:
  eth1:

Running inside the container ping for example ping -I eth0 google.com works fine However running ping -I eth1 google.com will get the result

PING google.com (142.250.200.238) from 172.21.0.2 eth1: 56(84) bytes of data.
From c4d3b238f9a1 (172.21.0.2) icmp_seq=1 Destination Host Unreachable
From c4d3b238f9a1 (172.21.0.2) icmp_seq=2 Destination Host Unreachable

Any idea how to have egress to the internet on both networks? Tried multiple combinations for creating the network, with external, bridge with custom config etc...

Update

After larsks answer, using ip route add for eth1 and running tcpdump -i any packets are coming in correctly:

11:26:12.098918 eth1  Out IP 8077ec32b69d > dns.google: ICMP echo request, id 3, seq 1, length 64
11:26:12.184195 eth1  In  IP dns.google > 8077ec32b69d: ICMP echo reply, id 3, seq 1, length 64

But still 100% packet loss...

docker-compose docker macos networking
1 Answers

The problem here is that while there are two interfaces inside the container, there is only a single default route. Given a container with two interfaces, like this:

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
70: eth0@if71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:c0:a8:10:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.16.2/20 brd 192.168.31.255 scope global eth0
       valid_lft forever preferred_lft forever
72: eth1@if73: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:c0:a8:30:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.48.2/20 brd 192.168.63.255 scope global eth1
       valid_lft forever preferred_lft forever

The routing table looks like this:

/ # ip route
default via 192.168.16.1 dev eth0
192.168.16.0/20 dev eth0 proto kernel scope link src 192.168.16.2
192.168.48.0/20 dev eth1 proto kernel scope link src 192.168.48.2

When you run ping google.com or ping -I eth0 google.com, in both cases your ICMP request egresses through eth0, goes to the appropriate default gateway, and eventually works it way to google.com.

But when you run ping -I eth1 google.com, there's no way to reach the default gateway from that address; the gateway is only reachable via eth0. Since the kernel can't find a useful route, it attempts to connect directly. If we run tcpdump on the host interface that is the other end of with eth1, we see:

23:47:58.035853 ARP, Request who-has 142.251.35.174 tell 192.168.48.2, length 28
23:47:59.083553 ARP, Request who-has 142.251.35.174 tell 192.168.48.2, length 28
[...]

That's the kernel saying, "I've been told to connect to this address using this specific interface, but there's no route, so I'm going to assume the address is on the same network and just ARP for it".

Of course that fails.

We can make this work by adding an appropriate route. You need to run a privileged container to do this (or at least have CAP_NET_ADMIN):

ip route add default via 192.168.48.1 metric 101

(The gateway address is the .1 address of the network associated with eth1.)

We need the metric setting to differentiate this from the existing default route; without that the command would fail with RTNETLINK answers: File exists.

After running that command, we have:

/ # ip route
default via 192.168.16.1 dev eth0
default via 192.168.48.1 dev eth1 metric 101
192.168.16.0/20 dev eth0 proto kernel scope link src 192.168.16.2
192.168.48.0/20 dev eth1 proto kernel scope link src 192.168.48.2

And we can successfully ping google.com via eth1:

/ # ping -c2 -I eth1 google.com
PING google.com (142.251.35.174) from 192.168.48.2 eth1: 56(84) bytes of data.
64 bytes from lga25s78-in-f14.1e100.net (142.251.35.174): icmp_seq=1 ttl=116 time=8.87 ms
64 bytes from lga25s78-in-f14.1e100.net (142.251.35.174): icmp_seq=2 ttl=116 time=8.13 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 8.127/8.497/8.868/0.370 ms

Having gone through all that, I'll add that I don't see many situations in which it would be necessary: typically you use additional networks in order to isolate things like database servers, etc, while using the "primary" interface (the one with which the default route is associated) for outbound requests.

I tested all this using the following docker-compose.yaml:

version: "3"

services:
  sleeper:
    image: alpine
    cap_add:
      - NET_ADMIN
    command:
      - sleep
      - inf
    networks:
      - eth0
      - eth1

networks:
  eth0:
  eth1:
user_1473560
Related