curl hangs in docker, but not on host machine

Viewed 346

We're currently having an issue where curl bigquery.googleapis.com sometimes hangs indefinitely inside a ubuntu:20.04 local docker container, but always works (returns a 404) on the host macOS machine and in production. The container is running a Python Flask application.

Some observations:

  • It's flakey, curl bigquery.googleapis.com works when the container initializes, but trying again ~10 mins later hangs. Eventually, the command will work again and curl -v shows it's trying to connect to a different IP.
  • We don't have this issue on production, or on any host machine.
  • Different people on our team experience the issue at different times, on different IPs.
  • During the issue, connections to other google services (e.g. curl servicemanagement.googleapis.com) works fine.
  • During the issue, the command works in another docker container with a different image.
  • Issue has occurred across many macOS Docker versions.
  • Curling directly to the IPv4 address also hangs.

Interestingly, doing netstat on the host machine shows all connections as ESTABLISHED, but doing netstat inside the container shows them as SYN_SENT.

We hypothesize that the connection is being kept alive on the host, but is killed in the container. Or, certain ACKs aren't being forwarded correctly to the container for some reason.

netstat on macOS netstat inside container

netstat inside container netstat on host

TCP Dump TCP Dump

curl -v output

root@cf8bd850e9ab:/code# curl -v bigquery.googleapis.com
*   Trying 142.251.35.170:80...
* TCP_NODELAY set
*   Trying 2607:f8b0:4006:81e::200a:80...
* TCP_NODELAY set
* Immediate connect fail for 2607:f8b0:4006:81e::200a: Cannot assign requested address
*   Trying 2607:f8b0:4006:81e::200a:80...
* TCP_NODELAY set
* Immediate connect fail for 2607:f8b0:4006:81e::200a: Cannot assign requested address
*   Trying 2607:f8b0:4006:81e::200a:80...
* TCP_NODELAY set
* Immediate connect fail for 2607:f8b0:4006:81e::200a: Cannot assign requested address
*   Trying 2607:f8b0:4006:81e::200a:80...
* TCP_NODELAY set
* Immediate connect fail for 2607:f8b0:4006:81e::200a: Cannot assign requested address

Additional output:

1 Answers

You can specify the DNS while running the container with --dns flag or you can configure it by adding an entry in /etc/docker/daemon.json which could probably help with the resolution.

Related