Cannot conect to Docker container running in VSTS

user_3554264 0
Viewed 877

I have a test which starts a Docker container, performs the verification (which is talking to the Apache httpd in the Docker container), and then stops the Docker container.

When I run this test locally, this test runs just fine. But when it runs on hosted VSTS, thus a hosted build agent, it cannot connect to the Apache httpd in the Docker container.

This is the .vsts-ci.yml file:

queue: Hosted Linux Preview

steps:
- script: |
    ./test.sh

This is the test.sh shell script to reproduce the problem:

#!/bin/bash
set -e
set -o pipefail

function tearDown {
    docker stop test-apache
    docker rm test-apache
}
trap tearDown EXIT

docker run -d --name test-apache -p 8083:80 httpd
sleep 10

curl -D - http://localhost:8083/

When I run this test locally, the output that I get is:

$ ./test.sh 
469d50447ebc01775d94e8bed65b8310f4d9c7689ad41b2da8111fd57f27cb38
HTTP/1.1 200 OK
Date: Tue, 04 Sep 2018 12:00:17 GMT
Server: Apache/2.4.34 (Unix)
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Content-Length: 45
Content-Type: text/html

<html><body><h1>It works!</h1></body></html>
test-apache
test-apache

This output is exactly as I expect.

But when I run this test on VSTS, the output that I get is (irrelevant parts replaced with …).

2018-09-04T12:01:23.7909911Z ##[section]Starting: CmdLine
2018-09-04T12:01:23.8044456Z ==============================================================================
2018-09-04T12:01:23.8061703Z Task         : Command Line
2018-09-04T12:01:23.8077837Z Description  : Run a command line script using cmd.exe on Windows and bash on macOS and Linux.
2018-09-04T12:01:23.8095370Z Version      : 2.136.0
2018-09-04T12:01:23.8111699Z Author       : Microsoft Corporation
2018-09-04T12:01:23.8128664Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2018-09-04T12:01:23.8146694Z ==============================================================================
2018-09-04T12:01:26.3345330Z Generating script.
2018-09-04T12:01:26.3392080Z Script contents:
2018-09-04T12:01:26.3409635Z ./test.sh
2018-09-04T12:01:26.3574923Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/02476800-8a7e-4e22-8715-c3f706e3679f.sh
2018-09-04T12:01:27.7054918Z Unable to find image 'httpd:latest' locally
2018-09-04T12:01:30.5555851Z latest: Pulling from library/httpd
2018-09-04T12:01:31.4312351Z d660b1f15b9b: Pulling fs layer
[…]
2018-09-04T12:01:49.1468474Z e86a7f31d4e7506d34e3b854c2a55646eaa4dcc731edc711af2cc934c44da2f9
2018-09-04T12:02:00.2563446Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2018-09-04T12:02:00.2583211Z                                  Dload  Upload   Total   Spent    Left  Speed
2018-09-04T12:02:00.2595905Z 
2018-09-04T12:02:00.2613320Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 8083: Connection refused
2018-09-04T12:02:00.7027822Z test-apache
2018-09-04T12:02:00.7642313Z test-apache
2018-09-04T12:02:00.7826541Z ##[error]Bash exited with code '7'.
2018-09-04T12:02:00.7989841Z ##[section]Finishing: CmdLine

The key thing is this:

curl: (7) Failed to connect to localhost port 8083: Connection refused

10 seconds should be enough for apache to start. Why can curl not communicate with Apache on its port 8083?

P.S.:

I know that a hard-coded port like this is rubbish and that I should use an ephemeral port instead. I wanted to get it running first wirth a hard-coded port, because that's simpler than using an ephemeral port, and then switch to an ephemeral port as soon as the hard-coded port works. And in case the hard-coded port doesn't work because the port is unavailable, the error should look different, in that case, docker run should fail because the port can't be allocated.

Update:

Just to be sure, I've rerun the test with sleep 100 instead of sleep 10. The results are unchanged, curl cannot connect to localhost port 8083.

Update 2:

When extending the script to execute docker logs, docker logs shows that Apache is running as expected. When extending the script to execute docker ps, it shows the following output:

2018-09-05T00:02:24.1310783Z CONTAINER ID        IMAGE                                                          COMMAND                  CREATED              STATUS              PORTS                  NAMES
2018-09-05T00:02:24.1336263Z 3f59aa014216        httpd                                                          "httpd-foreground"       About a minute ago   Up About a minute   0.0.0.0:8083->80/tcp   test-apache
2018-09-05T00:02:24.1357782Z 850bda64f847        microsoft/vsts-agent:ubuntu-16.04-docker-17.12.0-ce-standard   "/home/vsts/agents/2…"   2 minutes ago        Up 2 minutes                               musing_booth
azure-pipelines docker azure-devops
1 Answers

The problem is that the VSTS build agent runs in a Docker container. When the Docker container for Apache is started, it runs on the same level as the VSTS build agent Docker container, not nested inside the VSTS build agent Docker container.

There are two possible solutions:

  • Replacing localhost with the ip address of the docker host, keeping the port number 8083
  • Replacing localhost with the ip address of the docker container, changing the host port number 8083 to the container port number 80.

Access via the Docker Host

In this case, the solution is to replace localhost with the ip address of the docker host. The following shell snippet can do that:

host=localhost
if grep '^1:name=systemd:/docker/' /proc/1/cgroup
then
    apt-get update
    apt-get install net-tools
    host=$(route -n | grep '^0.0.0.0' | sed -e 's/^0.0.0.0\s*//' -e 's/ .*//')
fi
curl -D - http://$host:8083/

The if grep '^1:name=systemd:/docker/' /proc/1/cgroup inspects whether the script is running inside a Docker container. If so, it installs net-tools to get access to the route command, and then parses the default gw from the route command to get the ip address of the host. Note that this only works if the container's network default gw actually is the host.

Direct Access to the Docker Container

After launching the docker container, its ip addresses can be obtained with the following command:

docker container inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' <container-id>

Replace <container-id> with your container id or name.

So, in this case, it would be (assuming that the first ip address is okay):

ips=($(docker container inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' nuance-apache))
host=${ips[0]}
curl http://$host/
user_35542640
Related