How to make GitLab Runner in Docker see a custom CA Root certificate

Viewed 14954

I have installed and configured:

  1. an on-premises GitLab Omnibus on ServerA running on HTTPS
  2. an on-premises GitLab-Runner installed as Docker Service in ServerB

ServerA certificate is generated by a custom CA Root

The Configuration

I've have put the CA Root Certificate on ServerB:

/srv/gitlab-runner/config/certs/ca.crt

Installed the Runner on ServerB as described in Run GitLab Runner in a container - Docker image installation and configuration:

docker run -d --name gitlab-runner --restart always \
           -v /srv/gitlab-runner/config:/etc/gitlab-runner \
           -v /var/run/docker.sock:/var/run/docker.sock \
           gitlab/gitlab-runner:latest

Registered the Runner as described in Registering Runners - One-line registration command:

docker run --rm -t -i 
            -v /srv/gitlab-runner/config:/etc/gitlab-runner 
           --name gitlab-docker-runner gitlab/gitlab-runner register \
           --non-interactive \
           --executor "docker" \
           --docker-image alpine:latest \
           --url "https://MY_PRIVATE_REPO_URL_HERE/" \
           --registration-token "MY_PRIVATE_TOKEN_HERE" \
           --description "MyDockerServer-Runner" \
           --tag-list "TAG_1,TAG_2,TAG_3" \
           --run-untagged \
           --locked="false"

This command gave the following output:

Updating CA certificates...
Runtime platform arch=amd64 os=linux pid=5 revision=cf91d5e1 version=11.4.2
Running in system-mode.

Registering runner... succeeded runner=8UtcUXCY
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

I checked with

$ docker exec -it gitlab-runner bash 

and once in the container with

$ awk -v cmd='openssl x509 -noout -subject' '
/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt

and the custom CA root is correctly there.

The Problem

When running Gitlab-Runner from GitLab-CI, the pipeline fails miserably telling me that:

$ git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git


Cloning into 'My-Project.wiki'...


fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@ServerA/foo/bar/My-Project.wiki.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none


ERROR: Job failed: exit code 1

It does not recognize the Issuer (my custom CA Root), but according to The self-signed certificates or custom Certification Authorities, point n.1, it should out-of-the-box:

Default: GitLab Runner reads system certificate store and verifies the GitLab server against the CA’s stored in system.

I've then tried the solution from point n.3, editing

/srv/gitlab-runner/config/config.toml:

and adding:

[[runners]]
tls-ca-file = "/srv/gitlab-runner/config/certs/ca.crt"

But it still doesn't work.

How can I make Gitlab Runner read the CA Root certificate?

4 Answers

You have two options:

Ignore SSL verification

Put this at the top of your .gitlab-ci.yml:

variables:
  GIT_SSL_NO_VERIFY: "1"

Point GitLab-Runner to the proper certificate

As outlined in the official documentation, you can use the tls-*-file options to setup your certificate, e.g.:

[[runners]]
  ...
  tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"
  [runners.docker]
  ...

As the documentation states, "this file will be read every time when runner tries to access the GitLab server."

Other options include tls-cert-file to define the certificate to be used if needed.

While I've still not got why it doesn't work out-of-the-box, I've found the Egg of Columbus:

Gitlab-Runner configuration:

[[runners]]
  name = "MyDockerServer-Runner"
  url = "https://MY_PRIVATE_REPO_URL_HERE/"
  token = "MY_TOKEN_HERE"
  executor = "docker"
  ...
  [runners.docker]
    image = "ubuntu:latest"

  # The trick is the following:
    volumes = ["/cache","/srv/gitlab-runner/config:/etc/gitlab-runner"]
    ...

Gitlab-ci.yml pipeline:

MyJob:
    image: ubuntu:latest

    script:
      - awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
      - git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git
      - wget -O foo.png https://ServerA/foo/bar/foo.png 

    before_script:
      - apt-get update -y >/dev/null
      - apt-get install -y apt-utils dialog >/dev/null
      - apt-get install -y git >/dev/null
      - apt-get install -y wget >/dev/null

    # The trick is the following:
      - cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt
      - update-ca-certificates

That's it:

  • Mount the volume once (per Docker executor)
  • Update the CA certificates once (per job)

And everything will work as expected: git clone, wget https, etc...

A great workaround, until someone at GitLab will fix it or explain me where I'm wrong (be my guest!)

Not sure it's the best approach, but at least it worked for me. You can create a customized gitlab runner image and add your root CA inside:

├── Dockerfile
└── myca.crt
# Dockerfile
FROM gitlab/gitlab-runner:latest
COPY myca.crt /usr/local/share/ca-certificates
RUN update-ca-certificates

Build it:

docker build -t custom-gitlab-runner .

And rerun all your commands, just remember to use this new image name.

Off-topic, but related and might be useful

Dockerized gitlab-runner seem to also ignore entries in your /etc/hosts, so if you have launched Gitlab on a custom domain, e.g. https://gitlab.local.net, you need to pass the values from /etc/hosts when launching/registering gitlab runner:

docker run -d --name gitlab-runner --restart always \
       --add-host="gitlab.local.net:192.168.1.100" \
       ...

If you want to launch docker:dind (docker in docker service) container to build docker images, you also need to set these values inside /srv/gitlab-runner/config/config.toml:

[[runners]]
  url = "https://gitlab.local.net/"
  executor = "docker"
  pre_clone_script = "echo '192.168.1.100 gitlab.local.net registry.local.net' >> /etc/hosts"
  ...

From the output you provided i think that the certificate might be OK but you are lacking the CRL file : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

The CRL file is used to verify that even if the certificate is valid is hasn't been revoked by the CA owner. You shoudl then need to :

1) Generate a CRL file based on your CA:

openssl ca -gencrl -keyfile ca.key -cert ca.crt -out crl.pem

source: https://blog.didierstevens.com/2013/05/08/howto-make-your-own-cert-and-revocation-list-with-openssl/

2) Instruct the runner to use it :

[[runners]]
  ...
  tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"
  crl-file = "/etc/gitlab-runner/ssl/ca.crl"

3) Of course setting GIT_SSL_NO_VERIFY will work but you will be more sensitive to man-in-the-middle attacks

Related