Having trouble running a manual GitLab CI job on remote docker swarm cluster

user_6084948 0
Viewed 304

Here's some background: I have a Django app running on a DigitalOcean Droplet pre-installed with Docker 19.03.1~3. I am running a single-node swarm cluster and the application is working fine for the most part.

Here's the issue I'm having: I'm trying to put some management commands into manually triggered GitLab CI jobs so I don't have to SSH into the Droplet and run the commands (migrate and collectstatic). I can successfully run these commands when SSH'd into the Droplet. For example, to run collectstatic I would look up the container ID of the Django container (e710394e5449), and then run:

docker exec -it e710394e5449 python3 manage.py collectstatic

To run the management command, I was thinking that I could do something like this:

docker exec $(docker ps -q -f name="backend") python3 manage.py collectstatic --no-input

Here's the error that I'm seeing in my GitLab CI job logs:

error during connect: Get http://docker/v1.40/containers/e710394e5449/json: command [ssh -l root 123.45.6.789 -- docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host 123.45.6.789 port 22: Connection refused

Can you run docker exec from a GitLab CI job on a remote docker host? I'm able to get the container ID ($(docker ps -q -f name="backend")), but the outer docker exec command is failing. I can run the command ( docker exec $(docker ps -q -f name="backend") python3 manage.py collectstatic --no-input) from my laptop with no issues.

Here's my .gitlab-ci.yml file. The django-collectstatic job has two options for how I might run the command: one uses docker exec to run the command in an existing container, and the other uses docker run. Neither of these options work, this is just to show what I have tried.

stages:
  - build
  - deploy
  - management

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == "master"'
      when: always

image: docker:19.03.1
services:
  - docker:19.03.5-dind

.add-ssh-key: &add-ssh-key
  before_script:
    - apk update && apk add openssh-client bash
    - mkdir -p ~/.ssh
    - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - eval "$(ssh-agent -s)"
    - ssh-add ~/.ssh/id_rsa
    - ssh-keyscan -H $DROPLET_IP >> ~/.ssh/known_hosts
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

build-backend:
  stage: build
  before_script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
  script:
    - |
      docker build \
        -t $CI_REGISTRY_IMAGE/backend:$CI_COMMIT_SHORT_SHA \
        -f backend/docker/Dockerfile.prod \
        ./backend/
    - docker push $CI_REGISTRY_IMAGE/backend:$CI_COMMIT_SHORT_SHA

build-nginx:
  stage: build
  before_script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
  script:
    - |
      docker build \
        -t $CI_REGISTRY_IMAGE/nginx:$CI_COMMIT_SHORT_SHA \
        -f nginx/prod/Dockerfile \
        .
    - docker push $CI_REGISTRY_IMAGE/nginx:$CI_COMMIT_SHORT_SHA

docker-stack-deploy:
  <<: *add-ssh-key
  rules:
    - if: '$CI_COMMIT_BRANCH == "master"'
      when: delayed
      start_in: 1 minute
  stage: deploy
  variables:
    DOCKER_HOST: "ssh://root@$DROPLET_IP"
  script:
    - docker stack deploy --with-registry-auth -c stack.yml my-stack

.task-base: &task-base
  <<: *add-ssh-key
  stage: management
  variables:
    DOCKER_HOST: "ssh://root@$DROPLET_IP"
  rules:
    - when: manual

django-collectstatic:
  <<: *task-base
  script:
    # should I do this?
    - docker exec $(docker ps -q -f name="backend") python3 manage.py collectstatic --no-input
    # or this? or something else?
    # - |
    #   docker run \
    #     --rm \
    #     --network main \
    #     -e POSTGRES_PASSWORD=$POSTGRES_PASSWORD \
    #     -e SECRET_KEY=$SECRET_KEY \
    #     -e DEBUG=$DEBUG \
    #     -e DOMAIN_NAME=$DOMAIN_NAME \
    #     -v backendassets:/code/assets \
    #     $CI_REGISTRY_IMAGE/backend:$CI_COMMIT_SHORT_SHA \
    #     python3 manage.py collectstatic --no-input

django-migrate:
  <<: *task-base
  script:
    - docker exec $(docker ps -q -f name="backend") python3 manage.py migrate --no-input

django-createsuperuser:
  <<: *task-base
  script:
    - docker exec $(docker ps -q -f name="backend") python3 manage.py createsuperuser --no-input

Please let me know if there is anything else I can provide to help clarify my question.

docker-swarm gitlab gitlab-ci docker ssh
2 Answers

For now I think the easiest way to do this is to run docker commands directly over SSH rather than connecting my docker CLI (in GitLab CI) using the DOCKER_HOST environment variable.

This ends up looking something like this:

django-migrate:
  before_script:
    - apk update && apk add openssh-client bash
    - mkdir -p ~/.ssh
    - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - eval "$(ssh-agent -s)"
    - ssh-add ~/.ssh/id_rsa
    - ssh-keyscan -H $DROPLET_IP >> ~/.ssh/known_hosts
  stage: management
  rules:
    - when: manual
  script:
    - ssh root@$DROPLET_IP \
      'docker exec $(docker ps -q -f name="backend") python3 manage.py migrate --no-input'

For whatever reason, running docker exec $(docker ps -q -f name="backend") python3 manage.py migrate --no-input would fail. I would be able to get the container ID (the docker ps command would complete successfully, but the docker exec command would fail with a connection refused.

I'm not sure about the security of running the docker commands over an SSH connection.

The output of the command does show up nicely in the logs of the GitLab CI job which is exactly what I wanted.

user_60849480

I answered the same question here, you can run docker command on a remote host using option -h or DOCKER_HOST env, you need to enable TLS if not done yet.

user_140590970
Related