Gitlab-runner + Docker-compose deploying scheme: how to properly restart containers after reboot of host server

user_359866 0
Viewed 744

Suppose I have repository on Gitlab and following deploying scheme:

  1. Setup docker and gitlab-runner with docker executor on host server.
  2. In .gitlab-ci.yml setup docker-compose to build and up my service together with dependencies.
  3. Setup pipeline to be triggering by pushing commits to production branch.

Suppose docker-compose.yml has two services: app (with restart: always) and db (without restarting rule). app depends on db so docker-compose up starts db and then app.

It works perfectly until host server reboots. After it is only app container restarts.

Workarounds I've found and their cons:

  1. add restart: always to db service. But app can start before db and hence fails.
  2. use docker-compose on host machine and setup docker-compose up to autorun. But in that case I should setup docker-compose, deploy ssh-keys, clone code somewhere to the host server and update it. It seems like violating DRY principle and overcomplicating scheme.
  3. trigger pipleline after reboot. The only way I've found is to trigger it by API and trigger token. But in that case I have to setup trigger token which seems like not as bad as before but violating DRY principle and overcomplicating scheme.

How can one improve deploying scheme to make docker restart containers after reboot in right order.

P.S. Configs are as following:

.gitlab-ci.yml:

image:
  name: docker/compose:latest

services:
  - docker:dind
stages:
  - deploy

deploy:
  stage: deploy
  only:
    - production
  script:
    - docker image prune -f
    - docker-compose build --no-cache
    - docker-compose up -d

docker-compose.yml:

version: "3.8"
services:
  app:
    build: .
    container_name: app
    depends_on:
      - db
    ports:
      - "80:80"
    restart: always
  db:
    image: postgres
    container_name: db
    ports:
      - "5432:5432"
continuous-deployment gitlab gitlab-ci-runner docker deployment
2 Answers

When you add restart: always to db service, your app can start before db and fails. But your app must restart after fails, becase "restart:always" policy, if it doesn't work probably you have wrong exit code from failed app.
So You can add healthcheck and restart app after delay, which you suppose app must work.
Simple check of 80 port can help.

user_128056210

It is basically happen because you are failing fast in your app due to an unavailable database.

It can be useful in some cases, but for your use case, you can implement the app in a way that you retry to establish the connection if it fails. Ideally a backoff strategy could be implemented so that you don't overload your database in case of a real issue.

Losing the connection to the database can happen, but does it make sense to kill your app if the database is unavailable? Can you implement any fallback e.g. "Sorry we have an issue but we are working on it". In a user perspective letting them know that you have an issue and you are working to fix it, has a much better user experience than just don't open your app.

user_15632970
Related