For several months now I've had issues with gitlab-runner which is randomly failing with the following log:
Running with gitlab-runner 13.7.0 (943fc252)
on <gitlab-runner-name> <gitlab-runner-id>
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:00
Running on <hostname>...
Getting source from Git repository
00:00
Fetching changes...
Reinitialized existing Git repository in /var/gitlab-runner/builds/<gitlab-runner-id>/0/<gtlab-group>/<gitlab-project>/.git/
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@<hostname>/<gtlab-group>/<gitlab-project>.git/': Problem with the SSL CA cert (path? access rights?)
ERROR: Job failed: exit status 1
This line is the crucial one:
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@<hostname>/<gtlab-group>/<gitlab-project>.git/': Problem with the SSL CA cert (path? access rights?)
I tried unregistering the runner and registering a new one. It also failed with the same error after a while (the first run usually worked well).
Furthermore, runners on other machines are working correctly and never fail with the error message above.
I believe the issue is caused by the missing CI_SERVER_TLS_CA_FILE file in:
/var/gitlab-runner/builds/<gitlab-runner-id>/0/<gtlab-group>/<gitlab-project>.tmp/CI_SERVER_TLS_CA_FILE
I tried doing a git pull in the faulty directory and I got the same message. After I copied this missing file from another directory which had it, I got the following:
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab-ci-token:<gitlab-runner-token>@gitlab.lab.sk.alcatel-lucent.com/<gtlab-group>/<gitlab-project>.git/'
As far as I know, these tokens are generated for a one-time use and are discarded after the job finishes. This leads me to believe the missing file is the issue.
Where is this file copied from? Why is it missing? What can I do to fix this issue?
I've been looking through the GitLab issues without luck.