I use Gitlab for doing Continuous Integration and Development and all of a sudden I get this error message "There has been a runner system failure, please try again"
There's no real error message or error code.
I've tried restarting the gitlab runner, using gitlab-runner restart, I've done a reboot of the server its running on but I keep getting this error message on Gitlab whenever I push a code change.
After a couple of hours, I realized the issue is that the server that Gitlab Runner is running on has no space left.
I logged into the server in question. Looked at the Gitlab log file using the following command:
journalctl -u gitlab-runner
And it showed me the following logs:
May 21 08:20:41 gitlab-runner[18936]: Checking for jobs... received job=178911 repo_url=https://.......git runner=f540b942
May 21 08:20:41 gitlab-runner-01 gitlab-runner[18936]: WARNING: Failed to process runner builds=0 error=open /tmp/trace543210445: no space left on device executor=docker runner=f540b942
To fix this issue I ran docker conatiner prune which clears out stopped containers.
Alternatively you could use docker system prune which would remove all unused objects.
See https://linuxize.com/post/how-to-remove-docker-images-containers-volumes-and-networks/ for more information about those docker commands.
Afterwards, I no longer got the error on Gitlab when pushing changes.