Self-Hosted GitLab Runner fails to recognize PowerShell commands about half the time

Viewed 4

We have a GitLab runner running on one of our Windows 2016 boxes. For the most part it runs fine. However, a problem I encounter a lot when I'm trying to get a new pipeline up and running is that the runner seems to forget that to user PowerShell about half the time.

The GitLab Runner's config.toml looks like this:

concurrent = 1
check_interval = 0
log_level = "debug"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Application Template"
  url = "https://my-gitlab-instance.com"
  token = "mYt0k3n"
  executor = "shell"
  shell = "powershell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

Each time I add a new runner I have to manually change the shell = "pwsh" line to shell = "powershell" otherwise it doesn't work at all.

I'm currently working on a new pipeline which includes this deploy job:

deploy-job:
  stage: deploy
  dependencies: 
    - build-job
  environment: production
  script:
    - 'Remove-Item -Path "$env:DEPLOY_PATH\*.*"'
    - 'xcopy /y /s ".\public\*.*" "$env:DEPLOY_PATH"'

When the pipeline runs, at least 50% of the time this is the result: screenshot of failed GitLab runner job output

See there on line 24 how it says Remove-Item: command not found? That's the problem I keep getting. Thing is, if I rerun the job a few times, it'll eventually work without me changing anything.

I suspect the problem is related to the fact that line 24 begins with bash, indicating that the runner is trying to use Bash (which isn't installed on the server) to execute the script. When the job runs successfully the output is missing bash: screenshot of successful GitLab runner job output

But why is the runner sometimes using Bash to execute the script? I have explicitly told it in the config.toml to use PowerShell. And why would it sometimes use Bash and sometimes not?

1 Answers

I'd written this whole question before I suddenly remembered that I'd worked out the answer myself a long time ago, but I thought I'd post it anyway, along with the answer, in case it helps someone else (including me in a few months, probably).

TL;DR: Disable shared runners for CI/CD in your GitLab project.

The issue arises because, by default, new projects in GitLab have shared runners enabled for CI/CD. Since the shared runner that exists in our GitLab instance has never been configured otherwise (and because our self-hosted GitLab instance is running on a Linux server), it executes jobs with the Bash shell, which doesn't understand PowerShell commands. When I run my pipeline, it's pot luck whether GitLab uses my nicely-configured GitLab Runner or the shared Runner with default configuration to execute the jobs. The times when it uses the shared runner, my jobs fail. Makes perfect sense.

The fix is very simple. In the GitLab project, go to Settings > CI/CD > Runners and disable shared runners. Alternatively you could configure your shared runner(s) as you require, but in my case simply disabling shared runners is much simpler!

Related