Why does `exec bash` not work in a CI pipeline?

Question

I have written a github workflow file. I want to run a python program in github actions to validate few changes. I have one environment.yml file which contains all conda environment dependencies required by this program. The thing is, actual program is not running at all, and my workflow is completed with success.

Following is jobs section of workflow.yml file

jobs:
  build-linux:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
        with:
          ref: refs/pull/${{ github.event.pull_request.number }}/merge

      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: 3.8

      - name: Cache conda
        uses: actions/cache@v2
        env:
          # Increase this value to reset cache if etc/example-environment.yml has not changed
          CACHE_NUMBER: 0
        with:
          path: ~/conda_pkgs_dir
          key:
            ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{hashFiles('**/environment.yml') }}

      - uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: test-env
          environment-file: environment.yml
          use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!

      - name: Test
        run: |
          export PATH="./:$PATH"
          conda init bash
          exec bash
          conda activate test-env
          echo "Conda prefix: $CONDA_PREFIX"
          python test.py
        shell: bash

I also tried removing shell:bash in the last step, but this is also giving me the same result. The logs in last step looks like this:

Run export PATH="./:$PATH"
  export PATH="./:$PATH"
  conda init bash
  exec bash
  conda activate test-env
  echo "Conda prefix: $CONDA_PREFIX"
  python test.py
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.11/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.11/x64/lib
    CONDA_PKGS_DIR: /home/runner/conda_pkgs_dir
no change     /usr/share/miniconda/condabin/conda
no change     /usr/share/miniconda/bin/conda
no change     /usr/share/miniconda/bin/conda-env
no change     /usr/share/miniconda/bin/activate
no change     /usr/share/miniconda/bin/deactivate
no change     /usr/share/miniconda/etc/profile.d/conda.sh
no change     /usr/share/miniconda/etc/fish/conf.d/conda.fish
no change     /usr/share/miniconda/shell/condabin/Conda.psm1
no change     /usr/share/miniconda/shell/condabin/conda-hook.ps1
no change     /usr/share/miniconda/lib/python3.9/site-packages/xontrib/conda.xsh
no change     /usr/share/miniconda/etc/profile.d/conda.csh
modified      /home/runner/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

As we can clearly see, the line echo "Conda prefix: $CONDA_PREFIX" is not getting executed at all, and the workflow terminates with success. We should expect it to either run or fail the job, but nothing happens. The workflow simply ignores these commands and marks the workflow as success.

user_3840170 · Answer

Your CI script contains the line:

exec bash

When this line is executed, the shell process is replaced with a new one, and the new shell process has no idea it should continue executing the script the previous process was: all the execution state is lost. GitHub Actions passes the script to execute as a command-line argument to the initial shell process and sets standard input to /dev/null; as the new shell process is started with an empty command line and an empty file on standard input, it simply exits immediately. The fact that this works well with an interactive shell is something of a lucky coincidence.

The reason the installer directs you to restart your shell is to apply the environment variable changes added to the shell’s initialisation file. As such, it should probably be enough to replace the exec bash line with

source "$HOME/.bashrc"

However, even with this line the environment modifications will not be applied in subsequent steps, as the documentation of the setup-miniconda action warns:

Bash shells do not use ~/.profile or ~/.bashrc so these shells need to be explicitely declared as shell: bash -l {0} on steps that need to be properly activated (or use a default shell). This is because bash shells are executed with bash --noprofile --norc -eo pipefail {0} thus ignoring updated on bash profile files made by conda init bash. See Github Actions Documentation and thread.

Based on this advice, I think the best course of action is to end the actions step at the point where you would put exec bash, and apply the shell: setting to all further steps (or at least those which actually need it):

      - name: Set up Conda environment
        run: |
          export PATH="./:$PATH"
          conda init bash
      - name: Perform Conda tests
        shell: bash -l {0}
        run: |
          export PATH="./:$PATH"
          conda activate test-env
          echo "Conda prefix: $CONDA_PREFIX"
          python test.py

user_12201890 · Answer

As @user3840170 mentioned, bash shells do not use ~/.profile or ~/.bashrc. Then, a way to make it work would be to run what conda initialization would run. On GitHub Actions the path to the conda installation is on the variable $CONDA, so you can use it to run the initialization on each step needing conda activate. The following code worked for me on GitHub Actions (the one provided above didn't work in my case).

      - name: Set up Conda environment
        run: |
          echo "${HOME}/$CONDA/bin" >> $GITHUB_PATH
          conda init --all --dry-run
      - name: On the steps you want to use
        shell: bash
        run: |
          source $CONDA/etc/profile.d/conda.sh
          conda activate test-env
          
          python test.py

Why does `exec bash` not work in a CI pipeline?

2 Answers