gitlab runner pytest fails but it shows job success

user_7526090 0
Viewed 807

I have searched for this all over the internet and couldn't find an answer.

The output of the job is something like this:

test/test_something.py:25: AssertionError
========================= 1 failed, 64 passed in 2.10s =========================

Job succeeded

my .gitlab-ci.yml file for the test:

run_tests:
  stage: test
  tags:
    - tests
  script:
    - echo "Running tests"
    - ./venv/bin/python -m pytest

I'm using shell executor. anyone faced this problem before? as I understand that gitlab CI depends on the exit code of the pytest and it should fail if the exit code is not zero, but in this case pytest should have exit code 1 since a test failed.

gitlab gitlab-ci pytest continuous-integration python
1 Answers

It's not something about your gitlab-ci script but rather your pytest script (the script or module you are using to run your tests).

Following I included an example for you, assuming that you might use something like Flask CLI to manage your tests.

You can use SystemExit to raise the exit code. If anything other than 0 is returned, it will fail the process. In a nutshell, GitLab stages are going to succeed if the exit code that is returned is 0.

Pytest only runs the tests but doesn't return the exit code. You can implement this into your code:

your manage.py (assuming you are using flask CLI) will look like something as follows:

import pytest
import click
from flask import Flask

app = Flask(__name__)


@app.cli.command("tests")
@click.argument("option", required=False)
def run_test_with_option(option: str = None):
    if option is None:
        raise SystemExit(pytest.main())

Note how the above code is raiseing and defining a flask CLI command of tests. To run your code you can simply add the following to your gitlab-ci script:

run_tests:
  stage: test
  tags:
    - tests
  variables:
    FLASK_APP: manage.py
  script:
    - echo "Running tests"
    - ./venv/bin/activate
    - flask tests

The script that will run your test will be flask tests which is raising SystemExit as shown.

FYI: you may not use Flask CLI to manage your tests script, and simply want to run a test script. In that case, this answer might also help.

user_112073580
Related