How to retry only failed tests in the CI job run on Gitlab?

user_13981171 0
Viewed 7895

Our automation tests run in gitlab CI environment. We have a regression suite of around 80 tests. If a test fails due to some intermittent issue, the CI job fails and since the next stage is dependent on the Regression one, the pipeline gets blocked. We retry the job to rerun regression suite expecting this time it will pass, but some other test fails this time. So, my question is: Is there any capability using which on retrying the failed CI job, only the failed tests run (Not the whole suite)?

gitlab-ci automated-tests continuous-integration
3 Answers
user_120784690

[Retry Only Failed Scenarios]

Yes, but it depends. let me explain. I'll mention the psuedo-steps which can be performed to retry only failed scenarios. The steps are specific to pytest, but can be modified depending on the test-runner.

  1. Execute the test scenarios with --last-failed. At first, all 80 scenarios will be executed.
  2. The test-runner creates a metadata file containing a list of failed tests. for example, pytest creates a folder .pytest_cache containing lastfailed file with the list of failed scenarios.
  3. We now have to add the .pytest_cache folder in the GitLab cache with the key=<gitlab-pipeline-id>.
  4. User checks that there are 5 failures and reruns the failed job.
  5. When the job is retried it will see that now .pytest_cache folder exists in the GitLab cache and will copy the folder to your test-running directory. (shouldn't fail if the cache doesn't exist to handle the 1st execution)
  6. you execute the same test cases with the same parameter --last-failed to execute the tests which were failed earlier.
  7. In the rerun, 5 test cases will be executed.

Assumptions:

  1. The test runner you are using creates a metadata file like pytest.

POC Required:

  • I have not done POC for this but in theory, it looks possible. The only doubt I have is how Gitlab parses the results. Ideally in the final result, all 80 scenarios should be pass. If it doesn't work out this way, then we have to have 2 jobs. execute tests -> [manual] execute failed tests to get 2 parsed results. I am sure with 2 stages, it will definitely work.
user_95074930

You can use Retry Analyser. This will help you definitely.

user_137431540
Related