Gitlab provides option to mute/quarantine flaky tests as mentioned in GitLab Documentation. I Understand its not the best of practices but do want to explore it. Some of the examples I was able to find are here - Quarantine tests that are very flaky but in rubi language. Please help to understand if that can be done in languages like C# (Dotnet) or Java. An example with that will be highly appreciated. As per Documentation: If the test cannot be fixed in a timely fashion, there is an impact on the productivity of all the developers, so it should be placed in quarantine by assigning the :quarantine metadata . I am not able to figure out how we can do that in Tests using NUnit (dotnet) or JUnit(Java)

Quarantine Test in GitLab

Document Flaky test is under contributor section must not be read as product documentation intended for users of GitLab. The page linked to the question describes the guidelines and procedures that the developers of GitLab need to perform when they encounter a flaky test in GitLab's own codebase. Feature to mute/quarantine flaky test does not appear to be currently planned for implementation. You can up vote or comment on this issue opened for the same here!.

As a Solution to this with existing resources, You can make a use of allow_failure: the required step/Job in pipeline.

In this case, you will have 2 Jobs :

With Non-Flaky test + allow_failure: false
WIth Flaky Test + allow_failure: true
example 1 example 2

In this way the test cases meant to be flaky are only handle by JOB 2 which is allowed to fail and never blocks the pipeline.

Now as a suggestion to identify/Filter test at Job level: you can use filter at test level with some category attribute. Or Can read it from TextFile/DataSource and filter basis on that.

Quarantine Test in GitLab

1 Answers