Is it possible to configure Pytest to exit with non-zero code if it skipped some tests that were not expected to be skipped (i.e., not decorated with @pytest.mark.skip)?
Context:
I'm developing a library and have configured GitHub Actions to run pytest after every commit. After some time, I realized that despite the CI finishing successfully, it doesn't execute some of my tests. The CI output looked like this:
tests/test_this.py .. [ 33%]
tests/test_that.py ..ss..s [ 66%]
tests/test_other.py ... [100%]
...
PytestUnhandledCoroutineWarning: async def functions are not natively supported and have been skipped.
...
============ 9 passed, 3 skipped in 10.10s =============
This turned out to be crucial, as I found out that the functionality covered by these tests contained a bug that didn't occur locally, but occurred on an older version of Python (which was supposed to be covered by the CI).
The reason why these tests were skipped is no problem, I fixed it. But I want to make sure that if something like this occurs in the future – my CI fails, so I don't have to manually inspect its stdout to ensure that none of the tests were skipped unintentionally.
Semi-acceptable solution: https://pypi.org/project/pytest-error-for-skips/ looks like a semi-acceptable option, but it cannot be used if we want some tests to be skipped (using @pytest.mark.skip or @pytest.mark.skipif), which is not ideal.