Summary
For a work project we use tox and pytest-cov to make sure we have a high coverage of unit tests against our code. Now on one git branch my colleague gets a higher coverage than me, which is enough on his side for the whole test suite to pass (we have 92 % set as a fail_under score), and for me it's failing because I'm just under the score.
Describe what you’ve tried
- We've compared the output of
git status, of course, to make sure nothing uncommitted was flying around in any working directories which would affect the score. [x] We both have clean working directories. - We've both re-run tox with
-rto make sure that it's not some caching issue.
Some of the configuration code
Pyproject.toml
[build-system]
requires = [
"flit_core >=2,<4",
"setuptools>=42",
"wheel",
]
build-backend = "flit_core.buildapi"
[tool.flit.metadata]
module = "our_proprietary_module_name"
author = "Alex Blesius"
author-email = "my@work.mail"
maintainer = "Our Dev Team"
maintainer-email = ""
home-page = "https://gitlab.devops.mycompany.de/voice-training-pipeline"
description-file = "README.md"
requires-python = "~=3.7"
requires = [
"azure.storage.blob",
"filetype",
"requests",
]
classifiers=[
"Intended Audience :: Developers",
# TODO: choose licence: "License :: OSI Approved :: BSD License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Software Development :: Libraries :: Python Modules",
]
[tool.flit.metadata.requires-extra]
test = [
"coverage>=5.3",
"pydocstyle",
"pytest",
"pytest-cov",
"types-requests",
]
[tool.black]
line-length = 79
[tool.coverage.report]
fail_under = 92
[tool.coverage.run]
branch = true
source_pkgs = ["voice_training_pipeline"]
source = ["tests"]
[tool.pydocstyle]
inherit = false
# docstrings in public package (__init__.py),
# docstrings in __init__.py,
# Multi-line docstring summary should start at the first line
add_ignore = "D104,D107,D212"
[tool.pytest.ini_options]
addopts = [
"--cov",
]
[[tool.mypy.overrides]]
module = "filetype"
ignore_missing_imports = true
[tool.tox]
legacy_tox_ini = """
[tox]
isolated_build = True
envlist = py37,py38,py39,py310
[flake8]
# E501: ignore line too long, since this is already caught by pylint
ignore = E501
[testenv]
deps =
black
coverage
flake8
isort
mypy
pydocstyle
pylint
pytest
pytest-cov
types-requests
commands =
isort voice_training_pipeline tests
black voice_training_pipeline tests
flake8 voice_training_pipeline
pydocstyle voice_dataset --ignore-decorators="ignore_non_imperative_docstyle_instruction"
pylint voice_training_pipeline
mypy voice_training_pipeline
pytest tests/
"""
Sample output of tox -e py310
py310 inst-nodeps: /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/.tox/.tmp/package/1/voice_training_pipeline-0.1.2.tar.gz
^Kpy310 installed: astroid==2.12.7,attrs==22.1.0,azure-core==1.25.1,azure-storage-blob==12.13.1,black==22.8.0,certifi==2022.6.15,cffi==1.15.1,charset-normalizer==2.1.1,click==8.1.3,coverage==6.4.4,cryptography==37.0.4,dill==0.3.5.1,filetype==1.1.0,flake8==5.0.4,idna==3.3,iniconfig==1.1.1,isodate==0.6.1,isort==5.10.1,lazy-object-proxy==1.7.1,mccabe==0.7.0,msrest==0.7.1,mypy==0.971,mypy-extensions==0.4.3,oauthlib==3.2.0,packaging==21.3,pathspec==0.10.1,platformdirs==2.5.2,pluggy==1.0.0,py==1.11.0,pycodestyle==2.9.1,pycparser==2.21,pydocstyle==6.1.1,pyflakes==2.5.0,pylint==2.15.0,pyparsing==3.0.9,pytest==7.1.3,pytest-cov==3.0.0,requests==2.28.1,requests-oauthlib==1.3.1,six==1.16.0,snowballstemmer==2.2.0,tomli==2.0.1,tomlkit==0.11.4,types-requests==2.28.9,types-urllib3==1.26.23,typing_extensions==4.3.0,urllib3==1.26.12,voice_training_pipeline @ file:///home/alex/Code/Voice-Processing/Voice_Training_Pipeline/.tox/.tmp/package/1/voice_training_pipeline-0.1.2.tar.gz,wrapt==1.14.1
py310 run-test-pre: PYTHONHASHSEED='3294154224'
py310 run-test: commands[0] | isort voice_training_pipeline tests
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/voice_training_pipeline/internals/conversion.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/voice_training_pipeline/asr/microsoft/stt_service.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/voice_training_pipeline/asr/microsoft/internals/azure_storage.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/voice_training_pipeline/asr/microsoft/internals/stt_rest_api.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/tests/test_ms_acoustic_dataset.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/tests/test_stt_service.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/tests/test_stt_rest_api.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/tests/test_artefact_builder.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/tests/test_azure_storage.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/tests/test_conversion.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/tests/test_voice_dataset.py
Fixing /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/tests/test_file_reader.py
py310 run-test: commands[1] | black voice_training_pipeline tests
reformatted tests/test_file_reader.py
reformatted tests/test_voice_dataset.py
reformatted tests/test_ms_acoustic_dataset.py
reformatted tests/test_artefact_builder.py
reformatted tests/test_azure_storage.py
reformatted tests/test_conversion.py
reformatted voice_training_pipeline/asr/microsoft/internals/azure_storage.py
reformatted tests/test_stt_service.py
reformatted tests/test_stt_rest_api.py
reformatted voice_training_pipeline/asr/microsoft/stt_service.py
reformatted voice_training_pipeline/internals/conversion.py
reformatted voice_training_pipeline/asr/microsoft/internals/stt_rest_api.py
All done! ✨ ✨
12 files reformatted, 21 files left unchanged.
py310 run-test: commands[2] | flake8 voice_training_pipeline
py310 run-test: commands[3] | pydocstyle voice_dataset --ignore-decorators=ignore_non_imperative_docstyle_instruction
WARNING: The /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/pyproject.toml configuration file was ignored, because the `toml` package is not installed.
py310 run-test: commands[4] | pylint voice_training_pipeline
--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
py310 run-test: commands[5] | mypy voice_training_pipeline
Success: no issues found in 21 source files
py310 run-test: commands[6] | pytest tests/
================================================================================== test session starts ==================================================================================
platform linux -- Python 3.10.5, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/py310/.pytest_cache
rootdir: /home/alex/Code/Voice-Processing/Voice_Training_Pipeline, configfile: pyproject.toml
plugins: cov-3.0.0
collected 50 items
tests/test_artefact_builder.py .. [ 4%]
tests/test_asr_service.py . [ 6%]
tests/test_azure_storage.py ... [ 12%]
tests/test_conversion.py ........ [ 28%]
tests/test_file_reader.py .. [ 32%]
tests/test_ms_acoustic_dataset.py .. [ 36%]
tests/test_stt_rest_api.py ............... [ 66%]
tests/test_stt_service.py ........ [ 82%]
tests/test_utils.py ... [ 88%]
tests/test_voice_dataset.py ... [ 94%]
tests/test_voice_record.py ... [100%]
---------- coverage: platform linux, python 3.10.5-final-0 -----------
Name Stmts Miss Branch BrPart Cover
----------------------------------------------------------------------------------------------------------
tests/__init__.py 0 0 0 0 100%
tests/test_artefact_builder.py 39 0 6 0 100%
tests/test_asr_service.py 6 0 0 0 100%
tests/test_azure_storage.py 52 0 8 0 100%
tests/test_conversion.py 49 0 20 0 100%
tests/test_file_reader.py 16 0 8 0 100%
tests/test_ms_acoustic_dataset.py 24 0 8 0 100%
tests/test_stt_rest_api.py 97 2 2 1 97%
tests/test_stt_service.py 69 0 4 0 100%
tests/test_utils.py 16 0 0 0 100%
tests/test_voice_dataset.py 20 0 6 0 100%
tests/test_voice_record.py 15 0 8 0 100%
voice_training_pipeline/__init__.py 1 0 0 0 100%
voice_training_pipeline/asr/__init__.py 0 0 0 0 100%
voice_training_pipeline/asr/microsoft/__init__.py 3 0 0 0 100%
voice_training_pipeline/asr/microsoft/artefacts/__init__.py 0 0 0 0 100%
voice_training_pipeline/asr/microsoft/artefacts/ms_acoustic_dataset.py 30 0 16 3 93%
voice_training_pipeline/asr/microsoft/internals/__init__.py 0 0 0 0 100%
voice_training_pipeline/asr/microsoft/internals/azure_storage.py 38 6 14 3 79%
voice_training_pipeline/asr/microsoft/internals/stt_rest_api.py 115 18 40 13 79%
voice_training_pipeline/asr/microsoft/internals/types.py 14 0 4 0 100%
voice_training_pipeline/asr/microsoft/stt_service.py 88 16 32 8 75%
voice_training_pipeline/components/__init__.py 0 0 0 0 100%
voice_training_pipeline/components/artefact_builder.py 13 0 2 0 100%
voice_training_pipeline/components/asr_service.py 22 1 6 1 93%
voice_training_pipeline/components/file_reader.py 7 0 2 0 100%
voice_training_pipeline/internals/__init__.py 0 0 0 0 100%
voice_training_pipeline/internals/conversion.py 86 2 40 4 94%
voice_training_pipeline/internals/errors.py 9 0 16 0 100%
voice_training_pipeline/internals/filetypes.py 8 0 0 0 100%
voice_training_pipeline/internals/utils.py 31 2 4 0 94%
voice_training_pipeline/voice_dataset.py 4 0 0 0 100%
voice_training_pipeline/voice_record.py 9 0 4 0 100%
----------------------------------------------------------------------------------------------------------
TOTAL 881 47 250 33 92%
FAIL Required test coverage of 92.0% not reached. Total coverage: 91.87%
============================================================================= 50 passed in 75.43s (0:01:15) =============================================================================
ERROR: InvocationError for command /home/alex/Code/Voice-Processing/Voice_Training_Pipeline/.tox/py310/bin/pytest tests/ (exited with code 1)
________________________________________________________________________________________ summary ________________________________________________________________________________________
ERROR: py310: commands failed
The difference is specifically in voice_training_pipeline/asr/microsoft/stt_service.py, where my colleague gets more than 77 %, whereas I only have 75. But we've compared the code even line by line and they're exactly the same files.
Any ideas what we're overlooking here?