Rust coverage using kcov does not appear correct

user_2576287 0
Viewed 879

When I record code coverage of my Rust project using codecov.io, the coverage does not appear correct.

  1. The unwrap() function and the end bracket are not covered

    unwrap and end bracket not covered

  2. The function declaration is not covered

    function declaration not covered

This is very strange.

I cannot provide the full project for reproducing.

I'm using the standard TravisCI configuration for Rust. Here is my .travis.yml:

language: rust
cache: cargo
dist: trusty
sudo: required

rust:
  - stable
  - beta
  - nightly

matrix:
  allow_failures:
    - rust: nightly

script:
  - cargo build --verbose --all
  - cargo test --verbose --all

after_success: |
  wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz &&
  tar xzf master.tar.gz &&
  cd kcov-master &&
  mkdir build &&
  cd build &&
  cmake .. &&
  make &&
  make install DESTDIR=../../kcov-build &&
  cd ../.. &&
  rm -rf kcov-master &&
  for file in target/debug/myproject-*[^\.d]; do mkdir -p "target/cov/$(basename $file)"; ./kcov-build/usr/local/bin/kcov --exclude-pattern=/.cargo,/usr/lib --verify "target/cov/$(basename $file)" "$file"; done &&
  bash <(curl -s https://codecov.io/bash)
  echo "Uploaded code coverage"
kcov codecov rust
1 Answers

Assuming Cargo's and Travis' behavior hasn't changed significantly since this question was posted, there are a couple of things at play here.

  • Whenever a build configuration changes, your build's fingerprint changes, resulting in a full or partial rebuild and a new filename for the resulting binary in target. Admittedly I'm not aware of the intricacies of exactly when or why this happens, I just know that it happens. In fact, for the project I'm working on, Cargo seems so confused about one of the dependencies that it forces a rebuild almost every single time.
  • Travis' cache: cargo default is pretty dumb; it caches all of $CARGO_HOME and target without exceptions. Note that this in combination with the former also means that these caches grow without bound, so you need to throw them away once in a while or use a smarter caching scheme.
  • for file in target/debug/myproject-*[^\.d] runs kcov for all builds of myproject, regardless of whether it's newly built or from Travis' build cache. Older builds may of course have different line numbers since they were built from different (older) sources, and coverage may be different.
  • coverage.io merges coverage results by making a line red if it is included in any report, unless it's covered by any (other) report. It doesn't show any indication whatsoever if the line numbers from different reports don't match up, or even if one of the reports contains line numbers beyond EOF. In fact, as far as I could find, it doesn't even show which binaries covered/did not cover a line number even though it has this information. You have to download the XML reports and interpret them manually to see that.

Therefore, those uncovered lines might not (all?) be due to the way Rust compiles its binaries as the comments to the question suggest, but might in fact be referring to a different (older) source file entirely. This became pretty obvious in our project after a while...

borked coverage results

If it's not this obvious, the easiest way to verify that this is what's going on is to just throw away Travis' build cache and force a rebuild.

Since incremental builds don't really work for our project anyway, the solution we used was to just not have Travis cache the target directory, as suggested here. Depending on how much your CI build time depends on incremental builds you may be forced to do something smarter.

user_78268500
Related