Instrumentation: Fuse gcov / ASan or split into independent builds (+ follow-up tests)?

user_2320035 0
Viewed 119

In my (linux-based) CI/CD pipeline i'm currently doing (basically):

  • Build Release (cmake out-of-source-build)
    • Test/(Custom-)Fuzz the build
  • Build Instrumentation-focused = gcov + ASan (again: cmake)
    • Test/(Custom-)Fuzz the build
    • Process Coverage (gcovr)

This is based on GCC 10 right now.

While thinking about adding Sonarqube-based analysis (which means using a build-wrapper which hopefully is as non-intrusive as described -> i even considered a 3rd build due to being somewhat scared!), i wondered about the original approach i have taken:

Is it safe / correct / a good idea to fuse the code-coverage and memory-sanitization instrumentation into a single build?

Is there any chance that i lose something (e.g. power of ASan-reasoning, coverage count accuracy, non-linear slowdowns) compared to independent builds (which sadly increases the time needed as we need to run the tests twice)?

Edit:

For what it's worth, a few more details on what it means using gcov/asan for me (yes: it's not as clean as it could be in regards to other parts of the cmake-def):

# Code Coverage Analysis
option(ENABLE_CODE_COVERAGE_ANALYSIS "Enable instrumentalization-based code-coverage" OFF)
if (${ENABLE_CODE_COVERAGE_ANALYSIS})
  message(STATUS "Instrumentalize for code-coverage analysis.")
  add_definitions(--coverage)
  set(COV_LINKING gcov)
endif()

# ASAN Memory Sanitization
option(ENABLE_ASAN "Enable ASAN" OFF)
if (${ENABLE_ASAN})
  message(STATUS "Instrumentalize for ASAN")
  set (CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fno-omit-frame-pointer -fsanitize=address")
  set (CMAKE_LINKER_FLAGS_DEBUG "${CMAKE_LINKER_FLAGS_DEBUG} -fno-omit-frame-pointer -fsanitize=address")
  # suppress GRPC ASAN stuff
  # https://github.com/grpc/grpc/pull/22325/files
  add_definitions("-DGRPC_ASAN_SUPPRESSED")
endif()
address-sanitizer gcov g++ gcc c++
1 Answers

If you have tests where you verify very short timings, then those tests should be run with a Release type of compile anyway (no debug, so no sanitizer either, and full optimizations).

Any other tests, those that verify correctness opposed to timings, should be run with full debug, sanitizer, coverage, and no optimizations. This increases your chances of finding issues early on.

On my end, I have three general modes while doing development:

  1. Debug
  2. Sanitizer (with debug)
  3. Release

In most cases, I run with Debug and when there seems to be a memory issue, I test with the Sanitizer version.

When ready to run my coverage tests, though, I run a script which turns on debug, the sanitizer, coverage, and removes all optimizations. When I run this script, the first thing it does is delete the COVERAGE folder and rebuild everything, for example, it could be something like this (error checks removed):

SOURCE=`pwd`
rm -rf ../COVERAGE
cd ../COVERAGE
cmake $SOURCE
make -DCOVERAGE=ON -DSANITIZER=ON -DCMAKE_BUILD_TYPE=Debug ...
tests/unittests
lcov ...
genhtml ...

So I have been running both for a while now, but I don't have tests that require timings. That being said, with the getrusage() function, I could add timing tests. It's just not really something that struk me as useful, in most cases. Especially since timings are so dependent on external things such as the CPU, network, SSD vs HDD, etc.

The main issue I had with my older computer was memory. Using the sanitizer uses much more RAM, so if you have to be able to run the tests on smaller systems (say a VirtualBox computer with 4Gb of RAM) then it can become unfeasible. On my new computer, I have 512Gb of RAM, so I don't have that issue.

user_2123780
Related