Best practice for debug Asserts during Unit testing

Assertions in your code are (ought to be) statements to the reader that say "this condition should always be true at this point." Done with some discipline, they can be part of ensuring that the code is correct; most people use them as debug print statements. Unit Tests are code that demonstrates that your code correctly performs a particular test case; don'e well, they can both document the reuirements, and raise your confidence that the code is indeed correct.

Get the difference? The program assertions help you make it correct, the unit tests help you develop someone else's confidence that the code is correct.

A good unit test setup will have the ability to catch asserts. If an assert is triggered the current test should fail and the next is run.

In our libraries low-level debug functionality such as TTY/ASSERTS have handlers that are called. The default handler will printf/break, but client code can install custom handlers for different behavior.

Our UnitTest framework installs its own handlers that log messages and throw exceptions on asserts. The UnitTest code will then catch these exceptions if they occur and log them as an fail, along with the asserted statement.

You can also include assert testing in your unit test - e.g.

CHECK_ASSERT(someList.getAt(someList.size() + 1); // test passes if an assert occurs

I took the approach of disabling the assert only where needed as opposed to doing it project wide. Here is an approach where the assert can be suspend so it doesn't interfere with the test flow.

public static class TraceListenerCollectionEx
{
    /// <summary>
    /// This is a helper class that allows us to suspend asserts / all trace listeners
    /// </summary>
    public class SuspendTrackerDisposable : IDisposable
    {
        private readonly TraceListenerCollection _traceListenerCollection;
        private readonly TraceListener[] _suspendedListeners;

        public SuspendTrackerDisposable(TraceListenerCollection traceListenerCollection)
        {
            _traceListenerCollection = traceListenerCollection;

            var numListeners = traceListenerCollection.Count;
            _suspendedListeners = new TraceListener[numListeners];
            for( int index = 0; index < numListeners; index += 1 )
                _suspendedListeners[index] = traceListenerCollection[index];

            traceListenerCollection.Clear();
        }

        public void Dispose()
        {
            _traceListenerCollection.AddRange(_suspendedListeners);
        }
    }

    public static SuspendTrackerDisposable AssertSuspend(this TraceListenerCollection traceListenerCollection) => new SuspendTrackerDisposable(traceListenerCollection);
}

Here is an example usage within a test:

    [TestMethod]
    public void EnumDefaultTest()
    {
        using(Trace.Listeners.AssertSuspend()) {
            Enum<CarClass>.DefaultValue.ShouldBe(CarClass.Unknown);  
        }
    }

The code executed within the using block, only one line in this case, will have their asserts disabled.

Do you mean C++/Java asserts for "programming by contract" assertions, or CppUnit/JUnit asserts? That last question leads me to believe that it's the former.

Interesting question, because it's my understanding that those asserts are often turned off at runtime when you deploy to production. (Kinda defeats the purpose, but that's another question.)

I'd say that they should be left in your code when you test it. You write tests to ensure that the pre-conditions are being enforced properly. The test should be a "black box"; you should be acting as a client to the class when you test. If you happen to turn them off in production, it doesn't invalidate the tests.

First to have both Design by Contract assertions and unit tests, your unit testing framework shall be able to catch the assertions. If your unit tests abort because of a DbC abort, then you simply cannot run them. The alternative here is to disable those assertions while running (read compiling) your unit tests.

Since you're testing non-public functions, what is the risk of having a function invoked with invalid argument ? Don't your unit tests cover that risk ? If you write your code following the TDD (Test-Driven Development) technique, they should.

If you really want/need those Dbc-type asserts in your code, then you can remove the unit tests that pass the invalid arguments to the methods having those asserts.

However, Dbc-type asserts can be useful in lower level functions (that is not directly invoked by the unit tests) when you have coarse-grained unit tests.

Does heavy use of unit tests discourage the use of debug asserts?

No. The opposite. Unit testing makes Debug asserts much more valuable, by double-checking the internal state while running the white-box tests you've written. Enabling Debug.Assert during unit test is essential, because you rarely ship DEBUG-enabled code (unless performance is not important at all). The only two times DEBUG code is run is when you are either 1) doing that tiny bit of integration testing you really do, all good intentions aside, and 2) running unit tests.

It is easy to instrument code with Debug.Assert tests to check invariants as you write it. These checks serve as sanity checks when unit tests run.

The other things Assert does is point to exactly the first point in the code where things went wrong. This can greatly reduce debugging time when your unit test does find the problem.

This increases the value of unit tests.

It seems like a debug assert firing in the code under test implies the unit test shouldn't exist or the debug assert shouldn't exist.

Case in point. This question is about a real thing that happens. Right? Therefore you need debug asserts in your code, and you need them to trigger during unit tests. The possibility that a debug assert might fire during a unit test clearly demonstrates that debug asserts should be enabled during unit tests.

An assert firing means that either your tests are using your internal code incorrectly (and should be fixed), or some of the code under test is calling other internal code incorrectly, or that somewhere a fundamental assumption is wrong. You do not write tests because you think your assumptions are wrong, you... actually, you do. You write tests because at least some of your assumptions are probably wrong. Redundancy is OK in this situation.

"There can be only one" seems like a reasonable principle. Is this the common practice? Or do you disable your debug asserts when unit testing, so they can be around for integration testing?

Redundancy hurts nothing but the runtime of your unit tests. If you really have 100% coverage, runtime might be an issue. Otherwise, no I strongly disagree. There is nothing wrong with checking your assumption automatically in the middle of a test. That's practically the definition of "testing".

Also here is an example that I believe shows the dilema: A unit test passes invalid inputs for a protected function that asserts it's inputs are valid. Should the unit test not exist? It's not a public function. Perhaps checking the inputs would kill perf? Or should the assert not exist? The function is protected not private so it should be checking it's inputs for safety.

Usually, it is not the purpose of a unit test framework is to test the behavior of your code when the invariant assumptions have been violated. In other words, if the documentation you wrote says "if you pass null as a parameter, results are undefined", you do not need to verify that results are indeed unpredictable. If the failure results are clearly defined, they are not undefined, and 1) it should not be a Debug.Assert, 2) you should define exactly what the results are, and 3) test for that result. If you need to unit test the quality of your internal debug assertions, then 1) Andrew Grant's approach of making Assertion frameworks a testable asset should probably be checked as the answer, and 2) wow you have awesome test coverage! And I think this is largely a personal decision based on the project requirements. But I still think debug asserts are essential and valuable.

In other words: Debug.Assert() greatly increases the value of unit tests, and the redundancy is a feature.