What are the downsides using random values in Unit Testing?

user_407511 0
Viewed 8871

I'm talking about a large scales system, with many servers and non deterministic input in high capacity. When i say non deterministic i'm talking about messages that are sent and you catch what you can and do the best you can. There are many types of messages, so the input could be very complicated. I can't imagine writing the code for so many scenarios and a simple non random (deterministic) message's generator is not good enough.

That's why i want to have a randomized unittest or server test that in case of a failure could write a log.

And i prefer the unittest instead of a random injector because i want it to run as part of the night build automated tests.

Any downsides?

unit-testing
11 Answers

Upsides: they reveal when your other tests have not covered all the invariants. Whether you want your CI server to run nondeterministic tests is another issue. Given how incredibly useful I've found https://www.artima.com/shop/scalacheck, I don't intend to do without it from now on. Let's say you're implementing a pattern-matching algorithm. Do you really know all the different corner cases? I don't. Randomized inputs may flush them out.

user_4769420

Additional Downside that hasn't been mentioned as yet is that your tests may intermittently fail at random, especially when you randomly generate several test variables so they form a convoluted and sometimes untractable dependencies. See example here.

Debugging these is a right pain in the backside and sometimes is (next to) impossible.

Also, it's often hard to tell what your test actually tests (and if it tests anything at all).

Historically in my company we use random tests at multiple levels (Unit, Integration, SingleService Tests), and that seemed like a great idea initially - it saves you code, space and time allowing to test multiple scenarios in one test.

But increasingly that gets to be a sticky point in our development, when our (even historic and reliable in the past) test start failing at random - and fixing these is way labour-intensive.

user_57780990
Related