Test with Kotlin Coroutines is randomly failing

user_4198887 0
Viewed 296

Let us suppose we have a class member whose purpose is to bring 2 objects (let's say object1 and object2) from two different places and then create the final result merging these two object in another one, which is finally returned.

Suppose then the operation of retrieving object1 and object2 can be done concurrently, so this leads to a typical use case of kotlin coroutines.

What has been described so far is shown in the following example:

fun parallelCall(): MergedObject {
    return runBlocking(context = Dispatchers.Default) {
        try {
            val object1 : Deferred<Object1> = async {
                bringObject1FromSomewhere()
            }
            val object2 : Deferred<Object2> = async {
                bringObject2FromSomewhere()
            }

            creteFinalObject(object1.await(), object2.await())

        } catch (ex: Exception) {
            throw ex
        }
    }
}

The surrounding try block should intercept any kind of exception thrown while object1 and object2 are retrieved, as well as in the createFinalObject method.

This latter simply merges together the awaited results from previous calls, waiting for both of them to be accomplished. Note that the awaiting of the deferred object1 and object2 happens almost at the same time, since they are both awaited when passed as arguments to the createFinalObject method.

In this scenario I can perform a test using mockk as mocking library such that whenever bringObject1FromSomewhere() throws an exception, then the creteFinalObject method is NEVER called. Namely, something like:

@Test
fun `GIVEN bringObject1FromSomewhere throws exception WHEN parallelCall executes THEN creteFinalObject is never executed`() {
      every { bringObject1FromSomewhere() } throws NullPointerException()
      every { bringObject2FromSomewhere() } returns sampleObject2

      assertThrows<NullPointerException> { parallelCall() }

      verify(atMost = 1) { bringObject1FromSomewhere() }
      verify(atMost = 1) { bringObject2FromSomewhere() }
      //should never be called since bringObject1FromSomewhere() throws nullPointer exception
      verify(exactly = 0) { creteFinalObject(any(), any()) }
  }

The problem is that the test above works almost always, but, there are some cases in which it randomly fails, calling the createFinalObject method regardless of the mocked values.

Is this issue related to the slight difference in time in which the deferred object1 and object2 are awaited when creteFinalObject(object1.await(), object2.await()) is called?

Another thing which comes to my mind could be the way in which I am expecting argument in the last line of the test: verify(exactly = 0) { creteFinalObject(any(), any()) } does mockk could have any problem when any() is used?.

Further, can potentially be an issue the fact that the try { } block is not able to detect the exception before the createFinalObject method is called? I would never doubt about this in a non-parallel environment but probably the usage of runBlocking as coroutineScope changes the rule of the game? Any hints will be helpful, thanks!

Kotlin version:1.6.0 Corutines version: 1.5.2 mockk version: 1.12.2

mockk kotlin-coroutines kotlin concurrency mocking
1 Answers

Are you sure it fails because it attempts to call the creteFinalObject function? Because when reading your code, I think that should be impossible (of course, never say never :D). The creteFinalObject function can only be called if both object1.await() and object2.await() return successfully.

I think something else is going on. Because you're doing 2 separate async tasks (getting object 1 and getting object 2), I suspect that the ordering of these 2 tasks would result in either a success or a failure.

Running your code locally, I notice that it sometimes fails at this line:

verify(atMost = 1) { bringObject2FromSomewhere() }

And I think there is your error. If bringObject1FromSomewhere() is called before bringObject2FromSomewhere(), the exception is thrown and the second function invocation never happens, causing the test to fail. The other way around (2 before 1) would make the test succeed. The Dispatchers.Default uses an internal work queue, where jobs that are cancelled before they are even started will never start at all. And the first task can fail fast enough for the second task to not being able to start at all.

I thought the fix would be to use verify(atLeast = 0, atMost = 1) { bringObject2FromSomewhere() } instead, but as I see on the MockK GitHub issues page, this is not supported (yet): https://github.com/mockk/mockk/issues/806

So even though you specify that bringObject2FromSomewhere() should be called at most 1 time, it still tries to verify it is also called at least 1 time, which is not the case.

You can verify this by adding a delay to the async call to get the first object:

val object1 : Deferred<Object1> = async {
    delay(100)
    bringObject1FromSomewhere()
}

This way, the test always succeeds, because bringObject2FromSomewhere() always has enough time to be called.

So how to fix this? Either hope MockK fixes the functionality to specify verify(atLeast = 0, atMost = 1) { ... }, or disable the verification on this call for now.

user_63765390
Related