I'm running Espresso tests using JUnit4, using Android 9. When I run tests individually, they successfully pass. However, when I run tests using classes or suites of classes, one or two tests tend to fail (out of ~100). This does not happen every time I try.
When tests do fail, they're never the same ones that failed previously. It is seemingly random. When I rerun that test individually, it of course passes. This makes it extremely hard to debug. The stack trace looks "normal", in that, it is usually a NoMatchingView or AmbigiousView exception, and doesn't mention any other issue.
I've read so many SO posts and articles at this point, I feel dizzy. I've implemented nearly every applicable solution but haven't found a sincere fix. Some of the solutions I've tried that resulted in more stable tests:
- Turning off animations via developer options in the emulator
- Disabling virtual, on-screen keyboard
- Using androidx orchestrator, which clears package data after every test:
testInstrumentationRunnerArguments clearPackageData: 'true'
- Adding idling resources for certain screens.
- Calling the
finishActivity()method on myactivityTestRuleafter every test:
@After
fun finishActivity() {
mainActivityTestRule.finishActivity()
}
Generally, these failures tend to happen less when I reinstall the app and/or cold boot the emulator. This makes me suspect it could be app storage or cache. But that seems unlikely when I'm using the orchestrator to clear data. Any thoughts?