We are developing a plugin for IntelliJ IDEA using Gradle with the official Gradle IntelliJ Plugin as build tool.
We have written user interface tests using the IntelliJ UI Test Robot as suggested by the official documentation. These UI tests are run by first running gradle task runIdeForUiTests in the background (starting the IDE in a way which allows it to be controlled remotely) and then running the test task to remotely control the IDE in an automated way in order to verify its behavior:
$ ./gradlew runIdeForUiTests &
$ ./gradlew test
Now we want to measure the coverage using JaCoCo. This happens mostly by itself for the test task, but we would like to include the runIdeForUiTests task, so that the coverage of the UI classes is measured as well. For this, the build.gradle file has been adjusted as follows:
plugins {
id 'org.jetbrains.intellij' version '1.2.1'
id 'java'
id "org.jetbrains.kotlin.jvm" version "1.6.10"
id 'jacoco'
}
// ...
group 'my.package'
// ...
test {
useJUnitPlatform()
}
// ...
jacoco {
applyTo runIdeForUiTests
}
test {
jacoco {
includes = ['my.package.*']
}
}
runIdeForUiTests {
jacoco {
includes = ['my.package.*']
}
}
jacocoTestReport {
getExecutionData().setFrom(fileTree(buildDir).include("/jacoco/*.exec"))
reports {
csv.required = true
html.required = true
xml.required = false
}
}
The issue here is that no execution data is generated for the classes of the plugin, i.e. the runIdeForUiTests.exec file remains empty. If I dump the classes of the classpath (and comment out the includes setting), only classes by Java, IntelliJ etc. appear there (and also in runIdeForUiTests.exec), though classes of the package my.package do not:
$ find build/jacoco/classpathdumps -mindepth 3 -maxdepth 3 -type d
build/jacoco/classpathdumps/jdk/nio/zipfs
build/jacoco/classpathdumps/jdk/internal/jimage
build/jacoco/classpathdumps/jdk/internal/jrtfs
build/jacoco/classpathdumps/jdk/nashorn/internal
build/jacoco/classpathdumps/jdk/nashorn/api
build/jacoco/classpathdumps/kotlin/script/experimental
build/jacoco/classpathdumps/com/jetbrains/cef
build/jacoco/classpathdumps/com/sun/jdi
build/jacoco/classpathdumps/com/sun/jndi
build/jacoco/classpathdumps/com/sun/tools
build/jacoco/classpathdumps/com/sun/security
build/jacoco/classpathdumps/com/intellij/util
build/jacoco/classpathdumps/com/intellij/concurrency
build/jacoco/classpathdumps/com/intellij/lang
build/jacoco/classpathdumps/com/intellij/ide
build/jacoco/classpathdumps/com/intellij/openapi
build/jacoco/classpathdumps/com/intellij/idea
build/jacoco/classpathdumps/javax/lang/model
build/jacoco/classpathdumps/sun/util/resources
build/jacoco/classpathdumps/sun/nio/cs
build/jacoco/classpathdumps/sun/security/pkcs11
build/jacoco/classpathdumps/sun/security/jgss
build/jacoco/classpathdumps/sun/security/smartcardio
build/jacoco/classpathdumps/sun/security/ec
build/jacoco/classpathdumps/org/jetbrains/kotlin
build/jacoco/classpathdumps/org/cef/handler
build/jacoco/classpathdumps/org/cef/callback
build/jacoco/classpathdumps/org/jcp/xml
build/jacoco/classpathdumps/org/w3c/dom
How can we include the classes of our plugin in the coverage?
I have read that mocking frameworks cause certain classes to be excluded, however we do not use such a framework so this cannot be the issue. I feel like it could be due to the plugin's classes being dynamically loaded by IntelliJ at runtime. I'm not sure though whether this is the case at all, and haven't been able to find out anything about this regarding JaCoCo.