How to run all tests in Bazel from a single java_test() rule?

In Bazel, we wrote a custom Junit Suite that finds all the Junit classes on the classpath in or under the package of the annotated class. You can find the code here. It's pretty short and straight forward and you can copy it into your project or do something similar.

You can then have your rules like this:

java_library(
    name = "tests",
    testonly = 1,
    srcs = glob(["*.java"])
)

java_test(
   name = "MyTests",
   test_class = "MyTests",
   runtime_deps = [":tests"],
)

and the MyTests.java file should look like this:

import package.ClasspathSuite;

import org.junit.runner.RunWith;

@RunWith(ClasspathSuite.class)
public class MyTests { } 

You can write a JUnit test suite class, which will run your other tests. For example, if you have test classes Test1.java and Test2.java, you can do something like this:

AllTests.java

import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;

@RunWith(Suite.class)
@SuiteClasses({
    Test1.class,
    Test2.class
})
public class AllTests {}

BUILD

java_test(
    name = "AllTests",
    test_class = "AllTests",
    srcs = [
        "AllTests.java",
        "Test1.java",
        "Test2.java",
    ],
)

EDIT in response to comment:

If you don't want to specify the test class names in your test suite, you could do something via reflection. The following example assumes all your tests are in the "com.foo" package and that all the tests are srcs of the java_test rule:

package com.foo;

import java.io.File;
import java.io.IOException;
import java.net.URLClassLoader;
import java.util.Enumeration;
import java.util.Set;
import java.util.TreeSet;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import junit.framework.JUnit4TestAdapter;
import junit.framework.TestSuite;
import org.junit.runner.RunWith;

@RunWith(org.junit.runners.AllTests.class)
public class AllTests {
  public static TestSuite suite() throws IOException {
    TestSuite suite = new TestSuite();
    URLClassLoader classLoader = (URLClassLoader) Thread.currentThread().getContextClassLoader();
    // The first entry on the classpath contains the srcs from java_test
    findClassesInJar(new File(classLoader.getURLs()[0].getPath()))
        .stream()
        .map(c -> {
          try {
            return Class.forName(c);
          } catch (ClassNotFoundException e) {
            throw new RuntimeException(e);
          }
        })
        .filter(clazz -> !clazz.equals(AllTests.class))
        .map(JUnit4TestAdapter::new)
        .forEach(suite::addTest);
    return suite;
  }

  private static Set<String> findClassesInJar(File jarFile) {
    Set<String> classNames = new TreeSet<>();
    try {
      try (ZipFile zipFile = new ZipFile(jarFile)) {
        Enumeration<? extends ZipEntry> entries = zipFile.entries();
        while (entries.hasMoreElements()) {
          String entryName = entries.nextElement().getName();
          if (entryName.startsWith("com/foo") && entryName.endsWith(".class")) {
            int classNameEnd = entryName.length() - ".class".length();
            classNames.add(entryName.substring(0, classNameEnd).replace('/', '.'));
          }
        }
      }
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
    return classNames;
  }
}

Here is a solution that doesn't require using Suite. Keep in mind, it will generate coverage reports .dat files individually for each class.

.bzl macro to handle all tests:

def run_tests(name, srcs, package, deps):
  for src in srcs:
    src_name = src[:-5]
    native.java_test(name=src_name, test_class=package + "." + src_name, srcs=srcs, deps=deps, size="small")

Calling that macro from the test files location:

run_tests(
    name = "test",
    srcs = glob(["*Test.java"]),
    package = "pkg",
    deps = [
        ":src_lib",
    ]
)

The Gerrit project contains a Starlark function called junit_tests. It takes a list of srcs and generates an AllTestsTestSuite.java file that runs tests in each Java class. It also generates a java_test target that includes the generated java file and all of the specified sources, deps, etc. Here is how to set it up.

First add these lines to your WORKSPACE file:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

# Re-usable building blocks for Bazel build tool
# https://gerrit.googlesource.com/bazlets/
# https://gerrit.googlesource.com/bazlets/+/968b97fa03a9d2afd760f2e8ede3d5643da390d2
git_repository(
    name = "com_googlesource_gerrit_bazlets",
    remote = "https://gerrit.googlesource.com/bazlets",
    commit = "968b97fa03a9d2afd760f2e8ede3d5643da390d2",
)
# We cannot use the tar.gz provided over HTTP because it contains timestamps and each download has a
# different hash.
#http_archive(
#    name = "com_googlesource_gerrit_bazlets",
#    sha256 = "...",
#    urls = [
#        "https://gerrit.googlesource.com/bazlets/+archive/968b97fa03a9d2afd760f2e8ede3d5643da390d2.tar.gz",
#    ],
#)
# This provides these useful imports:
# load("@com_googlesource_gerrit_bazlets//tools:maven_jar.bzl", "maven_jar")
# load("@com_googlesource_gerrit_bazlets//tools:junit.bzl", "junit_tests")

Now add this to your BUILD file:

load("@com_googlesource_gerrit_bazlets//tools:junit.bzl", "junit_tests")

junit_tests(
    name = "AllTests",
    srcs = glob(["*.java"]),
    deps = [
        "//java/com/company/a_package",
        "@maven//:junit_junit",
        "@maven//:org_hamcrest_hamcrest",
    ],
)

If your BUILD file is at $WORKSPACE_ROOT/javatests/com/company/a_package/BUILD then you can run these specific tests with:

bazel test //javatests/com/company/a_package:AllTests

You can run all of your java tests like this:

bazel test //javatests/...

If your directory contains a .java file that has no tests, the AllTests target will fail with "No runnable methods". The workaround is to add an empty test to the file:

/** Workaround for https://github.com/bazelbuild/bazel/issues/2539 */
@Test
public void emptyTest() {}

This is working for me with Bazel 2.0.0 on MacOS. I can also run the tests inside IntelliJ 2019.2 with the Bazel plugin.