JUnit tests fails on GitHub Actions but not locally

Question

I am trying to run JUnit tests on GitHub Actions but some of it fails. Locally all tests passed. On my PC I am using Ubuntu 20.04 with OpenJDK 1.8 (275), on CI using OpenJDK 1.8 (from standard actions).

java -version output from PC:

openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-0ubuntu1~20.04-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)

java-version output from GitHub Actions:

openjdk version "1.8.0_275"
OpenJDK Runtime Environment (Zulu 8.50.0.53-CA-linux64) (build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (Zulu 8.50.0.53-CA-linux64) (build 25.275-b01, mixed mode)

Test assertion:

@Test
public void test() {
    assertEquals(
        17,
        new CountingSheep(
            new Boolean[]{
                true, true, true, false,
                true, true, true, true,
                true, false, true, false,
                true, false, false, true,
                true, true, true, true,
                false, false, true, true
            }
        ).solution()
    );
}

Code:

package com.smlnskgmail.jaman.codewarsjava.kyu8;

import java.util.Arrays;

public class CountingSheep {

    private final Boolean[] input;

    public CountingSheep(Boolean[] input) {
        this.input = input;
    }

    public int solution() {
        return (int) Arrays
                .stream(input, 0, input.length)
                .filter(b -> b != null && b)
                .count();
    }

}

CI checks fails with:

com.smlnskgmail.jaman.codewarsjava.kyu8.CountingSheepTest > test FAILED
    java.lang.AssertionError: expected:<17> but was:<0>
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.junit.Assert.assertEquals(Assert.java:631)
        at com.smlnskgmail.jaman.codewarsjava.kyu8.CountingSheepTest.test(CountingSheepTest.java:11)

JUnit implementation:

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

GitHub Actions config:

name: Build

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

env:
  GRADLE_OPTS: -Dorg.gradle.daemon=false

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 1.8
        uses: actions/setup-java@v1
        with:
          java-version: 1.8
      - name: Build with Gradle
        run: ./gradlew build --info

But this test passed month ago and I am cannot change code.

Repository with code: https://github.com/fartem/codewars-java

SOLUTION: here is the class with reflection from the project. Without execute this class all tests are passed.

user_130659 · Answer

I think it was an error on GitHub. The tests appear to pass, and I created a fork with just that test at https://github.com/fartem/codewars-java/pull/1 which seemed to pass successfully.

Given the only thing that's changed is the version of Java materialised with your checkout action, I wonder whether or not GitHub had a temporarily broken version of Java that was rolled back.

By the way, you're using an array of Boolean in the test, but in this case you could use an array of boolean instead -- that way, your filter wouldn't need to have the b != null check there.

--

As per fartem's observation, there was an additional piece of code that ran and changed the value of Boolean.TRUE to false. The variation was a random ordering of the tests.

 Field field = Boolean.class.getField("TRUE");
 field.setAccessible(true);

 Field modifiedField = Field.class.getDeclaredField("modifiers");
 modifiedField.setAccessible(true);
 modifiedField.setInt(field, field.getModifiers() & ~Modifier.FINAL);

 field.set(null, false);

So any tests that used auto-boxing on Boolean after this test had run would get the wrong results.

JUnit tests fails on GitHub Actions but not locally

1 Answers