After upgrading our Spring web app maven build to Java 11 we see a constant increase in memory consumption from the java process.
Works fine: Build with Java 8 JDK + run on server with Java 11
Has leak: Build with Java 11 + run on server with Java 11
The leak is not visible in a heap dump or even Native Memory Tracking, the process keeps increasing until physical memory + swap is full and the process is killed by the system. What kind of issue could even be causing this kind of problem?
In Java 11 the ForkJoinPool class has a slightly different behaviour.
The default elapsed time since last use before a thread is terminated is 60 seconds. In Java 8 this was undocumented but actually hardcoded with 2 seconds. In case of oversized pools, the Java 8 implementation terminates idle threads two seconds after the pool was created. But the Java 9/11 version of the class keeps them alive for minutes.
Compare the number and lifetime of threads. Since unused threads may not be terminated early anymore when your application is started or ForkJoinPools objects are created the extended lifetime of threads could easily lead to a memory issue.
See the following question for a similar problem: ForkJoinPool performance Java 8 vs 11
In Java 9 a new constructor was introduced to configure the value. To get the same behavior as with Java 8 compilation, you have to set keepAliveTime explicitly to 2 seconds or reduce the size of your ForkJoinPool objects before you compile to Java 9.
After digging a bit into Java's compiler code I found an interesting change, introduced in Java 9 which I wasn't aware of yet. This change CAN cause different behaviour depending on your compilation target:
While its widely known that most optimization are done by the JIT compiler instead of javac, the later still does some code optimizations. Before Java 9, one of these optimizations was the translation of String concatenations into StringBuilder::append chains. Starting with Java 9, javac makes use of invokedynamic calls to the newly introduced java.lang.invoke.StringConcatFactory class instead of translating to StringBuilder:append calls. So when you compile to Java 8, javac will produce optimized byte code, while when you compile to Java 9 optimization is delegated to the mentioned built-in class at runtime.
The corresponding JEP 280 provides some more details about this change. One success metric of the JEP 280 was that String concatenation performance must not regress. But JDK-8221760 already reports a potential performance regression. According to the bug entry, code with string concatenation compiled to Java 8 seems to perform bettern on Java 11u than the same code compiled to Java 9 or 11. The bug entry is still unresolved, so maybe performance is not the only regression here.
