We have a large enterprise telecom application running over OpenJDK8 (25.171-b10) on RHEL6.10.
We observe a very strange issue that occurs randomly in production servers.
The process starts consuming high CPU and eventually we enter memory congestion due to Java heap being nearly full.
After Analyzing the Java process using thread dumps, we noticed that a thread is blocked inside Java “notifyAll” method. The thread remains blocked inside this method for very long time, and the process consumes high CPU while the thread remains blocked inside this method.
Below is the stack trace for the thread that remains blocked -
"pool-5-thread-1" #432 prio=5 os_prio=0 tid=0x00007f7c85a57000 nid=0x37fe runnable [0x00007f7c3cba0000]
java.lang.Thread.State: RUNNABLE
at java.lang.Object.notifyAll(Native Method)
at gov.nist.javax.sip.parser.Pipeline.close(Pipeline.java:165)
- locked <0x00000005c16007b8> (a java.util.LinkedList)
at gov.nist.javax.sip.stack.NioTcpMessageChannel.close(NioTcpMessageChannel.java:258)
at gov.nist.javax.sip.stack.ConnectionOrientedMessageChannel.close(ConnectionOrientedMessageChannel.java:203)
at gov.nist.javax.sip.stack.SocketTimeoutAuditor.runTask(SocketTimeoutAuditor.java:74)
The thread remains stuck inside “notifyAll()” for several minutes while consuming high CPU.
To provide more information on the Java wait/notify mechanism being used:
We have a class “Pipeline” which has a private member “buffList” which is of type LinkedList.
An instance of Pipeline starts a thread which is synchronized on “buffList” and calls ‘this.buffList.wait()’.
public int read() throws IOException {
synchronized (this.buffList) {
...
this.buffList.wait();
...
}
When the Pipeline instance has some data or needs to be closed, another independent thread would call notifyAll(). Example snippet shows close():
public void close() throws IOException {
...
synchronized (this.buffList) {
this.buffList.notifyAll();
}
}
Normally, this works fine but for some reason, (that currently we neither can understand, neither can replicate), the call to ‘notifyAll()’ above not only blocks for several minutes but also causes high CPU usage.
Since we tried with multiple ways to replicate the issue but currently appears only in production does anyone faced similar issue? Any idea on how we can troubleshoot/find the root cause of this behavior?