I have a complaint that a server application of mine is crashing on high load.
It is a web app running in Tomcat 5.
I see the thread dumps and I see that there is an OutOfMemory error
1TISIGINFO Dump Event "systhrow" (00040000) Detail
"java/lang/OutOfMemoryError" "Failed to create thread: retVal -1073741830, errno 12" >received 1TIDATETIME Date: 2012/07/17 at 20:03:17 1TIFILENAME >Javacore filename:C:\ServerApplication\Tomcat5\bin\javacore.34545719.4646464.4172.0003.txt
The heap information is the following:
Maximum Java heap size : 1500m
Initial Java heap size : 256m
This is the configuration for initial and Max heap size (32 bit java)
I also see that there is available free heap space
1STHEAPFREE Bytes of Heap Space Free: 2D19F3C0
1STHEAPALLOC Bytes of Heap Space Allocated: 5DC00000
This is arround 750MB free space, right?
And from the thread method analysis I see that the number of Threads is 695 of which 49% is java/lang/Object.wait(Native Method) and 39% is in sun/misc/Unsafe.park(Native Method)
Also I see this NO JAVA STACK 1% not sure what that means.
Also 0 treads deadlocked and 2% is Runnable.
I am not sure how to interpret this information or how to go on from here to detect the underlying cause.
Any help on this?