I'm investigating the strange issue with tomcat shutdown process: after runnig shutdown.sh the java process still appears(I check it by using ps -ef|grep tomcat)
The situation is a bit complicated, because I have very limitted access to the server(no debug, for example)
I took thread dump(by using kill -3 <PID>) and heap dump by using remote jConsole and Hotspot features.
After looking into thread dump I found this:
"pool-2-thread-1" #74 prio=5 os_prio=0 tid=0x00007f3354359800 nid=0x7b46 waiting on condition [0x00007f333e55d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c378d330> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
So, my understanding of the problem is follows: There is a resource(DB connection or something else) which is used in CachedThreadpool, and this resource is now locked,
and prevent to thread pool-2-thread-1 to stop. Assuming that this thread isn't deamon - JVM cannot gracefully stop.
Is there a way to find out which resource is locked, from where is it locked and how to avoid that? Another question is - how to prevent this situation? Another this is: what the adress 0x00007f333e55d000 is for?
Thanks!