C3P0 apparent deadlock when the threads are all empty?

user_171061 0
Viewed 69115

I'm using C3P0 as a connection pool in Tomcat, and I'm seeing very worrying errors:

2010-09-16 13:25:00,160 [Timer-0] WARN  com.mchange.v2.async.ThreadPoolAsynchronousRunner  - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@43502400 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2010-09-16 13:25:01,407 [Timer-0] WARN  com.mchange.v2.async.ThreadPoolAsynchronousRunner  - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@43502400 -- APPARENT DEADLOCK!!! Complete Status:
  Managed Threads: 10
  Active Threads: 0
  Active Tasks:
  Pending Tasks:
    com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6e4151a7
  Pool thread stack traces:
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4,5,main]
    java.lang.Object.wait(Native Method)

... many more, exact same stack trace

Line 534 is:

 while (true) {
   Runnable myTask;
   synchronized ( ThreadPoolAsynchronousRunner.this ) {
     while ( !should_stop && pendingTasks.size() == 0 )
       ThreadPoolAsynchronousRunner.this.wait( POLL_FOR_STOP_INTERVAL ); // <- here
     if (should_stop) ...

It looks very much like all the threads are idle. They're waiting for work. 0 active threads, and only 1 task to complete. Any clues on what's going wrong?

Here's the configuration:

ds.setUser(userName);
ds.setPassword(password);
ds.setMaxPoolSize(16);
ds.setMaxConnectionAge(1800);
ds.setAcquireRetryAttempts(4);
ds.setMaxIdleTime(900);
ds.setNumHelperThreads(10);
ds.setCheckoutTimeout(1000);
c3p0 connection-pooling java
11 Answers

My comment on answer https://stackoverflow.com/a/18192588/1019307 received enough up-votes to suggest it should be an answer.

I received this error because I couldn't get through the firewall to the database server. Check to see if that is your problem.

user_10193070

@eebbesen, I got the same error as you did. I am running Tomcat version 9.0.6. I have hibernate core ver 5.2.10, hibernate c3p0 ver 3.6.3 in my maven project. Mine was not an authentication error, but rather due to me having previously changed the name of my computer. This did not have an immediate effect on tomcat, but upon a restart of my machine, when I tried to bring up tomcat again via eclipse (Oxygen 2), I could no longer start eclipse due exactly to the issue you raised.

I googled this and I found this link, which tipped me off to the issue:

https://community.oracle.com/thread/339825

where it says:

First see if OracleServiceXE and OracleXETNSListener services are running. Replace 127.0.0.1 in the url, with the IP or the name of your machine. It must match the host declared in the tnsnames.ora file.

Later it mentions where to find this tnsnames.ora file, and for me it was here:

C:\oraclexe\app\oracle\product\11.2.0\server\network\ADMIN

Looking at this tnsnames.ora file, I saw this:

XE =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = MyMachineName-7)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = XE)
    )
  )

But I had recently renamed my machine to MyMachineName-5. I changed the 7 to a 5 and saved the file. I checked the "listener.ora" file in this directory, and it had the same issue:

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
      (ADDRESS = (PROTOCOL = TCP)(HOST = MyMachineName-7)(PORT = 1521))
    )
  )

I changed the 7 to a 5 and saved the file.

Then I opened Task Manager, clicked the "Services" tab, and looked at the "Oracle" services. I did a restart on: OracleXETNSListener, OracleXEClrAgent, OracleServiceXE. I went to restart tomcat in eclipse again, and this time there was now problem.

Appendix:

I also googled this:

https://community.oracle.com/thread/2267906

This lead me to try:

1) Turned off the firewall in Windows Defender (already turned off mcaffee firewall)

2) Started sqlplus to make sure that I could log in with the credentials I was using in my hibernate file: hibernate.cfg.xml

C:\oraclexe\app\oracle\product\11.2.0\server\bin\sqlplus.exe

3) Started the desktop shortcut to Oracle Database 11g

This failed for me even after I fixed the machine name issue though, something I still have to look into.

4) Using dbVisualizer, I tried making a connection to Oracle. This only worked after I resolved the .ora file machine name issue: double click the connection and click the "ping server" button.

user_34657950

I've just had the same problem suddenly: after noticing that the deadlock was present only when launching my application in debug mode (I'm using IntelliJ) and it was fine when running with normal run, I started to dig it.

I finally figured out that a breakpoint was blocking the connection: I don't know why Intellij didn't "listen" that the application was passing through that breakpoint, but was hanged somewhere cause of a breakpoint, and this was causing appartent deadlock

After removing all breakpoints in my project, everything started smooth again.

Hope this helps someone

user_19847670
Related