I'm trying to understand a problem we're having with a Java process that is hanging. This process has been running in production for about 4 months and earlier this week it started hanging. When I look at a thread dump of the process all of the relevant threads (3) have stacks like the following:
"TxnParser_1" prio=6 tid=0x69bd3400 nid=0x2534 runnable [0x6aa2f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1099)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:478)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:790)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1039)
at oracle.jdbc.driver.T4CStatement.executeMaybeDescribe(T4CStatement.java:830)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1132)
at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1687)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1653)
- locked <0x40e22f88> (a oracle.jdbc.driver.T4CStatement)
- locked <0x28f8d398> (a oracle.jdbc.driver.T4CConnection)
at com.gcg.data.LogParsingInfo.initFromDB(LogParsingInfo.java:262)
at com.gcg.om.OmQueueEntry.initParseInfoFromDB(OmQueueEntry.java:104)
at com.gcg.om.GenericQueueEntry.run(GenericQueueEntry.java:237)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
There are no threads waiting for locks so the process isn't deadlocked. These 3 threads that are doing the work are just blocked waiting for a response from Oracle, at least that is what it looks like to me.
Looking at Oracle, when I query v$session, it looks like one of the connections associated with these threads is currently executing a query, although I can't see the sql.
select ... from v$session where ...;
SQL_ADDRESS SQL_HASH_VALUE SQL_ID SQL_CHILD_NUMBER SQL_EXEC_START SQL_EXEC_ID PREV_SQL_ADDR PREV_HASH_VALUE PREV_SQL_ID PREV_CHILD_NUMBER PREV_EXEC_START PREV_EXEC_ID
---------------- -------------- ------------- ---------------- -------------- ----------- ---------------- --------------- ------------- ----------------- --------------- ------------
00 0 0000000239F59EE8 1483377872 fqr8pndc6p36h 5 26-JUL-12 32080545
00 0 0000000239F59EE8 1483377872 fqr8pndc6p36h 5 26-JUL-12 32080546
0000000148CABD88 1784444892 a16hxxtp5sxyw 0000000239F59EE8 1483377872 fqr8pndc6p36h 5 26-JUL-12 32080544
select * from v$sql where sql_id = 'a16hxxtp5sxyw';
no rows selected
My questions are:
- Am I correct in my analysis that the process is simply blocked waiting for a response from Oracle?
- What should I be looking for in Oracle to understand why this process is blocking?
Updated:
Based on the comment regarding looking in DBA_WAITERS and DBA_LOCKS
select * from dba_waiters;
no rows selected
select * from dba_locks where BLOCKING_OTHERS <> 'Not Blocking';
no rows selected
There were 98 rows in dba_locks but since all are 'Not Blocking' I don't think it is a locking issue? The process in question has been in this state for over 3 hours so any deadlock would have been detected by now.
I'm of the theory that the Oracle instance is not "healthy", but I'm at a loss as to what to look at. I have a request in to reboot the Oracle server, but that hasn't been done yet.
Follow-up question: Is it normal for the v$session to contain a sql_id that doesn't exist in v$sql and if so, under what conditions?