Occurrence of org.hibernate.exception.GenericJDBCException: Could not open connection for long running process

Question

In my application I use 20 threads for a job. Each thread calls a remote web service and updates the Oracle DB. After Jboss restart , the job fails in updating the db, with the exception that I find in log as:

2017-11-08 23:36:20,706 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 21) javax.resource.ResourceException: IJ000460: Error checking for a transaction
2017-11-08 23:36:20,706 ERROR [org.jboss.as.ejb3] (EJB default - 16) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
2017-11-08 23:36:20,707 ERROR [org.jboss.as.ejb3] (EJB default - 20) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
2017-11-08 23:36:20,707 ERROR [org.jboss.as.ejb3] (EJB default - 19) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
2017-11-08 23:36:20,710 ERROR [org.jboss.as.ejb3] (EJB default - 21) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
2017-11-08 23:36:20,711 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 16) JBAS014134: EJB Invocation failed on component PersonServiceDaoImpl for method public abstract long com.kaviletta.re.caltek.db.dao.PersonServiceDao.getNotUpdatedAddressCountBySubmissionId(int): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleInCallerTx(CMTTxInterceptor.java:162) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:341) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at com.kaviletta.re.caltek.db.dao.PersonServiceDao$$$view6.getNotUpdatedAddressCountBySubmissionId(Unknown Source) [classes:]
    at com.kaviletta.re.caltek.rest.service.impl.AsynchronousServiceImpl.asyncAddressDetails(AsynchronousServiceImpl.java:61) [classes:]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_55]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_55]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
    at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
    at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47) [jboss-as-jpa-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:272) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.AsyncFutureInterceptorFactory$1$1.runInvocation(AsyncFutureInterceptorFactory.java:89) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at org.jboss.as.ejb3.component.interceptors.AsyncInvocationTask.run(AsyncInvocationTask.java:73) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_55]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_55]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_55]
    at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
    at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) [hibernate-entitymanager-4.2.7.SP5-redhat-1.jar:4.2.7.SP5-redhat-1]
    at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) [hibernate-entitymanager-4.2.7.SP5-redhat-1.jar:4.2.7.SP5-redhat-1]
    at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:316) [hibernate-entitymanager-4.2.7.SP5-redhat-1.jar:4.2.7.SP5-redhat-1]
    at org.jboss.as.jpa.container.QueryNonTxInvocationDetacher.getSingleResult(QueryNonTxInvocationDetacher.java:69) [jboss-as-jpa-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]

Code used by thread for db operations :

    @PersistenceContext
EntityManager entityManager;
protected EntityManager getEntityManager() {
    return entityManager;
}

protected void setEntityManager(EntityManager manager) {
    this.entityManager = manager;
}
 public long getNotUpdatedAddressCountBySubmissionId(int submissionId) {
            Query q = getEntityManager().createQuery(
                    "select count(address.addressId.submissionId) from AddressStruc address where processState!=1  and address.addressId.submissionId = :submissionId");
            q.setParameter("submissionId", submissionId);
            return (long) q.getSingleResult();

        }

        public void updateStudentdAddress(List studnetsAddressDetails) {
            try {
                for (AddressStruc studentdAddress : studnetsAddressDetails) {
                    AddressStruc addressDetails = entityManager.find(AddressStruc.class, new AddressId(
                            studentAddress.getAddressId().getSubmissionId(), studentAddress.getAddressId().getRowId()));
                    if (addressDetails != null) {
                        entityManager.merge(updateStudentDetails(addressDetails, studentAddress));
                        entityManager.flush();
                    }

                }
            } catch (Exception e) {
                log.error("Exception occured while updating Address(method-->updateStudentAddress))-->" + e);
            }
        }

Asyn method calling updateStudentAddress method

public AsyncResult asyncAddressDetails(int startRow, int noOfRecord, int submissionId) {
    // method for fetching record based on the start row provided
    Collection addressList = studentEjb.getAddressBySubmissionId(startRow, noOfRecord, submissionId);
    StudentStatusModel status = new StudentStatusModel();
    status.setStartRow(startRow);
    status.setRetry(false);
    status.setMessage(null);
    // invoke the webservice and generate student for the given address
    if (StudentUtil.webServiceFailed <= 5) {
        OutputModel outputModel = invokeWebservice(addressList, status);

        if ((outputModel != null) && (outputModel.getStudentModels().size() > 0)) {
        // update the address details based on the submissionId+rowId
        studentEjb.updateStudentAddress(new ArrayList<>(convertModelToEntity(outputModel).values()));
    }
}
return new AsyncResult(status);
}

I am using Container managed entity manager for db transation.

Each thread calls updateStudentAddress method and updates 100 records.

Since this job is a long running , after some time jboss is unable to make further connections with oracle.

Is there any way to fix this issue ?

user_7355766 · Answer

org.hibernate.exception.GenericJDBCException: Could not open connection. could you show code?. it could be problems of synchronization threads. or bad implementation.

you need open transaction when you try to write against DB. a example

employee = em.find(Employee.class, 1);

  em.getTransaction().begin();
  employee.setNickname("Joe the Plumber");
  entityManager.merge(employee);
  em.getTransaction().commit();

user_8082162 · Answer

I think the problem may caused by the exception handling inside: asyncAddressDetails

We had this error occurring in jboss7 after RuntimeException or ApplicationException lead to the rollback of a transaction when this exception was caught and the applicationcode later did further operations on the EntityManager before the current transaction context was finished.

example:

void methodInNewTransaction() {
  try {
    dao.inRequiredTransaction();
  } catch (RuntimeException ex) {
    ; // ignore it
  }
  dao.secondCallInRequiredTransaction();
}

secondCallInRequiredTransaction might lead to the error when a RuntimeException was ignored.

How to fix this, I see the following possibilities:

prevent the Exception from occuring
use @ApplicationException-Annotation to control the transactional effects of the exception that occurs
rethrow the exception so that the outer transactioncontext will not be used anymore

user_2612030 · Answer

As another answer points out you may never catch and ignore persistence exceptions with Hibernate, it is explicitly forbidden in the docs. However, I think your problem is related to the nightly restarts. It is hard to say for sure without the complete logs, but when JBoss starts some parts start before the others. It is common to get inbound requests before everything is initialized and then there are errors. If you can prevent the restarts one night you can see if you still get the error.

The fix (except not restarting) is to ensure that the components start in the right order or to prevent incoming calls (easy if they pass through a load balancer) until the restart is complete.

To control the start order, edit application.xml and make sure initialize-in-order is true. Then list the modules in the desired start order. The web/job tier should start last.

A very dirty fix is to handle it in code by catching the error, sleeping for a few seconds and then retrying. I know some who have done that, although I wouldn't recommend it.

Occurrence of org.hibernate.exception.GenericJDBCException: Could not open connection for long running process

3 Answers