Develop Reference

How does em.flush causes LockAcquisitionException?

I'm refactoring a code from a project and during my lunch I started looking at some error logs from the application running the old code. It's somehow related to the EntityManager#flush method (this project is using Hibernate as the persistence framework).
There's a process that select some X rows (up to 10) from a process and lock them by using for update -- #Lock(LockModeType.PESSIMISTIC_WRITE). This is within a transaction boundary and it uses Spring Data JPA Repositories and Spring Transaction Management (#Transactional).
I saw the following code (just a representation, since the actual code is huge):
#Transactional
public void method() {
// uses #Lock(LockModeType.PESSIMISTIC_WRITE)
List<Foo> entities = fooRepository.selectForUpdate();
// iterates all entities, do some work and update them
for (Foo foo: entities) {
foo.setX(321);
}
// save the list of entities
fooRepository.saveAll(foo);
// flush() is causing LockAcquisitionException
fooRepository.flush();
}
By looking at the logs, the line fooRepository.flush() is causing a LockAcquisitionException. This is really strange to me and I've never saw this before. If the rows were already selected successfully, how do this exception occurs while flushing?
Funny enough, this only happens during high throughputs periods of the day.
A snippet from the StackTrace:
org.springframework.dao.CannotAcquireLockException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.LockAcquisitionException: could not execute statement
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:287)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:253)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:527)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:242)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:153)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:144)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$ExposeRepositoryInvocationInterceptor.invoke(CrudMethodMetadataPostProcessor.java:364)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy150.flush(Unknown Source)
at xxx.xxx.xxx.xxx.MyClass.method(MyClass.java:764)
....
As per the last line of this snippet, you see that the error occurs on line 764, which is exactly where the flush happens.
Why is this happening?

Without the full stack trace and logs it is impossible to help you. The exception usually indicates a deadlock or lock timeout, so maybe you are flushing something that isn't locked yet. When flushing, MySQL tries to lock the rows in some order and if some other TX acquires locks in a different order, you see a deadlock like this.

Multi Threaded App With Issues Aquiring DB Connections

I have a Java application that reads data from a CSV file, turns them into business objects, and then saves them to a db. Most of the work is done single-threaded, but some of it is done using multiple threads. The application uses Spring framework, Hibernate as the ORM provider, c3p0 as the connection pool provider, and MySQL as the database. The parts of the application which are multi-threaded are annotated with #Async, using a ThreadPoolTaskExecutor created as a bean for the TaskExecutor that will do this work. The #Aysnc annotated methods return CompletableFuture.
With a small CSV file (under 10k records) everything is fine. However, some of our clients have much larger files (150k records) and this increase in load produces issues during the multi-threaded portions of the application. Here is one of the #Async annotated methods:
public class AppointmentTreeDAO {
#Autowired
private AppointmentRepository appointmentRepository;
private static final Logger logger = LoggerFactory.getLogger(AppointmentTreeDAO.class);
#Async
public CompletableFuture<Appointment> getObjectTree(Long oldApptId) {
logger.info("getting appointment thread");
return CompletableFuture.completedFuture(appointmentRepository.findByApptId(oldApptId));
}
}
This is the AppointmentRepository class (the last method is the one the code in question is calling):
public interface AppointmentRepository extends JpaRepository<Appointment, Long> {
List<Appointment> findByAccountId(String accountId);
List<Appointment> findByAccountIdAndProductIdAndApptStartDatetimeBetween(String accountId,
String productId, Date startDate, Date endDte);
Slice<Appointment> findByAccountIdAndProductIdAndApptStartDatetimeBetween(String accountId,
String productId, Date startDate, Date endDte, Pageable pageable);
#Query(value = "select a.appt_id from appointment a where a.appt_start_datetime " +
"between :startDate AND :endDate and a.account_id = :accountId and a.product_id = :productId",
nativeQuery = true)
List<Long> findByApptIdBetween(#Param("startDate") String startDate, #Param("endDate") String endDate,
#Param("accountId") String accountId, #Param("productId") String productId);
Appointment findByApptId(Long apptId);
}
Since I have that log line in the AppointmentTreeDAO class, I am able to see in the logs that each thread is doing work:
2:58:12.771 PM
2021-11-11 14:58:12.771 INFO 1 --- [mtSave-38] c.t.etl.repository.AppointmentTreeDAO : getting appointment thread
Eventually, I start seeing exceptions like this from the threads (same thread as above in this case):
2:58:16.790 PM
2021-11-11 14:58:16.790 WARN 1 --- [mtSave-38] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08001
2:58:16.803 PM
2021-11-11 14:58:16.803 ERROR 1 --- [mtSave-38] o.h.engine.jdbc.spi.SqlExceptionHelper : Could not create connection to database server. Attempted reconnect 3 times. Giving up.
2:58:16.804 PM
2021-11-11 14:58:16.804 WARN 1 --- [mtSave-38] c.t.etl.pipeline.load.Loader : java.util.concurrent.CompletionException: org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.lambda$doSubmit$3(AsyncExecutionAspectSupport.java:279)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:255)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:242)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:152)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:145)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy148.findByApptId(Unknown Source)
at com.etl.repository.AppointmentTreeDAO.getObjectTree(AppointmentTreeDAO.java:25)
at com.etl.repository.AppointmentTreeDAO$$FastClassBySpringCGLIB$$fb95149d.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.lambda$doSubmit$3(AsyncExecutionAspectSupport.java:276)
... 4 more
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:50)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2103)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2040)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2018)
at org.hibernate.loader.Loader.doQuery(Loader.java:948)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
at org.hibernate.loader.Loader.doList(Loader.java:2849)
at org.hibernate.loader.Loader.doList(Loader.java:2831)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2663)
at org.hibernate.loader.Loader.list(Loader.java:2658)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414)
at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625)
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593)
at org.hibernate.query.internal.AbstractProducedQuery.getSingleResult(AbstractProducedQuery.java:1641)
at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getSingleResult(CriteriaQueryTypeQueryAdapter.java:111)
at org.springframework.data.jpa.repository.query.JpaQueryExecution$SingleEntityExecution.doExecute(JpaQueryExecution.java:196)
at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:88)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:155)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:143)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:152)
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:131)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:80)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
... 22 more
Caused by: java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:903)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:453)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:155)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:146)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:205)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:169)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
... 60 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor129.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:89)
at com.mysql.cj.NativeSession.connect(NativeSession.java:144)
at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:847)
... 73 more
Caused by: java.net.NoRouteToHostException: Cannot assign requested address (Address not available)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.cj.protocol.StandardSocketFactory.connect(StandardSocketFactory.java:155)
at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:63)
... 75 more
2:58:16.804 PM
2021-11-11 14:58:16.804 WARN 1 --- [mtSave-38] c.t.etl.pipeline.load.Loader : Thread 77 ProcessFutures Loop Exception: org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection;
nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Soon after throwing the above exception it starts doing work again and its now able to call the db when it couldn't just a short time ago:
2:58:16.807 PM
2021-11-11 14:58:16.807 INFO 1 --- [mtSave-38] c.t.etl.repository.AppointmentTreeDAO : getting appointment thread
This happens intermittently with many different threads, if not all. But eventually all the work is complete and all futures are returned from that method. Then the application returns to being single threaded and the exceptions stop. Once the application returns to using multiple threads the same exceptions as above occur again.
This is my persistence.xml file:
<properties>
<property name="javax.persistence.jdbc.driver" value="com.mysql.jdbc.Driver" />
<property name="hibernate.dialect" value="org.hibernate.dialect.MySQLDialect" />
<property name="hibernate.show_sql" value="false" />
<property name="hibernate.id.new_generator_mappings" value="false" />
<property name="org.hibernate.flushMode" value="ALWAYS" />
<property name="hibernate.c3p0.acquire_increment" value="1" />
<property name="hibernate.c3p0.idle_test_period" value="60"/>
<property name="hibernate.c3p0.max_size" value="54"/>
<property name="hibernate.c3p0.max_statements" value="50"/>
<property name="hibernate.c3p0.min_size" value="5"/>
</properties>
This is how the TaskExecutor is created:
public Executor taskExecutor() {
logger.info("Creating Async Task Executor");
final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(45);
executor.setMaxPoolSize(45);
executor.setThreadNamePrefix("mtSave-");
executor.initialize();
executor.setKeepAliveSeconds(120);
return executor;
}
I have autoReconnect=true&maxReconnects=10 on my db connection string.
It seems like the threads are being starved for a connection, but I don't understand why. There will always be more work items than there are threads. 150k records of work compared to 45 threads on the high end, and under 10k records of work on the low end. The application never threw these exceptions until multi threading was introduced. I have the connection pool max size (54) set higher than the number of threads I have (45).
Ive tried changing the number of threads, the max size of the connection pool, different size files and did tons of research. Please advise.

Java / JDBC connections aren't thread safe so each thread creates its own connections.
And, MySQL servers limit the number of simultaneous connections they allow. That's a hard limit set at MySQL server startup time. So it's possible your JDBC code in all your threads is attempting to make more connections than the server allows, and so the server rejects them. The app responds with the exceptions you saw.
org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection;
There's another factor to keep in mind. Concurrent queries to the same table (INSERTs or anything else) can contend with one another. Paradoxically, you may get much better throughput with a much smaller number of threads. Try four threads.
If you give the command SHOW GLOBAL STATUS LIKE 'Connection%' you'll see a count of connection errors. Give the command before and after you run your program and take the difference between the values. The values look like this.
Connection_errors_accept errors that occurred during calls to accept() on the listening port.
Connection_errors_internal connections refused due to internal errors in the server, such as failure to start a new thread or an out-of-memory condition.
Connection_errors_max_connections connections refused because the server max_connections limit was reached.
Connection_errors_peer_address errors while searching for connecting client IP addresses.
Connection_errors_select errors during calls to OS select() or poll() on the listening port. (Failure of this operation does not necessarily means a client connection was rejected.)
Connection_errors_tcpwrap connections refused by the libwrap library.
Connections connection attempts (successful or not) to the MySQL server.

Does hibernate SessionFactory.openSession() wait for a database connection to be available from pool

Does hibernate SessionFactory.openSession() wait for a database connection to be available from pool ?
I assumed it did, but I have customer with this exception
org.hibernate.exception.GenericJDBCException: Could not open connection
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:235)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:171)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1471)
at com.jthink.songlayer.hibernate.HibernateUtil.beginTransaction(HibernateUtil.java:192)
at com.jthink.songkong.analyse.analyser.MusicBrainzSongMatcher.call(MusicBrainzSongMatcher.java:83)
at com.jthink.songkong.analyse.analyser.MusicBrainzSongMatcher.call(MusicBrainzSongMatcher.java:35)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.jthink.songkong.analyse.analyser.MainAnalyserService$EnsureIncreaseCountIfRunOnCallingThread.rejectedExecution(MainAnalyserService.java:100)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at com.jthink.songkong.analyse.analyser.MainAnalyserService.submit(MainAnalyserService.java:121)
at com.jthink.songkong.analyse.analyser.MusicBrainzMetadataMatcher.processMetadataFailedToMatch(MusicBrainzMetadataMatcher.java:107)
at com.jthink.songkong.analyse.analyser.MusicBrainzMetadataMatcher.call(MusicBrainzMetadataMatcher.java:381)
at com.jthink.songkong.analyse.analyser.MusicBrainzMetadataMatcher.call(MusicBrainzMetadataMatcher.java:34)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:689)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at org.hibernate.c3p0.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:90)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:380)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:228)
... 20 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1418)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:606)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:526)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:755)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:682)
making me think it isn't, or is it waiting for a certain time and then giving up, I am using Hibernate 4.3.11 with C3p0 and H2 1.5
My Hibernate config is
config.setProperty("hibernate.c3p0.min_size","1");
config.setProperty("hibernate.c3p0.max_size","50");
config.setProperty("hibernate.c3p0.max_statements","3000");
config.setProperty("hibernate.c3p0.timeout","2000");
config.setProperty("hibernate.c3p0.maxStatementsPerConnection","50");
config.setProperty("hibernate.c3p0.idle_test_period","3000");
config.setProperty("hibernate.c3p0.acquireRetryAttempts","10");

Hibernate SessionFactory openSession() method always opens a new session. you should close this session object once you are done with all the database operations.
Session objects are not thread safe,you should open a new session for each request in multi-threaded environment
base on this refrence
https://docs.jboss.org/hibernate/orm/3.5/api/org/hibernate/SessionFactory.html#openSession()
Session openSession()
throws HibernateException Open a Session. JDBC connection(s will be obtained from the configured ConnectionProvider
as needed to perform requested work.
Returns: The created session. Throws: HibernateException - Indicates a
peroblem opening the session; pretty rare here.
also you can check this links to resolve problem:
A ResourcePool could not acquire a resource from its primary factory or source
com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source

Hibernate schema validation fails at Microsoft SQL Azure

I would like JPA application to validate schema during start. For this reason I added the following property to persistence.xml:
<property name="hibernate.hbm2ddl.auto" value="validate"/>
However hibernate fails to validate schema:
...
Caused by: org.hibernate.exception.SQLGrammarException: Error accessing table metadata
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:106)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.tool.schema.extract.internal.InformationExtractorJdbcDatabaseMetaDataImpl.convertSQLException(InformationExtractorJdbcDatabaseMetaDataImpl.java:98)
at org.hibernate.tool.schema.extract.internal.InformationExtractorJdbcDatabaseMetaDataImpl.getTables(InformationExtractorJdbcDatabaseMetaDataImpl.java:341)
at org.hibernate.tool.schema.extract.internal.DatabaseInformationImpl.getTablesInformation(DatabaseInformationImpl.java:120)
at org.hibernate.tool.schema.internal.GroupedSchemaValidatorImpl.validateTables(GroupedSchemaValidatorImpl.java:39)
at org.hibernate.tool.schema.internal.AbstractSchemaValidator.performValidation(AbstractSchemaValidator.java:89)
at org.hibernate.tool.schema.internal.AbstractSchemaValidator.doValidation(AbstractSchemaValidator.java:68)
at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.performDatabaseAction(SchemaManagementToolCoordinator.java:191)
at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.process(SchemaManagementToolCoordinator.java:72)
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:309)
at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:462)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:938)
... 41 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: USE statement is not supported to switch between databases. Use a new connection to connect to a different database.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:258)
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:104)
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:26)
at com.microsoft.sqlserver.jdbc.SQLServerConnection$1ConnectionCommand.doExecute(SQLServerConnection.java:2907)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7240)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2869)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectionCommand(SQLServerConnection.java:2912)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.setCatalog(SQLServerConnection.java:3227)
at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.switchCatalogs(SQLServerDatabaseMetaData.java:342)
at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getResultSetFromStoredProc(SQLServerDatabaseMetaData.java:296)
at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getResultSetWithProvidedColumnNames(SQLServerDatabaseMetaData.java:318)
at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getTables(SQLServerDatabaseMetaData.java:502)
at org.apache.tomcat.dbcp.dbcp.DelegatingDatabaseMetaData.getTables(DelegatingDatabaseMetaData.java:604)
at org.hibernate.tool.schema.extract.internal.InformationExtractorJdbcDatabaseMetaDataImpl.getTables(InformationExtractorJdbcDatabaseMetaDataImpl.java:329)
... 50 more
Is there any workaround except disable validation?

It's seems to there is syantx error in SQL file. Some time the SQL might get different from database to database, and the other another database not compitable for the sql syntax you using.

Stack trace of javax.transaction.RollbackException missing the cause

I have a stack trace with a javax.transaction.RollbackException:
[...] Caused by:
ch.ethz.id.wai.lakshmi.engine.common.LakshmiException: Error
processing user transaction. at
ch.ethz.id.wai.lakshmi.engine.common.TransactionHelper.commitTransaction(TransactionHelper.java:79)
at
ch.ethz.id.wai.lakshmi.stdcmp.persistency.StandardPersistency.registerOrder(StandardPersistency.java:110)
Caused by: javax.transaction.RollbackException: Transaction marked for
rollback. at
com.sun.enterprise.transaction.JavaEETransactionImpl.commit(JavaEETransactionImpl.java:473)
at
com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.commit(JavaEETransactionManagerSimplified.java:855)
at
com.sun.enterprise.transaction.UserTransactionImpl.commit(UserTransactionImpl.java:208)
at
ch.ethz.id.wai.lakshmi.engine.common.TransactionHelper.commitTransaction(TransactionHelper.java:74)
at
org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)
at
org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
at
org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1316)
at
org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1510)
at
org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorImpl.beforeCompletion(SynchronizationCallbackCoordinatorImpl.java:109)
at
org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53)
at
com.sun.enterprise.transaction.JavaEETransactionImpl.commit(JavaEETransactionImpl.java:435)
at
org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
at
org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:129)
at
org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81)
at com.sun.proxy.$Proxy233.executeUpdate(Unknown Source) at
org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:56)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3028)
at
org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3469)
at
org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:88)
at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:362)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:354)
at
org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:275)
at
org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:326)
at
org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1213) at
org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:402)
at
org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorImpl.beforeCompletion(SynchronizationCallbackCoordinatorImpl.java:104)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:439) at
oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:395) at
oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:802) at
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:436) at
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186) at
oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521) at
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)
at
oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1008)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1307)
at
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3449)
at
oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3530)
at
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1350)
at
com.sun.gjc.spi.base.PreparedStatementWrapper.executeUpdate(PreparedStatementWrapper.java:125)
at
org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:122)
The RollbackException has a cause that I can see while debugging:
javax.persistence.PersistenceException:
org.hibernate.exception.GenericJDBCException: ORA-22295: cannot bind
more than 4000 bytes data to LOB and LONG columns in 1 statement
which is in turn caused by
org.hibernate.exception.GenericJDBCException: ORA-22295: cannot bind
more than 4000 bytes data to LOB and LONG columns in 1 statement
But the cause of the RollbackExeption is not shown in the stack trace (which would ease the debugging of the errors).
Any idea of the reason?
Edit
I added a try-catch in the test method (JUnit):
#Test
public void foo() throws Throwable {
try {
// test code generating the exception
} catch ( Exception e ) {
// here the exception cause is still there
}
Since this method is called directly by JUnit the exception is still OK when it leaves my code.

If you like to have the cause shown in the exception, I guess your only chance is to open a bug/improvement request at Glassfish. I looked at the source code of JavaEETransactionImpl, which is where the RollbackException is thrown and at line 473 the code is:
rbe = new RollbackException(sm.getString("enterprise_distributedtx.mark_rollback"));
As you can see, I am checking version 3.1.1, so in case you have a different version please take a look to the right source code (which I doubt will differ anyway).

The problem was Ant erroneously filtering part of the exception trace
From the Junit task documentation:
filtertrace Filter out Junit and Ant stack frames from error and failure stack traces.
It defaults to on but it does not only filter Junit and Ant frames: setting it to off the cause of the rollback is shown.