Develop Reference

JmsTemplate cannot send response because javax.jms.IllegalStateException: Session is closed

I have a standalone application using Spring to connect to TIBCO (queues). Occasionally, for various reasons, TIBCO connection is closed by the server. Most of the things are recovering from this. However, sometimes JmsTemplate is not able to send response because of the error below. I have a retry in place but the same error keeps coming (see trace below).
Details that may be important:
I am using DefaultMessageListenerContainer to get the request and send the response in that receiving thread. Also, I am using the same connection factory for both DefaultMessageListenerContainer and JmsTemplate.
Caused by: org.springframework.jms.IllegalStateException: Session is closed; nested exception is javax.jms.IllegalStateException: Session is closed
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:487)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:559)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:682)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:670)
at org.springframework.integration.jms.JmsSendingMessageHandler.send(JmsSendingMessageHandler.java:149)
at org.springframework.integration.jms.JmsSendingMessageHandler.handleMessageInternal(JmsSendingMessageHandler.java:116)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
... 83 more
Caused by: javax.jms.IllegalStateException: Session is closed
at com.tibco.tibjms.TibjmsxSessionImp._createProducer(TibjmsxSessionImp.java:1067)
at com.tibco.tibjms.TibjmsxSessionImp.createProducer(TibjmsxSessionImp.java:5080)
at org.springframework.jms.core.JmsTemplate.doCreateProducer(JmsTemplate.java:1114)
at org.springframework.jms.core.JmsTemplate.createProducer(JmsTemplate.java:1095)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:591)
at org.springframework.jms.core.JmsTemplate$3.doInJms(JmsTemplate.java:562)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:484)
... 89 more
Communication with the TIBCO queue is done using Spring framework. Here is the configuration. A message is received by DefaultMessageListenerContainer, processed and JmsTemplate is used to send back the response. Connection factory is shared between receiver and sender (can this be an issue?).
<bean id="connectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory">
<constructor-arg ref="tibcoJNDI" />
<property name="targetConnectionFactory">
<bean class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate" ref="tibcoJNDI"/>
<property name="jndiName" value="${tibco.queueConnectionFactory}" />
</bean>
</property>
<property name="reconnectOnException" value="true"/>
</bean>
<bean id="client.req.msg.lstnr" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="autoStartup" value="false"/>
<property name="connectionFactory" ref="connectionFactory"/>
<property name="destination" ref="ext_client.request.queue"/>
<property name="sessionAcknowledgeMode" value="3"/>
<property name="concurrentConsumers" value="6"/>
<property name="receiveTimeout" value="60000"/>
</bean>
<jms:outbound-channel-adapter
jms-template="ext.outbound.jms.template"
channel="jms.to.ext.clnt.reply"/>
<bean id="ext.outbound.jms.template" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="connectionFactory"/>
<property name="defaultDestination" ref="ext_client.reply.queue"/>
<property name="timeToLive" value="10800000" />
<property name="explicitQosEnabled" value="true" />
</bean>
One more detail that might help. I just noticed that the first exception is actually different. There is a "Connection is closed" exception first followed by multiple "Session is closed" exception (on retry).
Caused by: javax.jms.JMSException: Connection is closed
at com.tibco.tibjms.TibjmsxLink.sendRequest(TibjmsxLink.java:322)
at com.tibco.tibjms.TibjmsxLink.sendRequest(TibjmsxLink.java:286)
at com.tibco.tibjms.TibjmsxLink.sendRequestMsg(TibjmsxLink.java:261)
at com.tibco.tibjms.TibjmsxSessionImp._createProducer(TibjmsxSessionImp.java:1075)
at com.tibco.tibjms.TibjmsxSessionImp.createProducer(TibjmsxSessionImp.java:5080)
at org.springframework.jms.core.JmsTemplate.doCreateProducer(JmsTemplate.java:1114)
at org.springframework.jms.core.JmsTemplate.createProducer(JmsTemplate.java:1095)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:591)
at org.springframework.jms.core.JmsTemplate$3.doInJms(JmsTemplate.java:562)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:484)
... 89 more

It seems that Spring does not handle properly this case. I solved this issue by overriding JmsTemplate (code below) and handling the exception myself (cleaning the session and the connection). I hope this helps.
public <T> T execute(SessionCallback<T> action, boolean startConnection) throws JmsException {
try {
return super.execute(action, startConnection);
} catch (JmsException jmse) {
logger.error("Exception while executing in JmsTemplate (will cleanup session & connection): ", jmse);
Object resourceHolder =
TransactionSynchronizationManager.getResource(getConnectionFactory());
if (resourceHolder != null && resourceHolder instanceof JmsResourceHolder) {
((JmsResourceHolder)resourceHolder).closeAll();
}
throw jmse;
}
}

This answer might help https://stackoverflow.com/a/24494739/208934 In particular:
When using JMS you shouldn't really cache the JMS Session (and
anything hanging of that such as a Producer). The reason being is that
the JMS Session is the unit of work within JMS and so should be a
short lived object. In the Java EE world that JMS Session might also
be enlisted with a global transaction for example and so needs to be
scoped correctly.
It sounds like you're reusing the same session for multiple operations.
I have seen this problem in some of my tests and telling Spring to use a pool of connections usually fixes it. If you were using active-mq then it can be done with the property spring.activemq.pooled=true. I'm not 100% how Tibco accomplishes the same thing.
You might get get same results by defining a PooledConnectionFactory bean that will override the default.

Connection pool leakage with jBPM6 transactional tables

Environment : jBPM 6.0.1.Final, jboss EAP 6.2,spring data jpa 1.4.3.RELEASE,hibernate 4.1.1.Final,spring 4.0.0.RELEASE
I'm facing connection pooling leakage issue with jBPM6 tables and hibernate is firing 50+ queries to complete a single task. Let me explain you clearly.
In my application I'm performing DML operations on application related tables, I'm observing the connection realeasing machanism by debug as
DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool]
(http-/127.0.0.1:8080-4) mysqlDS: returnConnection(3492713e, false)
[1/20]
Connection pooling mechanism works fine and then I submitted a task to be completed by jbpm,some data inserting into jbpm6 tables like SessionInfo, Task and when data inserted,connection is not releasing and for next task also it is not using opened connections and opening new connections and go on.. Finally I'm getting the following message due to jbpm6 tables.
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/mysqlDS
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147)
at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:70) [hibernate-entitymanager-4.2.7.SP1-redhat-3.jar:4.2.7.SP1-redhat-3]
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292) [hibernate-core-4.2.7.SP1-redhat-3.jar:4.2.7.SP1-redhat-3]
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214) [hibernate-core-4.2.7.SP1-redhat-3.jar:4.2.7.SP1-redhat-3]
... 111 more
I'm using org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean and org.springframework.orm.jpa.JpaTransactionManager for session and transaction which is working for tables i insert data. But not working for data inserting by jbpm6.
I configured <property name="hibernate.connection.release_mode" value="auto"/> as well but not useful to get the connection back.
<!--Following entity manager for Mysql database-->
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="persistenceUnitName" value="masterDataMysql"/>
</bean>
<!--Transaction manager for both H2 and Mysql-->
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory"/>
</bean>
Click here For error log
UPDATE : This is happening due to connections are not closing by jBPM6
Please help how to solve connection leaking issue with jbpm6 ???
Any help is highly appreciated.

java.sql.SQLException: Connections could not be acquired from the underlying database

I'm using Hibernate, MySQL connection is valid but I get the following.
How do I see more details on connection string that it's connecting to?
ERROR SchemaUpdate - could not get database metadata
java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:78)
at org.hibernate.tool.hbm2ddl.SuppliedConnectionProviderConnectionHelper.prepare(SuppliedConnectionProviderConnectionHelper.java:51)
at org.hibernate.tool.hbm2ddl.SchemaUpdate.execute(SchemaUpdate.java:168)
at org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:356)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1341)
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
... 10 more
ERROR SchemaUpdate - could not complete schema update
java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:78)
at org.hibernate.tool.hbm2ddl.SuppliedConnectionProviderConnectionHelper.prepare(SuppliedConnectionProviderConnectionHelper.java:51)
at org.hibernate.tool.hbm2ddl.SchemaUpdate.execute(SchemaUpdate.java:168)
at org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:356)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1341)
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
... 10 more
ERROR JDBCExceptionReporter - Connections could not be acquired from the underlying database!
org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1354)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.hibernate.context.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:342)
at com.sun.proxy.$Proxy0.beginTransaction(Unknown Source)
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:78)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 15 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
... 18 more
here's my hibernate xml:
<hibernate-configuration>
<session-factory>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://127.0.0.1:3306/test</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password">xxxxxx</property>
<property name="show_sql">false</property>
<property name="dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.hbm2ddl.auto">update</property>
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.max_statements">200</property>
<property name="hibernate.c3p0.min_size">20</property>
<property name="hibernate.c3p0.max_size">200</property>
<property name="hibernate.c3p0.idle_test_period">300</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">200</property>
<property name="current_session_context_class">thread</property>
</session-factory>
</hibernate-configuration>

In my case it was problem that c3p0-0.9.2.1.jar file was not copied by maven into src/main/webapp/web-inf/libs

I will recommend two things to get this problem resolved :
First, try mentioning your database name in capital letters. Though db is case insensitive still , sometimes it runs fine with caps.
Secondly , this problem may arise because of some extra stuff you might have appended with your session factory value. For example (i will take code i used and faced this problem and the got rid of it) :
Part of my xml goes like this :
<session-factory>
<property name="hbm2ddl.auto">update</property>
<property name="dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="connection.url">jdbc:mysql://10.184.56.183:3306/******?relaxAutoCommit=true&autoReconnect=true&useUnicode=true&characterEncoding=utf8</property>
<property name="connection.username">******</property>
<property name="connection.password">******</property>
<property name="hibernate.show_sql">true</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.connection.pool_size">5</property>
<property name="hibernate.connection.shutdown">true</property>
<property name="hibernate.cache.provider_class">org.hibernate.cache.EhCacheProvider</property>
<property name="hibernate.cache.use_second_level_cache">false</property>
<property name="hibernate.cache.use_query_cache">false</property>
<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.min_size">2</property>
<property name="hibernate.c3p0.max_size">5</property>
<property name="hibernate.c3p0.timeout">108000</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">1000</property>
<property name="hibernate.c3p0.validate">true</property>
<property name="hibernate.c3p0.acquire_increment">5</property>
So keep trimming extra values like clean your database connection url and other stuff. I faced problem while generating code from database table as well. So i created a different config xml and made that as simple as possible and thats it. It went very fine .

I solved this challange by following Confluence Support article.
The Causes and Resolutions as stated by the above blog post for the issue of
cannot establish a connection to its database. are
The database configuration details you entered are incorrect or have
changed. For example, the database name or database
username/password for your Confluence database. Confirm that the
database username, password, and connection URL in
/confluence.cfg.xml are incorrect, and if not
update them accordingly.
The URL for the database is incorrect.
For example, if the database connection is jdbc:mysql://yma/confluence?autoReconnect=true, does the host 'yma'
exist and is the database server installed on 'yma'? Check
/confluence.cfg.xml to find this entry.
The database server is not running. If your database server is down, restart it and check the logs to see why it had failed or
stopped.
The network is down (or there is a firewall in between Confluence and its database) and the connection to the database
cannot be established. If your network is down, or a new firewall is
installed, consult with your System Administrator for further
troubleshooting.
Your database password has expired.
If your password has expired (and you will only discover this if you try to connect to your
database via your database client or an external database tool such
as DbVisualizer), update your password. You may want to deactivate
expiring passwords for your Confluence database user to prevent the
problem from happening again. Be sure to consult with your DBA about
this.
Postgres specific issue: the Postgres encryption method has been altered or changed when migrating or upgrading Confluence, the
environment, etc. Check the pg_hba.conf file. If the method is set
to ident change it to md5.
MySQL specific issue: there are too many connection requests that triggered MySQL server to block Confluence from making any
requests. See this MySQL article for resolution.

I had this problem and the issue was our office IP address being blocked by MySQL server due to many connect attempts using bad credentials.
Make sure you can connect to the server using the same method/network as your application. If you get this or something like this when attempting to connect using some MySQL client (like Workbench)
Host 'XXX-XXX-XXX-XXX.static.XXXXXX.XXX' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'
Your IP is blocked and that is the issue.

Hibernate / MySQL connection timeouts -- Trying to deal with thread-pooled executors not releasing Hibernate connections to C3P0 after exit

I have a Tomcat application that uses MySQL, and Hibernate for ORM. The nature of our application demands that we have to pull and aggregate a lot of analytics data from a NoSQL store for each request , so we split the pulling and aggregating for each request into several tasks and delegate those to a thread-pooled executor service.
When each thread performs a task, it needs to query / update MySQL regarding certain things so it borrows Hibernate sessions from C3P0 ( which we use for connection pooling ).
Essential config :
<property name="current_session_context_class">thread</property>
<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.connection.shutdown">true</property>
<property name="hibernate.use_sql_comments">false</property>
<!-- C3p0 Performance Improvements -->
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.idle_test_period">300</property>
<property name="hibernate.c3p0.maxConnectionAge">3600</property>
<property name="hibernate.c3p0.timeout">120</property>
<property name="hibernate.c3p0.max_size">300</property>
<property name="hibernate.c3p0.min_size">1</property>
<property name="hibernate.c3p0.max_statements">100</property>
<property name="hibernate.c3p0.preferredTestQuery">select 1;</property>
The problem is Hibernate requests cause the MySQL / JDBC connection timeout error after 8 hours (our configured value of MySQL's wait_timeout parameter is the default i.e. 8 hours) . I replicated this by setting wait_timeout to 11 minutes but the result is the same for a 8-hour wait_timeout as well :
2013-01-27 20:08:00,088 ERROR [Thread-0] (JDBCExceptionReporter.java:234) - Communications link failure
The last packet successfully received from the server was 665,943 milliseconds ago. The last packet sent successfully to the server was 6 milliseconds ago.
org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:99)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2536)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1716)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
.....
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 665,943 milliseconds ago. The last packet sent successfully to the server was 6 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3102)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2991)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3532)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1953)
at org.hibernate.loader.Loader.doQuery(Loader.java:802)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
... 9 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2552)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3002)
... 22 more
2013-01-27 20:19:00,179 WARN [Thread-0] (NewPooledConnection.java:487) - [c3p0] Another error has occurred [ com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 1,326,037 milliseconds ago. The last packet sent successfully to the server was 660,100 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem. ] which will not be reported to listeners!
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 1,326,037 milliseconds ago. The last packet sent successfully to the server was 660,100 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3364)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1983)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1953)
at org.hibernate.loader.Loader.doQuery(Loader.java:802)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1716)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
....
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3345)
... 20 more
I thought C3P0 wasn't evicting stale connections frequently enough, so that during heavy load a connection would be returned to the pool, go stale and be borrowed again before eviction. So I created a c3p0.properties file and set "c3p0.testConnectionsOnCheckout" to true so that no stale connections would be borrowed from the pool. I got the same error again.
I figured that since my hibernate sessions are configured with session-context "thread" , sessions used by a thread aren't released until a transaction is committed or rolled back [ http://docs.jboss.org/hibernate/orm/3.6/javadocs/org/hibernate/context/ThreadLocalSessionContext.html ] . So the only explanation I have is that the executor threads make read-only DB calls where there's no committing going on , and once the task is done, the thread returns to the pool , holding the session with it.
What should I do here ? Our application has hibernate querying code inside Data-Access-Object classes, one for each kind of bean. I wish to avoid having to change each and every method in those classes that makes read-only DB calls so as to have them "commit" unnecessarily. I also wish to minimize the changes to the business logic of our app.
Is there a way I can check for the freshness of the session returned by sessionFactory.getCurrentSession() each time a Data-Access-Object is declared / used (all Data-Access-Object classes inherit a few things from a single base class, so I can change things in that base class constructor) , and maybe return stale sessions to the pool ? Or is there a better way to do it ?
Thanks.

NOTE: the below is based on an Oracle example, but can be modified for MySQL.
I have gotten around this problem of dropped connections by using Tomcat managed databased connections. This can be done by creating a context.xml file in your META-INF directory, something like this:
<Context crossContext="true" docBase="myBase" path="/myBase" reloadable="false" useHttpOnly="true">
<ResourceLink global="jdbc/dbOne" name="jdbc/dbOne" type="javax.sql.DataSource" />
<!-- Need another DB? -->
<!-- <ResourceLink global="jdbc/dbTwo" name="jdbc/dbTwo" type="javax.sql.DataSource" /> -->
</Context>
To create the connection your Tomcat server.xml file must also be updated. The below is based on Tomcat 6.0.26.:
<!-- Global JNDI resources
Documentation at /docs/jndi-resources-howto.html
-->
<GlobalNamingResources>
<!-- Editable user database that can also be used by
UserDatabaseRealm to authenticate users
-->
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
<!-- START MY MODS -->
<!-- Add the db connection(s)! The rest is standard in the server.xml file.-->
<Resource auth="Container" driverClassName="oracle.jdbc.driver.OracleDriver" initialSize="2" logAbandoned="true" maxActive="5" maxIdle="2" maxWait="120000" minEvictableIdleTimeMillis="1800000" minIdle="1" name="jdbc/dbOne" numTestsPerEvictionRun="3" password="openPlease" removeAbandoned="true" removeAbandonedTimeout="60" testOnBorrow="true" testOnReturn="true" testWhileIdle="true" timeBetweenEvictionRunsMillis="900000" type="javax.sql.DataSource" url="jdbc:oracle:thin:#servername:portnumber:schema" username="myUser" validationQuery="select sysdate from dual"/>
<!-- Need another connection? Copy and past the above making the required changes. -->
<!-- END MY MODS -->
Then in my hibernate.cfg.xml file I have specified:
<!-- Connection handling -->
<property name="connection.datasource">java:/comp/env/jdbc/dbOne</property>
<property name="dialect">org.hibernate.dialect.Oracle10gDialect</property>
<property name="connection.autoReconnect">true</property>
<property name="connection.autoReconnectForPools">true</property>
<property name="connection.is-connection-validation-required">true</property>
<property name="current_session_context_class">thread</property>
Having said that, the following config using just C3P0 worked for me as well, but I now prefer Tomcat managed connections as they can be shared between the apps on my Tomcat server, which is desirable in my case.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.connection.driver_class">oracle.jdbc.driver.OracleDriver</property>
<property name="hibernate.connection.password">#dbPassword#</property>
<property name="hibernate.connection.url">#dbURL#</property>
<property name="hibernate.connection.username">someUser</property>
<property name="dialect">org.hibernate.dialect.Oracle10gDialect</property>
<property name="show_sql">false</property>
<property name="format_sql">false</property>
<property name="current_session_context_class">thread</property>
<!-- Connection handling -->
<property name="connection.autoReconnect">true</property>
<property name="connection.autoReconnectForPools">true</property>
<property name="connection.is-connection-validation-required">true</property>
<property name="current_session_context_class">thread</property>
<property name="max_fetch_depth">1</property>
<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.breakAfterAcquireFailure">false</property>
<property name="hibernate.c3p0.acquireRetryAttempts">-1</property>
<property name="hibernate.c3p0.acquireRetryDelay">3000</property>
<property name="hibernate.c3p0.automaticTestTable">C3P0_TEST</property> <!-- c3p0 uses this table for testing connection. -->
<property name="hibernate.c3p0.initialPoolSize">2</property> <!-- 3 is c3p0 default. -->
<property name="hibernate.c3p0.minPoolSize">2</property> <!-- 1 is Hibernate default. -->
<property name="hibernate.c3p0.maxPoolSize">6</property> <!-- 100 is Hibernate default. -->
<property name="hibernate.c3p0.acquireIncrement">2</property> <!-- 3 is c3p0 default. -->
<property name="hibernate.c3p0.maxIdleTimeExcessConnections">600</property> <!-- 0 seconds is c3p0 default and means do not check. This is the num of seconds a connections in excess of minPoolSize are permitted to remain idle in the pool before being culled. -->
<property name="hibernate.c3p0.idleConnectionTestPeriod">30</property> <!-- 0 seconds is the c3p0 default and means do not check. (i.e. never test). Using only this provides the best performance even if individuals may occasionally receive an error message due to a connection being dropped. -->
<!-- Cache setup -->
<property name="hibernate.cache.use_second_level_cache">true</property>
<property name="hibernate.cache.use_query_cache">true</property>
<property name="hibernate.cache.region.factory_class">net.sf.ehcache.hibernate.EhCacheRegionFactory</property>
<!-- <property name="hibernate.cache.provider_class">net.sf.ehcache.hibernate.SingletonEhCacheProvider</property> -->
<property name="cache.use_minimal_puts">true</property>
<property name="hibernate.generate_statistics">true</property>
<property name="hibernate.cache.use_structured_entries">true</property>
<!-- Mapping files -->
...
</session-factory>
</hibernate-configuration>

Java Hibernate/C3P0 error: "Could not obtain connection metadata. An attempt by a client to checkout a Connection has timed out."

I'm trying to get some code I was passed up and running. It appears to use the Hibernate framework. I've gotten past most of the errors tweaking the configuration, but this one has me dead stumped.
It's trying to connect to two databases: gameapp and gamelog. Both exist. It seems to have issues connecting to gamelog, but none connecting to gameapp (later in the init, it connects to and loads other DBs just fine). Below, I've pasted the error and exception stack dump.
I imaging there's something else in the configs, so I've also included the configuration file for that db. I know this is very vague, but I'm hoping some pro can see the stupid mistake I'm missing.
<?xml version="1.0" encoding="GBK"?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="connection.url">jdbc:mysql://127.0.0.1:3306/gamelog</property>
<property name="connection.username">root</property>
<property name="connection.password"></property>
<property name="connection.useUnicode">true</property>
<property name="connection.characterEncoding">UTF-8</property>
<property name="hibernate.jdbc.batch_size">100</property>
<property name="jdbc.fetch_size">1</property>
<property name="hbm2ddl.auto">none</property><!-- update -->
<property name="connection.useUnicode">true</property>
<property name="show_sql">true</property>
<!-- c3p0-configuration -->
<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">10</property>
<property name="hibernate.c3p0.timeout">30</property>
<property name="hibernate.c3p0.idle_test_period">30</property>
<property name="hibernate.c3p0.max_statements">0</property>
<property name="hibernate.c3p0.acquire_increment">5</property>
</session-factory>
</hibernate-configuration>
Exception and stack trace:
2010-04-30 17:50:00,411 WARN [org.hibernate.cfg.SettingsFactory] - Could not obtain connection metadata
java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:35)
at org.hibernate.cfg.SettingsFactory.buildSettings(SettingsFactory.java:76)
at org.hibernate.cfg.Configuration.buildSettings(Configuration.java:1933)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1216)
at com.database.hibernate.util.HibernateFactory.<init>(Unknown Source)
at com.database.hibernate.util.HibernateUtil.<clinit>(Unknown Source)
at com.server.databaseop.goodOp.GoodOpImpl.initBreedGoods(Unknown Source)
at com.server.databaseop.goodOp.GoodOpImpl.access$000(Unknown Source)
at com.server.databaseop.goodOp.GoodOpImpl$1.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out while waiting to acquire a resource from com.mchange.v2.resourcepool.BasicResourcePool#ca470 -- timeout at awaitAvailable()
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
... 18 more

If you have set C3P0's "checkoutTimeout" property to something other than 0 you might be timing out too quickly (that was my problem, solution: bumped it to 2000 milliseconds from 500).
Alternatively, there's a workaround for this warning:
Set the hibernate.temp.use_jdbc_metadata_defaults property to false.
Found this in http://www.docjar.com/html/api/org/hibernate/cfg/SettingsFactory.java.html, though there may be side effects of not having Hibernate extract JDBC Metadata defaults.

Actually that's not even an authentication error. Is MySQL even running or bound to localhost?
does telnet 127.0.0.1 3306 work?
if so, install the mysql client on the box and try
mysql --user=root --ip=127.0.0.1
and see what happens

Check if you can connect to the gamelog mysql database on the command line with the root user and no password (!). As a side note, I'd recommend to set a password for root and to use a different account to connect to the database from your application, but that's another story.