Develop Reference

JDBC Pool Connection Validation

I experienced an outage with my application the other day and I need to understand how to avoid this in the future.
We have a Java based web application running on Tomcat 7. The application connected to several different data sources including an Oracle database.
Here are the details, the Oracle database server went down and had to be rebooted. My simple understanding tells me this would have severed the application's connections to the database, and in fact the users reported errors in the application.
The Oracle data source is setup in Tomcat's sever.xml as a GlobalNaming Resource:
<Resource name="datasource"
auth="Container"
type="javax.sql.DataSource"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
....
initialSize="4"
minIdle="2"
maxIdle="8"
maxActive="8"
maxAge="28800000"
maxWait="30000"
testOnBorrow="false"
testOnReturn="false"
testWhileIdle="false"
validationQuery="SELECT 1 FROM dual"
validationQueryTimeout="10"
validationInterval="600000"
timeBetweenEvictionRunsMillis="60000"
minEvictableIdleTimeMillis="900000"
removeAbandoned="true"
removeAbandonedTimeout="60"
logAbandoned="true"
jmxEnabled="true" />
So here is what I understand regarding connection validation.
Connections are not validated while idle (testWhileIdle = false), when borrowed (testOnBorrow = false), when returned (testOnReturn = false)
The PoolSweeper is enabled because timeBetweenEvictionRunsMillis > 0, removeAbandoned is true, and removeAbandonedTimeout > 0
What confuses me is the inclusion of the validation query and the validationInterval > 0. Since all of the tests are disabled, does the pool sweeper then use the validation query to check the connections? Or is the validation query irrelevant?
So when the database server went down, I believe the connection pool would not have tried to reestablish connections because there are no validation tests enabled. In my opinion, had testOnBorrow been enabled then when the database server came back up valid connections would have been established and the web application (meaning tomcat) would not have required a restart.
Do I have a correct understanding of how connection validation works?

Let us take a look at the relevant part of your configuration to avoid invalid connections in your pool.
maxAge="28800000"
The connections regardless if valid or not will be kept open for 8 hrs. After 8 hrs the connection is closed and a new connection will be established if requested and no free connection is available in the pool. [1]
testOnBorrow="false"
testOnReturn="false"
testWhileIdle="false"
A connection in the pool will not be tested if it is valid when or while it's been borrowed, returned and idle. [1]
validationInterval="600000"
This property has no effect since all connection tests are set to false. This interval defines when a connection is needed to be tested. In your example, a connection will be tested every 10 minutes in case a test property would be set to true. [1]
An invalid connection can stay open up to 8 hrs with your current configuration. To enable validation tests of opened connections you have to set at least one test property (testOnBorrow, testOnReturn, testWhileIdle) to true.
Please note, in case of validationInterval="600000" a connection test/validation will be done every 10 minutes. So, an invalid connection could be up to 10 minutes available in the pool regardless which test property is set.
For more information about the individual properties please take a look at [1]: Apache Tomcat 7: The Tomcat JDBC Connection Pool.

you must lower the maxAge="28800000" parameter, in addition to this your instance crashes due to application errors and you should use jdbc interceptor

Could not open connection - jboss config - maxStatements

The following exceptions when happened, this means there is something wrong on the environment especially if those exceptions happened only on production environment with big number of users without any problem in testing environment with normal number of users.
1. Could not open connection: org.hibernate.exception.GenericJDBCException: Could not open connection.
2. java.sql.SQLRecoverableException: Closed Connection.
3. java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
My environment information:
windows server 2012, oracle DB, jboss 7.1 application server and java application use hibernate 4 with c3p0 connection pools
I searched a lot about the related topics and I found many useful topics but the most useful topics was those two topics. topic_1 topic_2
I simulate this case when I decrease the configuration maxStatements inside c3p0 configuration file but as the definition for this parameter which defined here , it is the size of c3p0's global PreparedStatement cache.
I have the following questions:
Why I got error cannot open connection when I decrease maxStatements attributes which should effect only on the performance since this is cache attribute and this error disappears when I decrease it?
Why I got also cannot open connection when I increase maxStatements value with big number? How can I know the limitation from the database?
Is there any limitation from the database can limit this attributes and how can I check it?
I checked the connection pool maximum attributes with the database session parameter and I found that, the database session parameter is greater than the maximum pool size for my application which is right. I checked also the active and idle connections on the database and no problem in those numbers. Is there any other DB attributes should be checked instead of session parameter for database to be sure the database can give me the configured number of connection pools?
My c3p0-config is the following:
<default-config>
<property name="checkoutTimeout">300</property>
<property name="idleConnectionTestPeriod">70</property>
<property name="initialPoolSize">50</property>
<property name="maxIdleTime">270</property>
<property name="maxPoolSize">500</property>
<property name="minPoolSize">50</property>
<property name="maxStatements">400</property>
<property name="maxStatementsPerConnection">0</property>
<property name="testConnectionOnCheckout">true</property>
<property name="testConnectionOnCheckin">true</property>
<property name="preferredTestQuery">SELECT 1 from dual</property>
<property name="acquireRetryAttempts">0</property>
<property name="acquireRetryDelay">1000</property>
<property name="breakAfterAcquireFailure">false</property>
<property name="unreturnedConnectionTimeout">270</property>
<property name="debugUnreturnedConnectionStackTraces">true</property>
</default-config>

Getting Connection timeout error when application is idle

when ever I keep my application idle for 10 or 15 hours, I will get Connection time out error. But when I frequently use my application then I couldn't able to see this error any time. Could any one please guide me whether I am making some thing wrong in the below code. This application is used by only two users and that too be not frequently.
<Context path="/****" reloadable="true">
<Resource
name="XXXX"
type="javax.sql.DataSource"
username="XXXX"
password="XXXX"
driverClassName="com.microsoft.sqlserver.jdbc.SQLServerDriver"
maxIdle="4"
maxWait="30000"
initialSize="2"
url="jdbc:sqlserver://localhost;database=XXXX"
maxActive="20"/>
</Context>

Unfortunately there is not so much information given in your example. Assumed that is a small piece of a Spring (Spring Framework) context configuration, i would prefer to configure a connection pooling for the database. This pool can hold an amount of idle connections for each request an can open new ones if no connection is available in the pool for the current request. I found this piece of XML in the MySql documentation.
<bean id="dataSource" destroy-method="close"
class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="${db.driver}"/>
<property name="url" value="${db.jdbcurl}"/>
<property name="username" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="initialSize" value="3"/>
</bean>
Please be aware that the datasource is of type org.apache.commons.dbcp.BasicDataSource. For further information please refer to the Documentation of MySql, Spring Framework and DBCP

Based on information provided on question, i can see:
Most probably, application is not properly handling connection i.e. it is creating connection, but then somehow failed to close connection properly in application code.
As per question, if only 2 users are using this app then probably you might not able to catch connection exhausted error which usually thrown when connection pool reaches max limit.
But if you increase app users you will see your connection reached max limit very quickly.
Check number of connections that got created after application starts and observe behavior. Check if inactive connections are getting closed or not.
Also, I don't see inactive connection timeout property in configuration. Can you check if you have this property.

This is an issue with databases, it gets shutdown/locked up after a long idle period. In MySQL it is 8 hours by default. You have to use a connection pooling library like C3P0 which has configuration to talk to database and turn on the connection if it is closed.
It is possible to increase the timeout amount in databases, but not recommended. Therefor go for a system like I mentioned above which can turn on the connection for you.

Connection pool C3P0 logging

I have a web application that uses c3p0 as the connection pool. we use hibernate as the orm tool.
Recently, we have been getting connection timeout exceptions. To debug these exception, i enabled the logging fro C3p0 and got some information in logs. Can anybody help me in making complete sense out of it.
DEBUG 2012-08-05 14:43:52,590 [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] com.mchange.v2.c3p0.stmt.GooGooStatementCache: checkinAll(): com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache stats -- total size: 2; checked out: 0; num connections: 1; num keys: 2
from the above, i can observe that the total size of the connection pool is 2. No. of checked out connections are 0. Is this right? And what are num_connections and num keys in the above?
Thanks..

What you are seeing in the bit of log that you've quoted is a snapshot of the Statement cache, not the Connection pool. There were two cached PreparedStatements, which belonged to a single Connection, at the time that message was logged. Neither of the Statements was checked out/in use.
I hope this helps!

The exceptions might be due to incorrectly defining the C3P0 settings. Ensure that you link to the right jar file Your application must be linked to "hibernate-c3p0*.jar" and not to "c3p0*.jar". The hibernate.cfg file must have the hibernate.connection.provider_class property defined in order for the c3p0 settings to take effect. Below is a sample settings
<property name="hibernate.connection.provider_class"> org.hibernate.connection.C3P0ConnectionProvider </property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">10</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
The log entry specifies that there are 4 connections in the pool and all the 4 connections are available for the application to use. Running a query towards your database to list all the active connections will show the pool connections. For eg in Mysql you could run "show processlist;" to see these connections.

Hibernate/mysql connection pooling

We host a service (servlet running on jboss), which receives something like 5-6 requests per second. Every request needs to connect to mysql through hibernate. Most of our requests do selects, with an insert/update every 5th/6th request. The hibernate mysql connection gets timed out after mysql connection time out period (8 hours). Even after having a request pinging our service, every hour, the mysql connection sometimes gets randomly disconnected after a day or so. We want hibernate to reconnect in case the connection is disconnected due to any reason, and maintain the number of connections we want it to.
We tried C3P0, with the following configuration:
<property name=c3p0.acquire_increment>1</property>
<property name=c3p0.idle_test_period>3600</property>
<property name=c3p0.max_statements>0</property>
<property name=c3p0.min_size>1</property>
<property name=c3p0.timeout>3605</property>
<property name=hibernate.c3p0.preferredTestQuery>select 1;</property>
Hibernate connection_pool size was set to 1.
This made the timeout/disconnect problem go away. But we started facing another problem. Long waits. Normally the request serve time for us is 2-3ms for select and 40-50ms for insert/update. But, after using the above C3P0 configuration, we saw that every request completing after an update was taking more than a minute and sometimes even 4-5 minutes! From our logs, it seemed that randomly a select request would get stuck and will be able to complete only after an update request was received and served.
Above problem goes away if we remove the C3P0 config. Can somebody suggest what we might be doing wrong?
Here is the complete hibernate config for reference:
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="connection.url">jdbc:mysql://xxx.xxx.xxx</property>
<property name="connection.username">xxx</property>
<property name="connection.password">xxx</property>
<property name="connection.pool_size">1</property>
<property name="dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="current_session_context_class">thread</property>
<property name="hibernate.cache.use_query_cache">false</property>
<property name="hibernate.cache.use_second_level_cache">false</property>
<property name="show_sql">true</property>
<!-- Transaction isolation 2 = READ_COMMITTED -->
<property name="connection.isolation">2</property>
<property name="connection.autocommit">true</property>
<!-- configuration pool via c3p0-->
<property name="c3p0.acquire_increment">1</property>
<property name="c3p0.idle_test_period">3600</property> <!-- seconds -->
<property name="c3p0.max_size">1</property>
<property name="c3p0.max_statements">0</property>
<property name="c3p0.min_size">1</property>
<property name="c3p0.timeout">3605</property> <!-- seconds -->
<property name="hibernate.c3p0.preferredTestQuery">select 1;</property>
</session-factory>
</hibernate-configuration>

I'd recommend abandoning C3P0 and using the JNDI naming service and connection pooling that JBOSS provides.
Be sure that you are closing connections properly and returning them to the pool. Connect, query, close, all in the narrowest scope possible. No connection should be hanging around for eight hours.
Some more links might be pertinent: http://www.hibernate.org/117.html and http://www.informit.com/articles/article.aspx?p=353736&seqNum=4 regarding Hibernate and closing connections, and this MySQL bug that cites problems with MySQL, Hibernate, and connections: http://bugs.mysql.com/bug.php?id=10917

Something seems amiss with your configuration. All configuration parameters should be in the hibernate.c3p0 namespace, not c3p0.*.
But that's probably not the problem. I think most likely your pool is only one connection big and you are experiencing resource contention issues somewhere. Most likely not releasing a connection where you should, or a deadlock on some data. Try setting maxPoolsize to something higher, like 2 and see if the problem is mitigated any. This would probably mean you're not properly returning connections.

The JBoss Hibernate documentation specifically states to not use the Hibernate connection-pool manager for non-production systems (just like Hypersonic - I really don't know why they bother to ship known buggy components). As was previously commented, use the standard JBoss <datasource> setup and associated pool-manager (and configure as needed).
Where there is some misunderstanding is that the open() and close() methods are overridden by the connection manager. Even the Connection object isn't a direct connection to the database but rather a handle to the pool-manager.
The implementation upon getting a new connection (aka open) is to request a connection from the pool (which for efficiency is already opened and, if configured using the <prepared-statement-cache-size> property in the datasource definition, also has previous PreparedStatements still bound to that Connection). This call is uber efficient.
When close() is called on the Connection (again, implemented by the pool-manager), the connection reference state is simply changed to 'available' versus closing the actual database connection. This is also uber efficient. Therefore it is recommended that the smallest possible scope of open/exec/close calls be made to maximize multi-thread use of available connections in the pool.