We host a service (servlet running on jboss), which receives something like 5-6 requests per second. Every request needs to connect to mysql through hibernate. Most of our requests do selects, with an insert/update every 5th/6th request. The hibernate mysql connection gets timed out after mysql connection time out period (8 hours). Even after having a request pinging our service, every hour, the mysql connection sometimes gets randomly disconnected after a day or so. We want hibernate to reconnect in case the connection is disconnected due to any reason, and maintain the number of connections we want it to.
We tried C3P0, with the following configuration:
<property name=c3p0.acquire_increment>1</property>
<property name=c3p0.idle_test_period>3600</property>
<property name=c3p0.max_statements>0</property>
<property name=c3p0.min_size>1</property>
<property name=c3p0.timeout>3605</property>
<property name=hibernate.c3p0.preferredTestQuery>select 1;</property>
Hibernate connection_pool size was set to 1.
This made the timeout/disconnect problem go away. But we started facing another problem. Long waits. Normally the request serve time for us is 2-3ms for select and 40-50ms for insert/update. But, after using the above C3P0 configuration, we saw that every request completing after an update was taking more than a minute and sometimes even 4-5 minutes! From our logs, it seemed that randomly a select request would get stuck and will be able to complete only after an update request was received and served.
Above problem goes away if we remove the C3P0 config. Can somebody suggest what we might be doing wrong?
Here is the complete hibernate config for reference:
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="connection.url">jdbc:mysql://xxx.xxx.xxx</property>
<property name="connection.username">xxx</property>
<property name="connection.password">xxx</property>
<property name="connection.pool_size">1</property>
<property name="dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="current_session_context_class">thread</property>
<property name="hibernate.cache.use_query_cache">false</property>
<property name="hibernate.cache.use_second_level_cache">false</property>
<property name="show_sql">true</property>
<!-- Transaction isolation 2 = READ_COMMITTED -->
<property name="connection.isolation">2</property>
<property name="connection.autocommit">true</property>
<!-- configuration pool via c3p0-->
<property name="c3p0.acquire_increment">1</property>
<property name="c3p0.idle_test_period">3600</property> <!-- seconds -->
<property name="c3p0.max_size">1</property>
<property name="c3p0.max_statements">0</property>
<property name="c3p0.min_size">1</property>
<property name="c3p0.timeout">3605</property> <!-- seconds -->
<property name="hibernate.c3p0.preferredTestQuery">select 1;</property>
</session-factory>
</hibernate-configuration>
I'd recommend abandoning C3P0 and using the JNDI naming service and connection pooling that JBOSS provides.
Be sure that you are closing connections properly and returning them to the pool. Connect, query, close, all in the narrowest scope possible. No connection should be hanging around for eight hours.
Some more links might be pertinent: http://www.hibernate.org/117.html and http://www.informit.com/articles/article.aspx?p=353736&seqNum=4 regarding Hibernate and closing connections, and this MySQL bug that cites problems with MySQL, Hibernate, and connections: http://bugs.mysql.com/bug.php?id=10917
Something seems amiss with your configuration. All configuration parameters should be in the hibernate.c3p0 namespace, not c3p0.*.
But that's probably not the problem. I think most likely your pool is only one connection big and you are experiencing resource contention issues somewhere. Most likely not releasing a connection where you should, or a deadlock on some data. Try setting maxPoolsize to something higher, like 2 and see if the problem is mitigated any. This would probably mean you're not properly returning connections.
The JBoss Hibernate documentation specifically states to not use the Hibernate connection-pool manager for non-production systems (just like Hypersonic - I really don't know why they bother to ship known buggy components). As was previously commented, use the standard JBoss <datasource> setup and associated pool-manager (and configure as needed).
Where there is some misunderstanding is that the open() and close() methods are overridden by the connection manager. Even the Connection object isn't a direct connection to the database but rather a handle to the pool-manager.
The implementation upon getting a new connection (aka open) is to request a connection from the pool (which for efficiency is already opened and, if configured using the <prepared-statement-cache-size> property in the datasource definition, also has previous PreparedStatements still bound to that Connection). This call is uber efficient.
When close() is called on the Connection (again, implemented by the pool-manager), the connection reference state is simply changed to 'available' versus closing the actual database connection. This is also uber efficient. Therefore it is recommended that the smallest possible scope of open/exec/close calls be made to maximize multi-thread use of available connections in the pool.
Related
I realize that there are other questions similar to mine but I feel mine is a bit different so I'm going to ask it anyway. I've been using hibernate to develop a java web application and have run in to no problems when I run it on a localhost Tomcat server. I've, however, been running into problems after trying to deploy it on Heroku and using the free version of ClearDB as the website would randomly throw an exception. Specifically, I am getting a
org.hibernate.exception.JDBCConnectionException: Communications link failure
Things I've found on google but believe to not be my issue:
Database was idle for too long ( > 8hours). I've been continuously using the website so it definitely has not been idle for longer than 8 hours.
Too many connections to Database. I'm the only one currently connected to the database and the website says there is a maximum of 10 connections so I'm within their limit
My configuration file
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.password">PASSWORD</property>
<property name="hibernate.connection.url">jdbc:mysql://DBURL?reconnect=true</property>
<property name="hibernate.connection.username">USERNAME</property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
<property name= "hbm2ddl.auto">update</property>
<property name="show_sql">true</property>
</session-factory>
Thanks in advance
I'd be willing to bet that this goes away if you use JawsDB add-on instead of ClearDB:
$ heroku addons:destroy cleardb
$ heroku addons:create jawsdb:kitefin
I have so many problems with ClearDB.
If you must use ClearDB, I recommend adding a connectionTestQuery to your database connection pool configuration. I'm not sure how to do this generically, but if you are using HikariCP (which is great and you should use), you can add something like:
<property name="hibernate.hikari.connectionTestQuery">SELECT 1</property>
ClearDB routinely closes connections with idle/sleep status for more than 60 seconds. This happens when your system think it has a DB connection but the remote host has closed it. As the previous respondent mentioned, a quick connectionTestQuery should resolve this.
The following exceptions when happened, this means there is something wrong on the environment especially if those exceptions happened only on production environment with big number of users without any problem in testing environment with normal number of users.
1. Could not open connection: org.hibernate.exception.GenericJDBCException: Could not open connection.
2. java.sql.SQLRecoverableException: Closed Connection.
3. java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
My environment information:
windows server 2012, oracle DB, jboss 7.1 application server and java application use hibernate 4 with c3p0 connection pools
I searched a lot about the related topics and I found many useful topics but the most useful topics was those two topics. topic_1 topic_2
I simulate this case when I decrease the configuration maxStatements inside c3p0 configuration file but as the definition for this parameter which defined here , it is the size of c3p0's global PreparedStatement cache.
I have the following questions:
Why I got error cannot open connection when I decrease maxStatements attributes which should effect only on the performance since this is cache attribute and this error disappears when I decrease it?
Why I got also cannot open connection when I increase maxStatements value with big number? How can I know the limitation from the database?
Is there any limitation from the database can limit this attributes and how can I check it?
I checked the connection pool maximum attributes with the database session parameter and I found that, the database session parameter is greater than the maximum pool size for my application which is right. I checked also the active and idle connections on the database and no problem in those numbers. Is there any other DB attributes should be checked instead of session parameter for database to be sure the database can give me the configured number of connection pools?
My c3p0-config is the following:
<default-config>
<property name="checkoutTimeout">300</property>
<property name="idleConnectionTestPeriod">70</property>
<property name="initialPoolSize">50</property>
<property name="maxIdleTime">270</property>
<property name="maxPoolSize">500</property>
<property name="minPoolSize">50</property>
<property name="maxStatements">400</property>
<property name="maxStatementsPerConnection">0</property>
<property name="testConnectionOnCheckout">true</property>
<property name="testConnectionOnCheckin">true</property>
<property name="preferredTestQuery">SELECT 1 from dual</property>
<property name="acquireRetryAttempts">0</property>
<property name="acquireRetryDelay">1000</property>
<property name="breakAfterAcquireFailure">false</property>
<property name="unreturnedConnectionTimeout">270</property>
<property name="debugUnreturnedConnectionStackTraces">true</property>
</default-config>
when ever I keep my application idle for 10 or 15 hours, I will get Connection time out error. But when I frequently use my application then I couldn't able to see this error any time. Could any one please guide me whether I am making some thing wrong in the below code. This application is used by only two users and that too be not frequently.
<Context path="/****" reloadable="true">
<Resource
name="XXXX"
type="javax.sql.DataSource"
username="XXXX"
password="XXXX"
driverClassName="com.microsoft.sqlserver.jdbc.SQLServerDriver"
maxIdle="4"
maxWait="30000"
initialSize="2"
url="jdbc:sqlserver://localhost;database=XXXX"
maxActive="20"/>
</Context>
Unfortunately there is not so much information given in your example. Assumed that is a small piece of a Spring (Spring Framework) context configuration, i would prefer to configure a connection pooling for the database. This pool can hold an amount of idle connections for each request an can open new ones if no connection is available in the pool for the current request. I found this piece of XML in the MySql documentation.
<bean id="dataSource" destroy-method="close"
class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="${db.driver}"/>
<property name="url" value="${db.jdbcurl}"/>
<property name="username" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="initialSize" value="3"/>
</bean>
Please be aware that the datasource is of type org.apache.commons.dbcp.BasicDataSource. For further information please refer to the Documentation of MySql, Spring Framework and DBCP
Based on information provided on question, i can see:
Most probably, application is not properly handling connection i.e. it is creating connection, but then somehow failed to close connection properly in application code.
As per question, if only 2 users are using this app then probably you might not able to catch connection exhausted error which usually thrown when connection pool reaches max limit.
But if you increase app users you will see your connection reached max limit very quickly.
Check number of connections that got created after application starts and observe behavior. Check if inactive connections are getting closed or not.
Also, I don't see inactive connection timeout property in configuration. Can you check if you have this property.
This is an issue with databases, it gets shutdown/locked up after a long idle period. In MySQL it is 8 hours by default. You have to use a connection pooling library like C3P0 which has configuration to talk to database and turn on the connection if it is closed.
It is possible to increase the timeout amount in databases, but not recommended. Therefor go for a system like I mentioned above which can turn on the connection for you.
I have a web application that uses c3p0 as the connection pool. we use hibernate as the orm tool.
Recently, we have been getting connection timeout exceptions. To debug these exception, i enabled the logging fro C3p0 and got some information in logs. Can anybody help me in making complete sense out of it.
DEBUG 2012-08-05 14:43:52,590 [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] com.mchange.v2.c3p0.stmt.GooGooStatementCache: checkinAll(): com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache stats -- total size: 2; checked out: 0; num connections: 1; num keys: 2
from the above, i can observe that the total size of the connection pool is 2. No. of checked out connections are 0. Is this right? And what are num_connections and num keys in the above?
Thanks..
What you are seeing in the bit of log that you've quoted is a snapshot of the Statement cache, not the Connection pool. There were two cached PreparedStatements, which belonged to a single Connection, at the time that message was logged. Neither of the Statements was checked out/in use.
I hope this helps!
The exceptions might be due to incorrectly defining the C3P0 settings. Ensure that you link to the right jar file Your application must be linked to "hibernate-c3p0*.jar" and not to "c3p0*.jar". The hibernate.cfg file must have the hibernate.connection.provider_class property defined in order for the c3p0 settings to take effect. Below is a sample settings
<property name="hibernate.connection.provider_class"> org.hibernate.connection.C3P0ConnectionProvider </property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">10</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
The log entry specifies that there are 4 connections in the pool and all the 4 connections are available for the application to use. Running a query towards your database to list all the active connections will show the pool connections. For eg in Mysql you could run "show processlist;" to see these connections.
I have a project that uses EclipseLink 2.1.3 under Tomcat 7.
What's bugging me is that the connection to the server doesn't persist through all of the app's life.
For a ServerSession to start I have to access the application manually.
Is there a way to automatically start a ServerSession when the application deploys and keep it running at all times ? So that I can access the application after a long idle time without having to wait ?
Thank you!
EDIT:
I have the following lines in persistence.xml
<property name="eclipselink.jdbc.read-connections.min" value="5"/>
<property name="eclipselink.jdbc.read-connections.max" value="10"/>
<property name="eclipselink.jdbc.write-connections.min" value="6"/>
<property name="eclipselink.jdbc.write-connections.max" value="12"/>
Server applications usually use a JDBC connection pool to get connections and reuse them. Depending on the pool you use, you may configure it to keep a given number of connections open.