I have a web application that uses c3p0 as the connection pool. we use hibernate as the orm tool.
Recently, we have been getting connection timeout exceptions. To debug these exception, i enabled the logging fro C3p0 and got some information in logs. Can anybody help me in making complete sense out of it.
DEBUG 2012-08-05 14:43:52,590 [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] com.mchange.v2.c3p0.stmt.GooGooStatementCache: checkinAll(): com.mchange.v2.c3p0.stmt.GlobalMaxOnlyStatementCache stats -- total size: 2; checked out: 0; num connections: 1; num keys: 2
from the above, i can observe that the total size of the connection pool is 2. No. of checked out connections are 0. Is this right? And what are num_connections and num keys in the above?
Thanks..
What you are seeing in the bit of log that you've quoted is a snapshot of the Statement cache, not the Connection pool. There were two cached PreparedStatements, which belonged to a single Connection, at the time that message was logged. Neither of the Statements was checked out/in use.
I hope this helps!
The exceptions might be due to incorrectly defining the C3P0 settings. Ensure that you link to the right jar file Your application must be linked to "hibernate-c3p0*.jar" and not to "c3p0*.jar". The hibernate.cfg file must have the hibernate.connection.provider_class property defined in order for the c3p0 settings to take effect. Below is a sample settings
<property name="hibernate.connection.provider_class"> org.hibernate.connection.C3P0ConnectionProvider </property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">10</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
The log entry specifies that there are 4 connections in the pool and all the 4 connections are available for the application to use. Running a query towards your database to list all the active connections will show the pool connections. For eg in Mysql you could run "show processlist;" to see these connections.
Related
The following exceptions when happened, this means there is something wrong on the environment especially if those exceptions happened only on production environment with big number of users without any problem in testing environment with normal number of users.
1. Could not open connection: org.hibernate.exception.GenericJDBCException: Could not open connection.
2. java.sql.SQLRecoverableException: Closed Connection.
3. java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
My environment information:
windows server 2012, oracle DB, jboss 7.1 application server and java application use hibernate 4 with c3p0 connection pools
I searched a lot about the related topics and I found many useful topics but the most useful topics was those two topics. topic_1 topic_2
I simulate this case when I decrease the configuration maxStatements inside c3p0 configuration file but as the definition for this parameter which defined here , it is the size of c3p0's global PreparedStatement cache.
I have the following questions:
Why I got error cannot open connection when I decrease maxStatements attributes which should effect only on the performance since this is cache attribute and this error disappears when I decrease it?
Why I got also cannot open connection when I increase maxStatements value with big number? How can I know the limitation from the database?
Is there any limitation from the database can limit this attributes and how can I check it?
I checked the connection pool maximum attributes with the database session parameter and I found that, the database session parameter is greater than the maximum pool size for my application which is right. I checked also the active and idle connections on the database and no problem in those numbers. Is there any other DB attributes should be checked instead of session parameter for database to be sure the database can give me the configured number of connection pools?
My c3p0-config is the following:
<default-config>
<property name="checkoutTimeout">300</property>
<property name="idleConnectionTestPeriod">70</property>
<property name="initialPoolSize">50</property>
<property name="maxIdleTime">270</property>
<property name="maxPoolSize">500</property>
<property name="minPoolSize">50</property>
<property name="maxStatements">400</property>
<property name="maxStatementsPerConnection">0</property>
<property name="testConnectionOnCheckout">true</property>
<property name="testConnectionOnCheckin">true</property>
<property name="preferredTestQuery">SELECT 1 from dual</property>
<property name="acquireRetryAttempts">0</property>
<property name="acquireRetryDelay">1000</property>
<property name="breakAfterAcquireFailure">false</property>
<property name="unreturnedConnectionTimeout">270</property>
<property name="debugUnreturnedConnectionStackTraces">true</property>
</default-config>
when ever I keep my application idle for 10 or 15 hours, I will get Connection time out error. But when I frequently use my application then I couldn't able to see this error any time. Could any one please guide me whether I am making some thing wrong in the below code. This application is used by only two users and that too be not frequently.
<Context path="/****" reloadable="true">
<Resource
name="XXXX"
type="javax.sql.DataSource"
username="XXXX"
password="XXXX"
driverClassName="com.microsoft.sqlserver.jdbc.SQLServerDriver"
maxIdle="4"
maxWait="30000"
initialSize="2"
url="jdbc:sqlserver://localhost;database=XXXX"
maxActive="20"/>
</Context>
Unfortunately there is not so much information given in your example. Assumed that is a small piece of a Spring (Spring Framework) context configuration, i would prefer to configure a connection pooling for the database. This pool can hold an amount of idle connections for each request an can open new ones if no connection is available in the pool for the current request. I found this piece of XML in the MySql documentation.
<bean id="dataSource" destroy-method="close"
class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="${db.driver}"/>
<property name="url" value="${db.jdbcurl}"/>
<property name="username" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="initialSize" value="3"/>
</bean>
Please be aware that the datasource is of type org.apache.commons.dbcp.BasicDataSource. For further information please refer to the Documentation of MySql, Spring Framework and DBCP
Based on information provided on question, i can see:
Most probably, application is not properly handling connection i.e. it is creating connection, but then somehow failed to close connection properly in application code.
As per question, if only 2 users are using this app then probably you might not able to catch connection exhausted error which usually thrown when connection pool reaches max limit.
But if you increase app users you will see your connection reached max limit very quickly.
Check number of connections that got created after application starts and observe behavior. Check if inactive connections are getting closed or not.
Also, I don't see inactive connection timeout property in configuration. Can you check if you have this property.
This is an issue with databases, it gets shutdown/locked up after a long idle period. In MySQL it is 8 hours by default. You have to use a connection pooling library like C3P0 which has configuration to talk to database and turn on the connection if it is closed.
It is possible to increase the timeout amount in databases, but not recommended. Therefor go for a system like I mentioned above which can turn on the connection for you.
I'm managing a Java Web Application with Spring and Hibernate.
I use Spring and Hibernate tools to handle the persistence level, so I don't need to commit\rollback my transactions.
The application is concurrent, so the users can modify the same records and I decided to use Read Committed as isolation level.
The problem is sometimes I find JDBC errors in the log, and all the next requests go in the same error, blocking the application behaviour.
These are the components involved in the transaction management:
#Bean
public SpringLocalSessionFactoryBean sessionFactory(DataSource dataSource){
SpringLocalSessionFactoryBean bean = new SpringLocalSessionFactoryBean();
bean.setConfigLocation(new ClassPathResource("hibernate.cfg.xml"));
bean.setDataSource(dataSource);
return bean;
}
#Bean
public HibernateTransactionManager transactionManager(SessionFactory sessionFactory){
HibernateTransactionManager tm = new HibernateTransactionManager();
tm.setSessionFactory(sessionFactory);
return tm;
}
In the db session monitor, when this stuff happens, I got an INACTIVE transaction.
The error I get is the following:
WARN - (SqlExceptionHelper.java:144) - SQL Error: 0, SQLState: null
14/03/2016 15:46:06 - ERROR - (SqlExceptionHelper.java:146) - Connection oracle.jdbc.driver.T4CConnection#1a6d7ad6 is closed.
14/03/2016 15:46:06 - ERROR - (AutoCompleterController.java:73) - could not prepare statement
org.hibernate.exception.GenericJDBCException: could not prepare statement
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:196)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:160)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1885)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1862)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1839)
at org.hibernate.loader.Loader.doQuery(Loader.java:910)
Caused by: java.sql.SQLException: Connection oracle.jdbc.driver.T4CConnection#1a6d7ad6 is closed.
at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.checkOpen(DelegatingConnection.java:398)
at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:279)
at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:162)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:186)
... 97 more
The problem is the transactions and the connections should be automatically opened and closed... And I expect that transactions that should fail for concurrent modification to get a rollback... But it seems they get inactive.
I attach my hibernate config.
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.dialect">org.hibernate.dialect.Oracle10gDialect</property>
<!-- <property name="hibernate.hbm2ddl.auto">update</property> -->
<property name="hibernate.connection.isolation">2</property>
<!-- Disable the second-level cache -->
<property name="hibernate.cache.use_second_level_cache">false</property>
<property name="hibernate.id.new_generator_mappings">true</property>
<property name="hibernate.connection.autocommit">false</property>
<!-- Show and print nice SQL on stdout -->
<property name="hibernate.show_sql">false</property>
<property name="hibernate.format_sql">false</property>
<property name="hibernate.use_sql_comments">false</property>
<property name="hibernate.generate_statistics">false</property>
</session-factory>
</hibernate-configuration>
As connection library I use ojdbc.
Any help will be appreciated. I don't know where to check anymore.
PS: I add that this error spawns like once every 2 days.
EDIT: Just another integration, this is what I have on my server.xml, could it be related to something here?
<Resource name="jdbc/ToolSfDB" global="jdbc/ToolSfDB" auth="Container" type="javax.sql.DataSource"
driverClassName="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:#//oracle01-internal.local:1521/orcl01"
username="tools"
password="mypwd"
maxActive="10"
maxIdle="2"
minIdle="1"
suspectTimeout="60"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="60000"
validationQuery="select 1 from dual"
validationInterval="30000"
testOnBorrow="true"
removeAbandoned="true"
removeAbandonedTimeout="60"
abandonWhenPercentageFull="10"
maxWait="10000"
maxAge="60000"/>
It seems that the problem of transaction happens because it failed to open connection with oracle. If you are using oracle 10g as you have configured in hibernate.cfg.xml
<session-factory>
<property name="hibernate.dialect">org.hibernate.dialect.Oracle10gDialect</property>
And
You are using oracle thin driver:
<Resource name="jdbc/ToolSfDB" global="jdbc/ToolSfDB" auth="Container" type="javax.sql.DataSource"
driverClassName="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:#//oracle01-internal.local:1521/orcl01"
The driver class might not be configured properly.
Use oracle.jdbc.driver.OracleDriver instead of oracle.jdbc.OracleDriver.
<Resource name="jdbc/ToolSfDB" global="jdbc/ToolSfDB" auth="Container" type="javax.sql.DataSource"
driverClassName="oracle.jdbc.driver.OracleDriver"
url="jdbc:oracle:thin:#//oracle01-internal.local:1521/orcl01"
Notes: Make sure of the oracle jar version. it will work for ojdbc6.jar.
References:
hibernate cfg xml settings for derby oracle and h2
difference between oracle jdbc driver classes
configuring jdbc pool high concurrency
At first, I want to give you couple of suggestions.
Please check version of oracle driver. If not same then use correct version of oracle driver. As you are using oracle 11, so download ojdbc6.jar file and use it.
Sometimes ORACLE_HOME was not set. So when the java application accessing the database, it causes the following error. So check that ORACLE_HOME is set or not.
Caused by: java.sql.SQLException: Connection
oracle.jdbc.driver.T4CConnection#1a6d7ad6 is closed.
From the error,
it is defined that It means the connection was successfully established at some point, but when you try to commit right there, the connection was no longer open. The parameters you mentioned sound like connection pool settings. In your hibernate.cfg.xml file add
<property name="hibernate.connection.provider_class">org.hibernate.c3p0.internal.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.timeout">1800</property>
<property name="hibernate.c3p0.max_statements">50</property>
Related Link:
hibernate connection pool configuration with c3p0 example
docs
If you want to add pool size in properties file, then follow this tutorial.
4.Are you using oracle 10g? if yes then it's ok otherwise Oracle10gDialect need to be changed for lower version.
<property name="hibernate.dialect">org.hibernate.dialect.Oracle10gDialect</property>
Use hibernate.hbm2ddl.auto = validate mode. It is used in oracle 11g.
UPDATE:
As you are not using c3p0 pooling, then configure the pool by setting in properties file
accessToUnderlyingConnectionAllowed = true
and then it may work.
Default is false, it is a potential dangerous operation and
misbehaving programs can do harmful things. (closing the underlying or
continue using it when the guarded connection is already closed) Be
careful and only use when you need direct access to driver specific
extensions
In your server.xml
please change
initialSize="10"
maxActive="100"
maxIdle="50"
minIdle="10"
The initialSize=10 is the number of connections that will be established when the connection pool is created
The maxActive=100 is the maximum number of established connections to the database.
The minIdle=10 is the minimum number of connections always established after the connection pool has reached this size.
The maxIdle attribute is a little bit trickier. It behaves differently depending on if the pool sweeper is enabled.
Please study configuring-jdbc-pool-high-concurrency tutorial for better understanding to configure server.xml. If you study this tutorial, I hope you can solve it by yours.
I'm investigating further more, and I suspect a scenario that could cause this issue.
Suppose I have a transactional method in the service layer. This method can call different objects on the DAO layer, each of them having his own connections.
Suppose the first call to DAO1 goes well, and updates a record.
Now another user make some other operations, and when my transactional method calls for DAO2 operation, it finds no connections available, causing the issue and leaving the transaction inactive after a timeout.
Could it be the issue?
If so, I found out that Tomcat Jdbc configuration has other parameters that I cound add:
rollbackOnReturn
(boolean) If autoCommit==false then the pool can terminate the transaction by calling rollback on the connection as it is returned to the pool Default value is false.
commitOnReturn
(boolean) If autoCommit==false then the pool can complete the transaction by calling commit on the connection as it is returned to the pool If rollbackOnReturn==true then this attribute is ignored. Default value is false.
At this point, could the first parameter set to true be the solution?
you said that I add that this error spawns like once every 2 days. if your database trafic hight, I think manage connection pools for example use C3P0 and manage total connections,timeouts ... etc
for example
<property name="hibernate.c3p0.timeout">500</property>
this idle connection is removed from the pool (500 second)
and more knowledge here.
I ran into similar issue, After tweaking removeAbandonedTimeout got it working. Here is how my configuration looks now.
maxActive="60"
maxIdle="30"
maxWait="12000"
validationQuery="select 1"
validationInterval="30000"
testOnBorrow="true"
testOnReturn="false"
testWhileIdle="true"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="600000"
logAbandoned="true"
removeAbandoned="true"
removeAbandonedTimeout="180"
Explanation:
Earlier my removeAbandonedTimeout is configured as 60. But some of my queries are taking more that 60 seconds to execute, So while the query is being executed the connection is closed by the pool as the abandoned timeout is reached. After query execution, spring transaction tries to commit that connection which is failing because the connection has been already closed.
For now i increased the removeAbandonedTimeout to 180, while we are trying optimize the query that is taking more time.
I have a machine running a java app talking to a mysql instance running on the same instance. the app
uses jdbc4 drivers from mysql. I keep getting com.mysql.jdbc.exceptions.jdbc4.CommunicationsException
at random times.
Here is the whole message.
Could not open JDBC Connection for transaction; nested exception is
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was25899 milliseconds ago.The last packet sent successfully to the server was 25899 milliseconds ago, which is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
For mysql, the value of global 'wait_timeout' and 'interactive_timeout' is set to 3600 seconds and 'connect_timeout' is set to 60 secs. the wait timeout value is much higher than the 26 secs(25899 msecs). mentioned in the exception trace.
I use dbcp for connection pooling and here is spring bean config for the datasource.
<bean id="dataSource" destroy-method="close" class="org.apache.commons.dbcp.BasicDataSource" >
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/db"/>
<property name="username" value="xxx"/>
<property name="password" value="xxx" />
<property name="poolPreparedStatements" value="false" />
<property name="maxActive" value="3" />
<property name="maxIdle" value="3" />
</bean>
Any idea why this could be happening? Will using c3p0 solve the problem ?
Try setting up the Apache Commons DBCP correctly.
You need to set:
validationQuery to SELECT 1+1
testOnBorrow to true
That should fix the problem.
Can you describe how your app is handling connection pooling? I doubt that autoReconnect=true in the JDBC driver would re-pool connections from your app. The app needs to reconnect when it loses a connection.
I'd follow the advice in the exception. You should consider either:
expiring and/or testing connection validity before use in your application,
increasing the server configured values for client timeouts, or
using the Connector/J connection property 'autoReconnect=true' to avoid this problem. Try adding that to your connection URL (consult the docs for the exact syntax) and see if it helps.
I doubt that C3P0 will be that much better than the DBCP that you're already using. The exception is giving you some specific advice. You've tried #3. What about the other two?
I know how to ask WebLogic to check connections before using them. You should find out how to do the same with Tomcat.
I have seen before that Windows machines which have been moved on the network have had trouble with connecting to themselves.
Is there any connectivity problems outside the JVM - i.e. mysql client connecting to the server, and timing out, etc?
We host a service (servlet running on jboss), which receives something like 5-6 requests per second. Every request needs to connect to mysql through hibernate. Most of our requests do selects, with an insert/update every 5th/6th request. The hibernate mysql connection gets timed out after mysql connection time out period (8 hours). Even after having a request pinging our service, every hour, the mysql connection sometimes gets randomly disconnected after a day or so. We want hibernate to reconnect in case the connection is disconnected due to any reason, and maintain the number of connections we want it to.
We tried C3P0, with the following configuration:
<property name=c3p0.acquire_increment>1</property>
<property name=c3p0.idle_test_period>3600</property>
<property name=c3p0.max_statements>0</property>
<property name=c3p0.min_size>1</property>
<property name=c3p0.timeout>3605</property>
<property name=hibernate.c3p0.preferredTestQuery>select 1;</property>
Hibernate connection_pool size was set to 1.
This made the timeout/disconnect problem go away. But we started facing another problem. Long waits. Normally the request serve time for us is 2-3ms for select and 40-50ms for insert/update. But, after using the above C3P0 configuration, we saw that every request completing after an update was taking more than a minute and sometimes even 4-5 minutes! From our logs, it seemed that randomly a select request would get stuck and will be able to complete only after an update request was received and served.
Above problem goes away if we remove the C3P0 config. Can somebody suggest what we might be doing wrong?
Here is the complete hibernate config for reference:
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="connection.url">jdbc:mysql://xxx.xxx.xxx</property>
<property name="connection.username">xxx</property>
<property name="connection.password">xxx</property>
<property name="connection.pool_size">1</property>
<property name="dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="current_session_context_class">thread</property>
<property name="hibernate.cache.use_query_cache">false</property>
<property name="hibernate.cache.use_second_level_cache">false</property>
<property name="show_sql">true</property>
<!-- Transaction isolation 2 = READ_COMMITTED -->
<property name="connection.isolation">2</property>
<property name="connection.autocommit">true</property>
<!-- configuration pool via c3p0-->
<property name="c3p0.acquire_increment">1</property>
<property name="c3p0.idle_test_period">3600</property> <!-- seconds -->
<property name="c3p0.max_size">1</property>
<property name="c3p0.max_statements">0</property>
<property name="c3p0.min_size">1</property>
<property name="c3p0.timeout">3605</property> <!-- seconds -->
<property name="hibernate.c3p0.preferredTestQuery">select 1;</property>
</session-factory>
</hibernate-configuration>
I'd recommend abandoning C3P0 and using the JNDI naming service and connection pooling that JBOSS provides.
Be sure that you are closing connections properly and returning them to the pool. Connect, query, close, all in the narrowest scope possible. No connection should be hanging around for eight hours.
Some more links might be pertinent: http://www.hibernate.org/117.html and http://www.informit.com/articles/article.aspx?p=353736&seqNum=4 regarding Hibernate and closing connections, and this MySQL bug that cites problems with MySQL, Hibernate, and connections: http://bugs.mysql.com/bug.php?id=10917
Something seems amiss with your configuration. All configuration parameters should be in the hibernate.c3p0 namespace, not c3p0.*.
But that's probably not the problem. I think most likely your pool is only one connection big and you are experiencing resource contention issues somewhere. Most likely not releasing a connection where you should, or a deadlock on some data. Try setting maxPoolsize to something higher, like 2 and see if the problem is mitigated any. This would probably mean you're not properly returning connections.
The JBoss Hibernate documentation specifically states to not use the Hibernate connection-pool manager for non-production systems (just like Hypersonic - I really don't know why they bother to ship known buggy components). As was previously commented, use the standard JBoss <datasource> setup and associated pool-manager (and configure as needed).
Where there is some misunderstanding is that the open() and close() methods are overridden by the connection manager. Even the Connection object isn't a direct connection to the database but rather a handle to the pool-manager.
The implementation upon getting a new connection (aka open) is to request a connection from the pool (which for efficiency is already opened and, if configured using the <prepared-statement-cache-size> property in the datasource definition, also has previous PreparedStatements still bound to that Connection). This call is uber efficient.
When close() is called on the Connection (again, implemented by the pool-manager), the connection reference state is simply changed to 'available' versus closing the actual database connection. This is also uber efficient. Therefore it is recommended that the smallest possible scope of open/exec/close calls be made to maximize multi-thread use of available connections in the pool.