I am using c3p0 for connection pooling and facing "Closed connection" error with large data sets. I expect my transaction to be atomic and to run for almost max 2 hours.
For large data sets, which take around 40-45 minutes in processing. When I try to persist this data in the DB, I get exceptions in the following sequence:
[WARN] [c3p0] A PooledConnection that has already signalled a Connection error is still in use!
[WARN] [c3p0] Another error has occurred [ java.sql.SQLRecoverableException: Closed Connection ] which will not be reported to listeners!
[ERROR] org.hibernate.transaction.JDBCTransaction: Could not toggle autocommit
java.sql.SQLRecoverableException: Closed Connection
[ERROR] org.hibernate.transaction.JDBCTransaction: JDBC rollback failed
[ERROR] org.springframework.transaction.interceptor.TransactionInterceptor: Application exception overridden by rollback exception.
I have tried exploring solution a lot and tried to update my configuration accordingly.
Here is my C3P0 configuration:
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="driverClass" value="${...}" />
<property name="jdbcUrl" value="${...}" />
<property name="user" value="${...}" />
<property name="password" value="${...}" />
<property name="initialPoolSize" value="2" />
<property name="minPoolSize" value="2" />
<property name="maxPoolSize" value="50" />
<property name="maxIdleTime" value="6000" />
<property name="maxIdleTimeExcessConnections" value="1800" />
<property name="idleConnectionTestPeriod" value="3600" />
<property name="checkoutTimeout" value="60000" />
<property name="acquireRetryAttempts" value="0" />
<property name="acquireRetryDelay" value="1000" />
<property name="numHelperThreads" value="1" />
<property name="preferredTestQuery" value="SELECT 1 FROM DUAL" />
</bean>
Please help.
Related
We are creating a Spring boot web app.
DB : JDBC Template and DBCP connection pool.
Java code: A runnable is called in Executors.newSingleThreadScheduledExecutor();
Time interval: 2 min
The code in runnable hits DB using JDBCTemplate.query().
Issue: The heap usage increases to several GBs in few min.
Any Pointers would be helpful to identify the memory leak.
Note: If we comment the JDBCTemplate.query() , memory usage is constant.
Settings of DBCP:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="oracle.jdbc.OracleDriver" />
<property name="url" value="${batch.jdbc.url}" />
<property name="username" value="******" />
<property name="password" value="******" />
<property name="connectionProperties" value="defaultRowPrefetch=10000;defaultBatchValue=200;" />
<property name="minIdle" value="10" />
<property name="maxIdle" value="12" />
<property name="maxActive" value="100" />
<property name="accessToUnderlyingConnectionAllowed" value="true" />
<property name="initialSize" value="${batch.jdbc.pool.size}"/>
<property name="validationQuery" value="select 1 from dual" />
<property name="validationQueryTimeout" value="5" />
<property name="timeBetweenEvictionRunsMillis" value="120000" />
<property name="minEvictableIdleTimeMillis" value="60000" />
<property name="testOnBorrow" value="true" />
</bean>
Suspect from Eclipse MAT report
One instance of "org.apache.commons.pool.impl.GenericObjectPool" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader # 0x7fc1d90124c8" occupies 1,421,543,264 (94.69%) bytes. The memory is accumulated in one instance of "org.apache.commons.pool.impl.GenericObjectPool" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader # 0x7fc1d90124c8".
I'm experiencing a weird behaviour with our Java web application, configured to access a PostgreSQL database through C3p0 ComboPooledDataSource.
First of all, PostgreSQL server installation has default parameters, we didn't need to change any variable in it..
We run our queries using Spring JdbcTemplate, version 3.2.12.RELEASE. As it is for PostgreSQL, it is configured with default parameters.
Here it is our context configuration with C3p0 and Spring;
<bean id="resoilDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<!-- access configuration -->
<property name="driverClass" value="org.postgresql.Driver" />
<property name="jdbcUrl" value="jdbc:postgresql://localhost:5432/*****" />
<property name="user" value="******" />
<property name="password" value="******" />
<!-- pool sizing -->
<property name="initialPoolSize" value="1" />
<property name="minPoolSize" value="1" />
<property name="maxPoolSize" value="6" />
<property name="acquireIncrement" value="3" />
<property name="maxStatements" value="150" />
<!-- refreshing connections -->
<property name="maxIdleTime" value="180" /> <!-- 3min -->
<property name="maxIdleTimeExcessConnections" value="120" /> <!-- 3min -->
<!-- timeouts e testing -->
<property name="idleConnectionTestPeriod" value="120" /> <!-- 60 -->
<property name="testConnectionOnCheckout" value="true" />
<property name="testConnectionOnCheckin" value="false" />
<property name="preferredTestQuery" value="SELECT 1" />
</bean>
<bean id="template" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="resoilDataSource"></property>
</bean>
Let me describe the problem we are encountering:
once we reach "maxPoolSize" of idle connections opened on Postgres side, this connections never expires, they remain in "idle" state and there's no way C3p0 will get any of them back for pooling..
I would expect that once one of these opened connections excess idle time, C3p0 is able to reuse it due to the "maxIdleTimeExcessConnections" parameter.
Unfortunately, this never happens.
I also tried to substitute C3p0 ComboPooledDataSource with Apache DBCP BasicDataSource, but nothing changed.
Before using PostgreSQL database for our application, our customers asked us to install our application integrating other popular databases instead of PostgreSQL (SQL Server and Oracle in particular) and we never experienced this behaviour.
Any ideas about what's going on it's truly appreciated, thanks in advance.
Am New to spring batch. with spring batch job i am inserting data in postgres db then i am getting this error . how to fix this?
Method org.postgresql.jdbc3.Jdbc3PreparedStatement.setQueryTimeout(int) is not yet implemented.; nested exception is java.sql.SQLException: Method org.postgresql.jdbc3.Jdbc3PreparedStatement.setQueryTimeout(int) is not yet implemented.'
This is my datasource code.
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<!-- DB connection properties -->
<property name="driverClass" value="${db.driver:oracle.jdbc.OracleDriver}" />
<property name="jdbcUrl" value="${db.url}" />
<property name="user" value="${db.user:}" />
<property name="password" value="${db.password:}" />
<!-- Pool sizing properties -->
<property name="initialPoolSize" value="${db.pool.initialSize:5}" />
<property name="maxPoolSize" value="${db.pool.maxSize:25}" />
<property name="minPoolSize" value="${db.pool.minSize:0}" />
<property name="maxStatements" value="${db.pool.maxStatements:10}" />
<!-- Connection testing and acquisition properties -->
<property name="maxIdleTime" value="${db.con.maxIdleTime:300}" />
<property name="idleConnectionTestPeriod" value="${db.con.testPeriod:30}" />
<property name="preferredTestQuery" value="${db.con.testQuery:select 1 from dual}" />
<property name="acquireIncrement" value="${db.con.acquireIncrement:5}" />
<property name="acquireRetryAttempts" value="${db.con.retryAttempts:0}" />
<property name="acquireRetryDelay" value="${db.con.retryDelay:3000}" />
<!-- JMX name -->
<property name="dataSourceName" value="Datasource" />
<!-- Debugging options -->
<property name="unreturnedConnectionTimeout" value="${db.con.unreturnedTimeout:0}" />
<property name="debugUnreturnedConnectionStackTraces" value="${db.con.debugUnreturned:false}" />
</bean>
The data source looks OK.... #duffymo, the Oracle driver is the default, but would be overridden by the value of 'db.driver' property if 'db.driver' is specified.
The setTimeout error is thrown by some versions of the PostgreSQL driver because they have not, in fact, implemented setTimeout, so they don't want users thinking the setTimeout actually has any effect.
What version of the PostreSQL driver are you using? Can you share some details of the Spring Batch job? I'm not sure how to prevent Spring from setting the timeout on a PreparedStatement. At a guess, you could set db.con.unreturnedTimeout to 0; I'm thinking that value may be passed to setTimeout; but I'm not sure.
I have a java spring hibernation application with postgresql. this is my C3p0 settings
<property name="acquireIncrement" value="5"/>
<property name="initialPoolSize" value="5"/>
<property name="minPoolSize" value="5" />
<property name="maxPoolSize" value="${c3p0.maxPoolSize}" />
<property name="maxIdleTime" value="1800"/>
<property name="numHelperThreads" value="6"/>
<property name="maxStatements" value="0" />
<property name="maxStatementsPerConnection" value="20" />
<property name="idleConnectionTestPeriod" value="3000" />
In Pgadmin, I am seeing connections that are sitting idle for more than 2 hours. Is there any other property that needs to be set
The maxIdleTime setting is for connections that are idle in the pool. Which means not checked-out. Is it possible that you have some connections that are not closed? In that case connections will be open and not considered as idle because they are checked-out but not checked-in. For such a case you can set unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces to find where this unclosed connection is.
http://www.mchange.com/projects/c3p0/#unreturnedConnectionTimeout
http://www.mchange.com/projects/c3p0/#debugUnreturnedConnectionStackTraces
Following is the my configuration file for C3p0 connection pool. But after some time on starting my application all the connections are reaching to max connections i.e 200 and sitting in idle after that there are no connections are being used by application and if I hit my application URL from browser it is says that service temp unavailable.
My application is client-socket connection application, on opening the connection from client I am doing the data insertions and updations on DB.
<bean id="datasource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="driverClass" value="org.postgresql.Driver" />
<property name="jdbcUrl"
value="jdbc:postgresql://localhost/dbname" />
<property name="user" value="developer" />
<property name="password" value="developerPassword" />
<!-- <property name="initialPoolSize" value="20"/> -->
<property name="minPoolSize" value="2" />
<property name="maxPoolSize" value="200" />
<property name="maxIdleTime" value="1" />
<property name="maxAdministrativeTaskTime" value="900" />
<property name="maxIdleTimeExcessConnections" value="2" />
<!-- <property name="checkoutTimeout" value="300"/> -->
<!-- <property name="maxStatements" value="500"/> -->
<property name="testConnectionOnCheckin" value="true" />
<property name="maxConnectionAge" value="60" />
<property name="unreturnedConnectionTimeout" value="1000" />
<property name="debugUnreturnedConnectionStackTraces" value="true" />
</bean>