I have a java spring hibernation application with postgresql. this is my C3p0 settings
<property name="acquireIncrement" value="5"/>
<property name="initialPoolSize" value="5"/>
<property name="minPoolSize" value="5" />
<property name="maxPoolSize" value="${c3p0.maxPoolSize}" />
<property name="maxIdleTime" value="1800"/>
<property name="numHelperThreads" value="6"/>
<property name="maxStatements" value="0" />
<property name="maxStatementsPerConnection" value="20" />
<property name="idleConnectionTestPeriod" value="3000" />
In Pgadmin, I am seeing connections that are sitting idle for more than 2 hours. Is there any other property that needs to be set
The maxIdleTime setting is for connections that are idle in the pool. Which means not checked-out. Is it possible that you have some connections that are not closed? In that case connections will be open and not considered as idle because they are checked-out but not checked-in. For such a case you can set unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces to find where this unclosed connection is.
http://www.mchange.com/projects/c3p0/#unreturnedConnectionTimeout
http://www.mchange.com/projects/c3p0/#debugUnreturnedConnectionStackTraces
Related
We are creating a Spring boot web app.
DB : JDBC Template and DBCP connection pool.
Java code: A runnable is called in Executors.newSingleThreadScheduledExecutor();
Time interval: 2 min
The code in runnable hits DB using JDBCTemplate.query().
Issue: The heap usage increases to several GBs in few min.
Any Pointers would be helpful to identify the memory leak.
Note: If we comment the JDBCTemplate.query() , memory usage is constant.
Settings of DBCP:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="oracle.jdbc.OracleDriver" />
<property name="url" value="${batch.jdbc.url}" />
<property name="username" value="******" />
<property name="password" value="******" />
<property name="connectionProperties" value="defaultRowPrefetch=10000;defaultBatchValue=200;" />
<property name="minIdle" value="10" />
<property name="maxIdle" value="12" />
<property name="maxActive" value="100" />
<property name="accessToUnderlyingConnectionAllowed" value="true" />
<property name="initialSize" value="${batch.jdbc.pool.size}"/>
<property name="validationQuery" value="select 1 from dual" />
<property name="validationQueryTimeout" value="5" />
<property name="timeBetweenEvictionRunsMillis" value="120000" />
<property name="minEvictableIdleTimeMillis" value="60000" />
<property name="testOnBorrow" value="true" />
</bean>
Suspect from Eclipse MAT report
One instance of "org.apache.commons.pool.impl.GenericObjectPool" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader # 0x7fc1d90124c8" occupies 1,421,543,264 (94.69%) bytes. The memory is accumulated in one instance of "org.apache.commons.pool.impl.GenericObjectPool" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader # 0x7fc1d90124c8".
I'm experiencing a weird behaviour with our Java web application, configured to access a PostgreSQL database through C3p0 ComboPooledDataSource.
First of all, PostgreSQL server installation has default parameters, we didn't need to change any variable in it..
We run our queries using Spring JdbcTemplate, version 3.2.12.RELEASE. As it is for PostgreSQL, it is configured with default parameters.
Here it is our context configuration with C3p0 and Spring;
<bean id="resoilDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<!-- access configuration -->
<property name="driverClass" value="org.postgresql.Driver" />
<property name="jdbcUrl" value="jdbc:postgresql://localhost:5432/*****" />
<property name="user" value="******" />
<property name="password" value="******" />
<!-- pool sizing -->
<property name="initialPoolSize" value="1" />
<property name="minPoolSize" value="1" />
<property name="maxPoolSize" value="6" />
<property name="acquireIncrement" value="3" />
<property name="maxStatements" value="150" />
<!-- refreshing connections -->
<property name="maxIdleTime" value="180" /> <!-- 3min -->
<property name="maxIdleTimeExcessConnections" value="120" /> <!-- 3min -->
<!-- timeouts e testing -->
<property name="idleConnectionTestPeriod" value="120" /> <!-- 60 -->
<property name="testConnectionOnCheckout" value="true" />
<property name="testConnectionOnCheckin" value="false" />
<property name="preferredTestQuery" value="SELECT 1" />
</bean>
<bean id="template" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="resoilDataSource"></property>
</bean>
Let me describe the problem we are encountering:
once we reach "maxPoolSize" of idle connections opened on Postgres side, this connections never expires, they remain in "idle" state and there's no way C3p0 will get any of them back for pooling..
I would expect that once one of these opened connections excess idle time, C3p0 is able to reuse it due to the "maxIdleTimeExcessConnections" parameter.
Unfortunately, this never happens.
I also tried to substitute C3p0 ComboPooledDataSource with Apache DBCP BasicDataSource, but nothing changed.
Before using PostgreSQL database for our application, our customers asked us to install our application integrating other popular databases instead of PostgreSQL (SQL Server and Oracle in particular) and we never experienced this behaviour.
Any ideas about what's going on it's truly appreciated, thanks in advance.
Am New to spring batch. with spring batch job i am inserting data in postgres db then i am getting this error . how to fix this?
Method org.postgresql.jdbc3.Jdbc3PreparedStatement.setQueryTimeout(int) is not yet implemented.; nested exception is java.sql.SQLException: Method org.postgresql.jdbc3.Jdbc3PreparedStatement.setQueryTimeout(int) is not yet implemented.'
This is my datasource code.
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<!-- DB connection properties -->
<property name="driverClass" value="${db.driver:oracle.jdbc.OracleDriver}" />
<property name="jdbcUrl" value="${db.url}" />
<property name="user" value="${db.user:}" />
<property name="password" value="${db.password:}" />
<!-- Pool sizing properties -->
<property name="initialPoolSize" value="${db.pool.initialSize:5}" />
<property name="maxPoolSize" value="${db.pool.maxSize:25}" />
<property name="minPoolSize" value="${db.pool.minSize:0}" />
<property name="maxStatements" value="${db.pool.maxStatements:10}" />
<!-- Connection testing and acquisition properties -->
<property name="maxIdleTime" value="${db.con.maxIdleTime:300}" />
<property name="idleConnectionTestPeriod" value="${db.con.testPeriod:30}" />
<property name="preferredTestQuery" value="${db.con.testQuery:select 1 from dual}" />
<property name="acquireIncrement" value="${db.con.acquireIncrement:5}" />
<property name="acquireRetryAttempts" value="${db.con.retryAttempts:0}" />
<property name="acquireRetryDelay" value="${db.con.retryDelay:3000}" />
<!-- JMX name -->
<property name="dataSourceName" value="Datasource" />
<!-- Debugging options -->
<property name="unreturnedConnectionTimeout" value="${db.con.unreturnedTimeout:0}" />
<property name="debugUnreturnedConnectionStackTraces" value="${db.con.debugUnreturned:false}" />
</bean>
The data source looks OK.... #duffymo, the Oracle driver is the default, but would be overridden by the value of 'db.driver' property if 'db.driver' is specified.
The setTimeout error is thrown by some versions of the PostgreSQL driver because they have not, in fact, implemented setTimeout, so they don't want users thinking the setTimeout actually has any effect.
What version of the PostreSQL driver are you using? Can you share some details of the Spring Batch job? I'm not sure how to prevent Spring from setting the timeout on a PreparedStatement. At a guess, you could set db.con.unreturnedTimeout to 0; I'm thinking that value may be passed to setTimeout; but I'm not sure.
I am working on a Spring-Batch where I am using an embedded DataSource (Apache Commons DBCP 1.3), JDBC3 db2jcc.jar for BD2 database and JDK1.5. I know DBCP2.x is already released but because of existing system (JDK 1.5) I cannot upgrade right now.
DB Configurations:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.ibm.db2.jcc.DB2Driver"/>
<property name="url" value="****"/>
<property name="username" value="***"/>
<property name="password" value="****"/>
<property name="initialSize" value="5"/>
<property name="maxActive" value="10"/>
<property name="maxIdle" value="10"/>
<property name="maxWait" value="10000"/>
<property name="minEvictableIdleTimeMillis" value="30000"/>
<property name="timeBetweenEvictionRunsMillis" value="5000"/>
<property name="validationQuery" value="select 1 from sysibm.sysdummy1"/>
<property name="testOnBorrow" value="true"/>
<property name="testOnReturn" value="true"/>
<property name="testWhileIdle" value="true"/>
</bean>
I noticed If I use validationQuery property along with testOnBorrow, testOnReturn and testWhileIdle to ture, the process taking 3 times higher time to complete.
While analyzing this I found there is a property "validationInterval" in tomcat JDBC connection pool.
My question:
1) Is there any way I can set a validationInterval in DBCP1.3 so that it would not validate the connection all the time but after a specified time period
2) If I don't use validationQuery is there any issue I may face?
3) If no validationQuery provided how DBCP1.3 will validate the connection?
[EDIT]:
Here is the test result after following Nitin suggestions:
Run 1 with Previous Configurations:
Total Time taken - 74 Secs
Run 2 Configurations: set testOnBorrow=true, testOnReturn=false, testWhileIdle=false
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.ibm.db2.jcc.DB2Driver"/>
<property name="url" value="****"/>
<property name="username" value="***"/>
<property name="password" value="****"/>
<property name="initialSize" value="5"/>
<property name="maxActive" value="10"/>
<property name="maxIdle" value="10"/>
<property name="maxWait" value="10000"/>
<property name="minEvictableIdleTimeMillis" value="1800000"/>
<property name="timeBetweenEvictionRunsMillis" value="1800000"/>
<property name="validationQuery" value="select 1 from sysibm.sysdummy1"/>
<property name="testOnBorrow" value="true"/>
<property name="testOnReturn" value="false"/>
<property name="testWhileIdle" value="false"/>
</bean>
Total time taken 47 secs
However tweaking "timeBetweenEvictionRunsMillis" not much changes in processing time but I have decided to set this to 30 mins
1) No. Try
testOnBorrow=true
testOnReturn=false
testWhileIdle=false
2)
you may get stale(broken) connection for escaping validation. however, you may 'tweak' timeBetweenEvictionRunsMillis ...that thread executes every 5 seconds to evict idle connection
3)
connection cannot be validated without validationQuery in DBCP 1.3
Following is the my configuration file for C3p0 connection pool. But after some time on starting my application all the connections are reaching to max connections i.e 200 and sitting in idle after that there are no connections are being used by application and if I hit my application URL from browser it is says that service temp unavailable.
My application is client-socket connection application, on opening the connection from client I am doing the data insertions and updations on DB.
<bean id="datasource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="driverClass" value="org.postgresql.Driver" />
<property name="jdbcUrl"
value="jdbc:postgresql://localhost/dbname" />
<property name="user" value="developer" />
<property name="password" value="developerPassword" />
<!-- <property name="initialPoolSize" value="20"/> -->
<property name="minPoolSize" value="2" />
<property name="maxPoolSize" value="200" />
<property name="maxIdleTime" value="1" />
<property name="maxAdministrativeTaskTime" value="900" />
<property name="maxIdleTimeExcessConnections" value="2" />
<!-- <property name="checkoutTimeout" value="300"/> -->
<!-- <property name="maxStatements" value="500"/> -->
<property name="testConnectionOnCheckin" value="true" />
<property name="maxConnectionAge" value="60" />
<property name="unreturnedConnectionTimeout" value="1000" />
<property name="debugUnreturnedConnectionStackTraces" value="true" />
</bean>