Develop Reference

Heap full of com.mysql.jdbc.JDBC4Connection

I'm analyzing a heap dump where I have 50% live set in the heap. This is caused by a large amount of heap reserved just to hold all the JDBC4Connection and it's internal properties hashmap. This is a heapdump of an application running for a couple of days.
It looks like it's holding thousands of JDBC connection objects and it's configuration.
I found a question which asked a similar question, but got dismissed suggesting that the user was not closing connections: Too many instances of "com.mysql.jdbc.JDBC4Connection"
However, I'm using the org.springframework.data.jpa.repository.JpaSpecificationExecutor.findAll, without querying the database directly. Code:
Specification<AccountProfile> spec = getUserInfoListSurfacing(userInfo);
JPAImpl.findAll(spec, new PageRequest(0, 1, Sort.Direction.DESC, "reportedDate")).getContent()
Here is the bean definition for this connection pool
<bean id="db" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="dataSourceName" value="users"/>
<property name="driverClass" value="${userdb.driver}"/>
<property name="forceUseNamedDriverClass" value="true"/>
<property name="jdbcUrl" value="${userdb.url}"/>
<property name="user" value="${userdb.username}"/>
<property name="password" value="${userdb.password}"/>
<property name="initialPoolSize" value="${userdb.hibernate.c3p0.initialPoolSize}"/>
<property name="maxPoolSize" value="${userdb.hibernate.c3p0.max_size}"/>
<property name="minPoolSize" value="${userdb.hibernate.c3p0.min_size}"/>
<property name="idleConnectionTestPeriod" value="${userdb.hibernate.c3p0.idle_test_period}"/>
<property name="maxStatements" value="${userdb.hibernate.c3p0.max_statements}"/>
<property name="maxIdleTime" value="${userdb.hibernate.c3p0.idle_test_period}"/>
<property name="preferredTestQuery" value="${userdb.hibernate.c3p0.validationQuery}"/>
<property name="testConnectionOnCheckout" value="${userdb.hibernate.c3p0.testOnBorrow}"/>
<property name="acquireIncrement" value="${userdb.hibernate.c3p0.acquireincrement}"/>
<property name="unreturnedConnectionTimeout"
value="${userdb.hibernate.c3p0.unreturnedConnectionTimeout}"/>
<property name="debugUnreturnedConnectionStackTraces" value="${userdb.hibernate.c3p0.debugUnreturnedConnectionStackTraces}"/>
<property name="maxConnectionAge" value="${userdb.hibernate.c3p0.maxconnectionage}"/>
<property name="numHelperThreads" value="${userdb.hibernate.c3p0.numHelperThreads}"/>
<property name="connectionCustomizerClassName" value="${userdb.hibernate.c3p0.connectionCustomizerClassName}"/>
</bean>
I have not been able to find confirmed reports of memory leaks in the Hibernate versions that I'm using.
I'm using spring-data-jpa version 2.0.6.Final
hibernate-core version 4.3.5.Final
hibernate-jpa-2.1-api version 1.0.2.Final

You show 224 instances of the object "managed". There is one instance of "managed" per Connection pool, so you have 224 Connection pools holding references to Connections, not just one. You have to understand why you are instantiating so many pools when a typical application just wants one.
The usual error that could cause this is to create a new Connection pool every time you mean to establish a new Connection. The less common cause is using the same DataSource to establish Connections under multiple authentications using dataSource.getConnection( user, password ). A new Connection pool is established for each distinct authentication.
Since you are not directly instantiating your Connection pool but using Spring to do it, it's not so easy to debug how why you are instantiating so any pools. One thing you should definitely add is a destroy-method attribute to your bean XML tag. That is...
<bean id="db" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
...
</bean>
You could be generating all these pools just by hot redeploying your app a lot, and failing to cleanup the old pools because Spring doesn't know it has to clean up your beans. See Spring's Destruction Callbacks.

JDBC Connection taking too long

I have an issue in my Spring MVC JDBC call. If I make the call quickly after starting the server, the JDBC connection is make in a second and the data is retrieved. Similarly, if the other DAOs are called in quick succession with one another, the connection is made soon. But if I try to call a DAO after a gap of even a few minutes, the JDBC connection takes forever to be done. It gets stuck on
"DataSourceUtils:110 - Fetching JDBC Connection from DataSource"
I have never had the patience to really check how long it takes to retrieve the connection but I've waited for 10 minutes and there was no sign of the connection being made.
Next, I try to restart the server at least. But JDBC obstructs even stopping of the server!! The console is stuck on this line:
"DisposableBeanAdapter:327 - Invoking destroy method 'close' on bean with name 'dataSource'"
Eventually I restart Eclipse and it works alright until there is a time gap again.
This is my bean definition for the data source:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="url" />
<property name="username" value="abc" />
<property name="password" value="abc" />
<property name="validationQuery" value="SELECT 1" />
<property name="testWhileIdle" value="true" />
<property name="maxActive" value="100" />
<property name="minIdle" value="10" />
<property name="initialSize" value="10" />
<property name="maxIdle" value="20" />
<property name="maxWait" value="1000" />
</bean>
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="getDataDao" class="com.project.dao.GetDataDao">
<constructor-arg index="0" ref="jdbcTemplate" />
<constructor-arg index="1" value="STORED_PROC_NAME"></constructor-arg>
</bean>
In my DAO file, I extend Spring's StoredProcedure class and this is the constructor:
public GetDataDao(JdbcTemplate jdbcTemplate, String spName) {
super(jdbcTemplate, spName);
declareParameter(new SqlParameter("p_input", Types.VARCHAR));
declareParameter(new SqlOutParameter("o_result", Types.VARCHAR));
compile();
}
In another function, this is how I call the SP:
spOutput = super.execute(spInput);
where spOutput and spInput are HashMaps.
Am I doing something wrong in my configuration? TIA.

I too had the exact same issue. I found that issue was consistent with a particular query, checked the query and found that issue was within query itself. Running query separately was also taking time. Query was converting a column to lower and that column was not indexed. Query was like lower(trim(column_name)), removed the lower and trim. It worked perfectly fine after that.

The additional code helps, but I do not see anything in it that would cause issue you are seeing. The most likely reason for issue you are seeing is that connections are being pulled out of the pool, but they are not being returned, and the pool eventually becomes starved. The dbcp pool is then later blocking your shutdown because these connections are still open, and probably hung.
To verify, you might try setting maxActive and similar settings to something much lower, perhaps even "1", and then verify that you get the same issue immediately.
Have you verified that your stored procedure is returning? i.e. You actually get spOutput for every call and the stored procedure itself is not hanging consistently or randomly?
If so, my only other suggestion is to post more code, especially from the call stack leading in to GetDataDao, and including whatever method in the DAO is making the sp.execute call. An assumption is that you are not using transactions, but if you are, then showing where you start/commit transaction in code would also be very important.

Mysql connection pool performance loss

I've a Java application that can run either in Jboss and Glassfish.
I've used Hibernate to handle the DB and hikariCP as connection pool.
All works perfectly but I've noticed a performance loss.
To make a stress test I've created a test servlet that select from DB a row using hibernate.
With JMeter I've set 500 simultaneusly request per sec to this servlet, while JMeter is running i've tryed to use the application and the behavior is the following:
When I set maximumPoolSize to 10 the application responds after very long time (~10 sec).
When I set the maximumPoolSize to 1 the application responds as expected (after 2 or 3 sec)
Another thing I've noticed is that when I run JMeter with 500 threads and 10 maximum connection pool, the apllication JMeter become freeze while with 1 maximum connection pool the application JMeter never become freeze.
What is the reason of this behavior? I've missed some configuration on mysql?
Hibernate config
<property name="hibernate.bytecode.use_reflection_optimizer">false</property>
<property name="hibernate.connection.provider_class">com.zaxxer.hikari.hibernate.HikariConnectionProvider</property>
<property name="hibernate.hikari.dataSourceClassName">com.mysql.jdbc.jdbc2.optional.MysqlDataSource</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.search.autoregister_listeners">false</property>
<property name="hibernate.connection.CharSet">utf8</property>
<property name="hibernate.connection.characterEncoding">utf8</property>
<property name="hibernate.connection.useUnicode">true</property>
<property name="hibernate.hikari.idleTimeout">15000</property>
<property name="hibernate.hikari.maxLifetime">900000</property><!-- 15 min -->
<property name="hibernate.hikari.leakDetectionThreshold">5000</property>
<property name="hibernate.hikari.validationTimeout">1000</property>
<property name="hibernate.hikari.minimumIdle">1</property>
<property name="hibernate.hikari.maximumPoolSize">10</property>
<property name="hibernate.hikari.dataSource.cachePrepStmts">true</property>
<property name="hibernate.hikari.dataSource.prepStmtCacheSize">250</property>
<property name="hibernate.hikari.dataSource.prepStmtCacheSqlLimit">2048</property>
<property name="hibernate.show_sql">true</property>
<property name="hibernate.jdbc.batch_size">50</property>
<property name="hibernate.hikari.dataSource.url">jdbc:mysql://localhost:3306/myDb?transformedBitIsBoolean=false&tinyInt1isBit=false&UseUnicode=true&characterEncoding=utf8</property>

C3po creates too many connections

This is how my c3po Spring configuration looks like;
<property name="idleConnectionTestPeriod" value="120" />
<property name="maxIdleTime" value="1800" />
<property name="maxPoolSize" value="2" />
<property name="unreturnedConnectionTimeout" value="600" />
<property name="numHelperThreads" value="10" />
<property name="maxStatementsPerConnection" value="1" />
I try to monitor how many connections are made to DB so I run this script against my Oracle DB after I restart my application;
select
substr(a.spid,1,9) pid,
substr(b.sid,1,5) sid,
substr(b.serial#,1,5) ser#,
substr(b.machine,1,6) box,
substr(b.username,1,10) username,
substr(b.osuser,1,8) os_user,
substr(b.program,1,30) program
from v$session b, v$process a
where
b.paddr = a.addr and type='USER' order by spid;
Bu this script result shows too many connections are open, So Assuming this script results really indicates the number of sessions/connections. How can I make them less with my spring configuration? Because no matter how much I change the parameters result does not change.

observe keenly....because maxPoolSize is for defining one pool size and a single datasource can contain multiple pools and in that case no of connections becomes
no of pools * maxPoolSize
so look for no of pools you've created

Hibernate and too many connections

I am working on web-project with Hibernate ans Spring MVC.
My hibernate configuration is:
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="connection.url">jdbc:mysql://localhost/xxx</property>
<property name="connection.username">xxx</property>
<property name="connection.password">xxx</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.timeout">1800</property>\<property name="hibernate.c3p0.max_statements">50</property>
<property name="dialect">org.hibernate.dialect.MySQL5Dialect</property>
<property name="connection.useUnicode">true</property>
<property name="connection.characterEncoding">UTF-8</property>
<property name="current_session_context_class">thread</property>
<property name="show_sql">true</property>
The hibernate sessions are closing in service-classes destructors (these service-classes have DAO objects). But after publishing at production server I've got too many connections exception from mysql.Every server call, the mysql connection was opened. When the connections amount become 101 - db failed. I think that destructors had not time enought for executes so the connections were opened all the time.
Then, I rebuilded the structure. Now, the Spring controllers call the service-class function, that releases the session mannualy. But it doesn't help: the connections are still opened and now I can not use LAZY-collections in views because the session already closed.
How can I solve that problem? What is the usual approach here?
Thank you.