Develop Reference

Heap full of com.mysql.jdbc.JDBC4Connection

I'm analyzing a heap dump where I have 50% live set in the heap. This is caused by a large amount of heap reserved just to hold all the JDBC4Connection and it's internal properties hashmap. This is a heapdump of an application running for a couple of days.
It looks like it's holding thousands of JDBC connection objects and it's configuration.
I found a question which asked a similar question, but got dismissed suggesting that the user was not closing connections: Too many instances of "com.mysql.jdbc.JDBC4Connection"
However, I'm using the org.springframework.data.jpa.repository.JpaSpecificationExecutor.findAll, without querying the database directly. Code:
Specification<AccountProfile> spec = getUserInfoListSurfacing(userInfo);
JPAImpl.findAll(spec, new PageRequest(0, 1, Sort.Direction.DESC, "reportedDate")).getContent()
Here is the bean definition for this connection pool
<bean id="db" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="dataSourceName" value="users"/>
<property name="driverClass" value="${userdb.driver}"/>
<property name="forceUseNamedDriverClass" value="true"/>
<property name="jdbcUrl" value="${userdb.url}"/>
<property name="user" value="${userdb.username}"/>
<property name="password" value="${userdb.password}"/>
<property name="initialPoolSize" value="${userdb.hibernate.c3p0.initialPoolSize}"/>
<property name="maxPoolSize" value="${userdb.hibernate.c3p0.max_size}"/>
<property name="minPoolSize" value="${userdb.hibernate.c3p0.min_size}"/>
<property name="idleConnectionTestPeriod" value="${userdb.hibernate.c3p0.idle_test_period}"/>
<property name="maxStatements" value="${userdb.hibernate.c3p0.max_statements}"/>
<property name="maxIdleTime" value="${userdb.hibernate.c3p0.idle_test_period}"/>
<property name="preferredTestQuery" value="${userdb.hibernate.c3p0.validationQuery}"/>
<property name="testConnectionOnCheckout" value="${userdb.hibernate.c3p0.testOnBorrow}"/>
<property name="acquireIncrement" value="${userdb.hibernate.c3p0.acquireincrement}"/>
<property name="unreturnedConnectionTimeout"
value="${userdb.hibernate.c3p0.unreturnedConnectionTimeout}"/>
<property name="debugUnreturnedConnectionStackTraces" value="${userdb.hibernate.c3p0.debugUnreturnedConnectionStackTraces}"/>
<property name="maxConnectionAge" value="${userdb.hibernate.c3p0.maxconnectionage}"/>
<property name="numHelperThreads" value="${userdb.hibernate.c3p0.numHelperThreads}"/>
<property name="connectionCustomizerClassName" value="${userdb.hibernate.c3p0.connectionCustomizerClassName}"/>
</bean>
I have not been able to find confirmed reports of memory leaks in the Hibernate versions that I'm using.
I'm using spring-data-jpa version 2.0.6.Final
hibernate-core version 4.3.5.Final
hibernate-jpa-2.1-api version 1.0.2.Final

You show 224 instances of the object "managed". There is one instance of "managed" per Connection pool, so you have 224 Connection pools holding references to Connections, not just one. You have to understand why you are instantiating so many pools when a typical application just wants one.
The usual error that could cause this is to create a new Connection pool every time you mean to establish a new Connection. The less common cause is using the same DataSource to establish Connections under multiple authentications using dataSource.getConnection( user, password ). A new Connection pool is established for each distinct authentication.
Since you are not directly instantiating your Connection pool but using Spring to do it, it's not so easy to debug how why you are instantiating so any pools. One thing you should definitely add is a destroy-method attribute to your bean XML tag. That is...
<bean id="db" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
...
</bean>
You could be generating all these pools just by hot redeploying your app a lot, and failing to cleanup the old pools because Spring doesn't know it has to clean up your beans. See Spring's Destruction Callbacks.

Mysql connection pool performance loss

I've a Java application that can run either in Jboss and Glassfish.
I've used Hibernate to handle the DB and hikariCP as connection pool.
All works perfectly but I've noticed a performance loss.
To make a stress test I've created a test servlet that select from DB a row using hibernate.
With JMeter I've set 500 simultaneusly request per sec to this servlet, while JMeter is running i've tryed to use the application and the behavior is the following:
When I set maximumPoolSize to 10 the application responds after very long time (~10 sec).
When I set the maximumPoolSize to 1 the application responds as expected (after 2 or 3 sec)
Another thing I've noticed is that when I run JMeter with 500 threads and 10 maximum connection pool, the apllication JMeter become freeze while with 1 maximum connection pool the application JMeter never become freeze.
What is the reason of this behavior? I've missed some configuration on mysql?
Hibernate config
<property name="hibernate.bytecode.use_reflection_optimizer">false</property>
<property name="hibernate.connection.provider_class">com.zaxxer.hikari.hibernate.HikariConnectionProvider</property>
<property name="hibernate.hikari.dataSourceClassName">com.mysql.jdbc.jdbc2.optional.MysqlDataSource</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.search.autoregister_listeners">false</property>
<property name="hibernate.connection.CharSet">utf8</property>
<property name="hibernate.connection.characterEncoding">utf8</property>
<property name="hibernate.connection.useUnicode">true</property>
<property name="hibernate.hikari.idleTimeout">15000</property>
<property name="hibernate.hikari.maxLifetime">900000</property><!-- 15 min -->
<property name="hibernate.hikari.leakDetectionThreshold">5000</property>
<property name="hibernate.hikari.validationTimeout">1000</property>
<property name="hibernate.hikari.minimumIdle">1</property>
<property name="hibernate.hikari.maximumPoolSize">10</property>
<property name="hibernate.hikari.dataSource.cachePrepStmts">true</property>
<property name="hibernate.hikari.dataSource.prepStmtCacheSize">250</property>
<property name="hibernate.hikari.dataSource.prepStmtCacheSqlLimit">2048</property>
<property name="hibernate.show_sql">true</property>
<property name="hibernate.jdbc.batch_size">50</property>
<property name="hibernate.hikari.dataSource.url">jdbc:mysql://localhost:3306/myDb?transformedBitIsBoolean=false&tinyInt1isBit=false&UseUnicode=true&characterEncoding=utf8</property>

ArrayIndexOutOfBounds - Hibernate

I am using hibernate with a database wich is in a WAS (Websphere Application Server [IBM]) and I am getting this error sometimes:
java.lang.ArrayIndexOutOfBoundsException
at com.ibm.ws.rsadapter.spi.CacheMap.removeLRU(CacheMap.java:378)
at com.ibm.ws.rsadapter.spi.CacheMap.add(CacheMap.java:180)
at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.cacheStatement(WSRdbManagedConnectionImpl.java:3150)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.closeWrapper(WSJdbcPreparedStatement.java:513)
at com.ibm.ws.rsadapter.jdbc.WSJccPreparedStatement.closeWrapper(WSJccPreparedStatement.java:290)
at com.ibm.ws.rsadapter.jdbc.WSJdbcObject.close(WSJdbcObject.java:241)
at com.ibm.ws.rsadapter.jdbc.WSJdbcObject.close(WSJdbcObject.java:194)
at org.hibernate.jdbc.AbstractBatcher.closePreparedStatement(AbstractBatcher.java:563)
at org.hibernate.jdbc.AbstractBatcher.closeStatement(AbstractBatcher.java:291)
at org.hibernate.engine.query.NativeSQLQueryPlan.performExecuteUpdate(NativeSQLQueryPlan.java:214)
at org.hibernate.impl.SessionImpl.executeNativeUpdate(SessionImpl.java:1300)
at org.hibernate.impl.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:365)
Is a multithread application, so may be some stuff related with the cache of hibernate when multiple access are performed, but I am not sure about it. The error is thrown when the hibernate query sentence is executed like
query.list();
or
query.executeUpdate();
I have tried to set the cacheable to false and the CacheMode to IGNORE:
Session session = getSession();
session.setCacheMode(CacheMode.IGNORE);
session.createSQLQuery('query').setCacheable(false);
for disabling the cache, but the exceptions keep appearing, as I said, sometimes.
My hibernate-config.xml is this one:
<hibernate-configuration>
<session-factory name="HibernateSessionFactory">
<property name="hibernate.dialect">${hibernate.dialect}</property>
<property name="hibernate.show_sql">${hibernate.show_sql}</property>
<property name="hibernate.format_sql">${hibernate.format_sql}</property>
<property name="hibernate.use_sql_comments">${hibernate.use_sql_comments}</property>
<property name="hibernate.connection.autocommit">true</property>
<property name="hibernate.connection.aggressive_release">false</property>
<property name="hibernate.connection.release_mode">after_transaction</property>
<property name="hibernate.cache.use_query_cache">false</property>
<property name="hibernate.cache.use_second_level_cache">false</property>
<property name="hibernate.connection.pool_size">0</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.transaction.auto_close_session">true</property>
<property name="hibernate.transaction.flush_before_completion">true</property>
<!-- Disable the second-level cache -->
<property name="cache.provider_class">org.hibernate.cache.NoCacheProvider</property>
</session-factory>
</hibernate-configuration>
Here you can see as I keep trying to totally disable the cache, but It still not work properly.
Any help or suggestion would be appreciated.
Thank you very much in advance! :D

Spring connection pooling configurations

I've been searching around the web for some time now and i've yet to fix this issue:
I have the following datasource configuration:
<bean id="cpms.prod.ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName"><value>com.mysql.jdbc.Driver</value></property>
<property name="url"><value>jdbc:mysql://localhost/mysql</value></property>
<property name="username"><value>test</value></property>
<property name="password"><value>test</value></property>
<property name="initialSize" value="1" />
<property name="maxActive" value="2" />
<property name="maxIdle" value="1"></property>
</bean>
This should be enough to make sure there are only 2 active connections at one point and those are the ones used for the pool.
In my java code i'm using SimpleJdbcTemplate to do three queries, and as I understand it this object should be returning the connections to the pool after each query ends, should also be blocking the third query while one of the others end.
When looking at my database administration console I see 3 connections appear, and then change to sleep state. If I run the queries again I see another 3 connections popup and the other 3 stay there.
The only way i've found for the connections to be closed is by setting:
<property name="removeAbandoned" value="true"/>
<property name="removeAbandonedTimeout" value="1"/>
<property name="minIdle" value="0"></property>
<property name="timeBetweenEvictionRunsMillis" value="1000"></property>
<property name="minEvictableIdleTimeMillis" value="1000"></property>
which forces the abandoned connections procedure to run and clean the old connections.
I shouldn't have to meddle with these parameters, and especially not be setting them so low as it might have performance issues.
I've also tried the solution shown here to the same effect until i change the timeBetweenEvictionRunsMillis and minEvictableIdleTimeMillis to the lower values. And it still doesn't limit the connections to 2.

All connections in the JdbcTemplate are via DataSourceUtils.doGetConnection. What you are seeing might be due to the 'intelligence' in BasicDataSource
From the API:
Abandonded connections are identified and removed when getConnection() is invoked and the following conditions hold:
getRemoveAbandoned() = true
getNumActive() > getMaxActive() - 3
getNumIdle() < 2
The data source seems to allow 3 more active connections than the specified max-active.

How to avoid stale MySQL/Hibernate connections (MySQLNonTransientConnectionException)

I have a Java webapp using Hibernate and MySQL. If the site isn't used for a few days, the MySQL connection goes stale, and I am met with the following exception:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Connection.close() has already been called. Invalid operation in this state.
From experience using raw JDBC it is possible to configure the connection to attempt to recover from errors or stale connections, but I don't know how to do this using Hibernate. I am not explicitly calling close() anywhere (but I am betting Hibernate does somewhere deep down in its guts).
Does anybody know what I should do?

What connection pool are you using?
The Hibernate suggestion is not to use the built-in pooling, but use the application server datasource or something like Commons DBCP or C3PO.

I had that problem. Using connection pooling (c3p0) made it go away. It is also a good idea in general to use some connection pooling.

Thanks for the advice. For posterity's sake, I changed the hibernate.cfg.xml from the following:
<property name="connection.url">jdbc:mysql://localhost/FooDB</property>
<property name="connection.username">root</property>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="connection.password">secret</property>
<property name="transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
... to the following:
<property name="connection.datasource">java:/comp/env/jdbc/FooDB</property>
<property name="dialect">org.hibernate.dialect.MySQLDialect</property>
This also required adding a standard 'context' entry in my web app's context.xml:
<Resource name="jdbc/FooDB"
auth="Container"
type="javax.sql.DataSource"
maxActive="100"
maxIdle="30"
maxWait="10000"
username="root"
password="secret"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/ss?autoReconnect=true" />

We had a similar problem of hibernate mysql connection getting timed out.
So we tried C3P0, with the following configuration:
<property name=c3p0.acquire_increment>1</property>
<property name=c3p0.idle_test_period>3600</property>
<property name=c3p0.max_statements>0</property>
<property name=c3p0.min_size>1</property>
<property name=c3p0.timeout>3605</property>
<property name=hibernate.c3p0.preferredTestQuery>select 1;</property>
Hibernate connection_pool size was set to 1.
This made the timeout problem go away. But we started facing another problem. Long waits.
We have a service (servlet running on jboss), which receives something like 5-6 requests per second. Every request needs to connect to mysql through hibernate. Most of our requests do selects, with an insert/update every 5th-6th request. Normally the request serve time for us is 2-3ms for select and 40-50ms for insert/update. But, after using the above C3P0 configuration, we saw that every request completing after an update was taking almost 4-5 minutes! From our logs, it seemed that randomly a select request will get stuck and will be able to complete only after an update request was received and served.
Above problem goes away if we remove the C3P0 config. Can somebody suggest what we could be doing wrong?
Here is the complete hibernate config for reference:
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="connection.url">jdbc:mysql://xxx.xxx.xxx</property>
<property name="connection.username">xxx</property>
<property name="connection.password">xxx</property>
<property name="connection.pool_size">1</property>
<property name="dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="current_session_context_class">thread</property>
<property name="hibernate.cache.use_query_cache">false</property>
<property name="hibernate.cache.use_second_level_cache">false</property>
<property name="show_sql">true</property>
<!-- Transaction isolation 2 = READ_COMMITTED -->
<property name="connection.isolation">2</property>
<property name="connection.autocommit">true</property>
<!-- configuration pool via c3p0-->
<property name="c3p0.acquire_increment">1</property>
<property name="c3p0.idle_test_period">3600</property> <!-- seconds -->
<property name="c3p0.max_size">1</property>
<property name="c3p0.max_statements">0</property>
<property name="c3p0.min_size">1</property>
<property name="c3p0.timeout">3605</property> <!-- seconds -->
<property name="hibernate.c3p0.preferredTestQuery">select 1;</property>
</session-factory>
</hibernate-configuration>

<property name="c3p0.acquire_increment">1</property>
<property name="c3p0.idle_test_period">120</property> <!-- seconds -->
<property name="c3p0.max_size">100</property>
<property name="c3p0.max_statements">0</property>
<property name="c3p0.min_size">10</property>
<property name="c3p0.timeout">180</property> <!-- seconds -->
override these settings on your config file. It ll helps to you.