Develop Reference

JDBC MySQL Connection Issue - Attempted reconnect 3 times. Giving up

I have a rest service application running the Java Spring framework. The application depends on a connection to an external MySQL DB, which is connected via JDBC.
My issue is maintaining a solid connection between the rest service and the MySQL db. I have what I consider a rudimentary connection failsafe in place that looks something like:
public Connection getConnection() throws SQLException {
if(connection == null){
this.buildConnection();
}
else if(!connection.isValid(10)){ //Rebuild connection if it is no longer valid
connection.close();
this.buildConnection();
}
return connection;
}
Using this method should ensure that the connection is valid before any query is executed. My problem is that I periodically get an exception thrown when calling this method:
Could not create connection to database server. Attempted reconnect 3 times. Giving up. SQLState: 08001. ErrorCode: 0.
The things that have me super perplexed about this are:
This error only happens periodically. Many times the connection works just find.
I test this same application on my developer machine and this error never occurs.
I custom configured the MySQL DB on my own server, so I control all its config options. From this I know that this issue isn't related to the maximum number of connections allowed, or a connection timeout.
Edit - Update 1:
This service is hosted as Cloud Service on Microsoft Azure platform.
I accidentally set it up as an instance in Northern Europe, while the DB is located in North America - probably not related, but trying to paint the whole picture.
Tried the advice at this link with no success. Not using thread pools, and all ResultSets and Statements/PreparedStatements are closed after.
Edit - Update 2
After some refactoring, I was able to successfully implement a HikariCP Connection Pool as outlined by #M.Deinum below. Unfortunately, the same problem persists. Everything works great on my local machine, and all Unit Tests pass, but as soon as I push it to Azure and wait more than a few minutes between requests, I get this error, when trying to grab a connection from the pool:
springHikariCP - Connection is not available, request timed out after 38268ms. SQLState: 08S01. ErrorCode: 0.
My HikariCP configuration is as follows:
//Set up connection pool
HikariConfig config = new HikariConfig();
config.setDriverClassName("com.mysql.jdbc.Driver");
config.setJdbcUrl("jdbc:mysql://dblocation");
//Connection pool properties
Properties prop = new Properties();
prop.setProperty("user", "Username");
prop.setProperty("password", "Password");
prop.setProperty("verifyServerCertificate", "false");
prop.setProperty("useSSL","true");
prop.setProperty("requireSSL","true");
config.setDataSourceProperties(properties);
config.setMaximumPoolSize(20);
config.setConnectionTestQuery("SELECT 1");
config.setPoolName("springHikariCP");
config.setLeakDetectionThreshold(5000);
config.addDataSourceProperty("dataSource.cachePrepStmts", "true");
config.addDataSourceProperty("dataSource.prepStmtCacheSize", "250");
config.addDataSourceProperty("dataSource.prepStmtCacheSqlLimit", "2048");
config.addDataSourceProperty("dataSource.useServerPrepStmts", "true");
dataSource = new HikariDataSource(config);
Any help would be greatly appreciated.

I suggest using a proper JDBC Connection Pool like HikariCP that together with a validation query which will execute on correct intervals should give you fresh and proper connections each time.
Assuming you are using Spring and xml to configure the datasource.
<bean id="dataSource" class="com.zaxxer.hikari.HikariDataSource">
<property name="poolName" value="springHikariCP" />
<property name="dataSourceClassName" value="com.mysql.jdbc.jdbc2.optional.MysqlDataSource" />
<property name="dataSourceProperties">
<props>
<prop key="url">${jdbc.url}</prop>
<prop key="user">${jdbc.username}</prop>
<prop key="password">${jdbc.password}</prop>
</props>
</property>
</bean>
It by default validates connections on checkout. I suggest a try out.
As you are using java bases config I suggest the following
#Bean
public DataSource dataSource() {
HikariDataSource ds = new HikariDataSource();
ds.setPoolName("springHikariCP");
ds.setMaxPoolSize(20);
ds.setLeakDetectionThreshold(5000);
ds.setDataSourceClassName("com.mysql.jdbc.jdbc2.optional.MysqlDataSource");
ds.addDataSourceProperty("url", url);
ds.addDataSourceProperty("user", username);
ds.addDataSourceProperty("password", password);
ds.addDataSourceProperty("cachePrepStmts", true);
ds.addDataSourceProperty("prepStmtCacheSize", 250);
ds.addDataSourceProperty("prepStmtCacheSqlLimit", 2048);
ds.addDataSourceProperty("useServerPrepStmts", true);
ds.addDataSourceProperty("verifyServerCertificate", false);
ds.addDataSourceProperty("useSSL", true);
ds.addDataSourceProperty("requireSSL", true);
return ds;
}

It seems to be caused by the system variable wait_timeout of MySQL.
For MySQL 5.0, 5.1, 5.5, 5.6, the default value for wait_timeout is 28800 seconds (8 hours), and the maximum value for wait_timeout:
Linux : 31536000 seconds (365 days, one year)
Windows : 2147483 seconds (2^31 milliseconds, 24 days 20 hours 31 min 23 seconds)
The number of seconds the server waits for activity on a noninteractive connection before closing it. This timeout applies only to TCP/IP and Unix socket file connections, not to connections made using named pipes, or shared memory.
So I think you can try to use a jdbc connection to keep pinging interval some seconds, or directly using a kind of JDBC Connection Pool framework to manage jdbc connections automatically.
Hope it help. Best Regards.

assuming code you have now for '//Set up connection pool' is called only once, like bean creation, to initialize dataSource.
with that, your getConnection() would be just:
public Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
and make sure wait_timeout in mysql is set to minute more than maxLifetime in hikaricp which is default 30 minutes
Hope this helps.

Oracle data source connection pooling not working used with Spring and JDBCTemplate

Question: Lot of active unclosed physical connections with database even with connection pooling. Can someone tell me why is it so?
I configured the connection pool settings using oracle.jdbc.pool.OracleDataSource. However it seems the physical connections are not getting closed after use.
I thought, Since it is connection pooling, the connections will be reused from the pool, so so many physical connections will not be made,
but thats not what is happening now!
There are 100+ active physical connections in the database generating from the application [not from plsql developer or any such client tools],
due to which it kicks off TNS error while trying to do write operations on database,
where as read operations are fine even with large number of active connections.
Here is the Spring configuration,
<bean id="oracleDataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close"
p:URL="${url}"
p:user="${username}"
p:password="${password}"
p:connectionCachingEnabled="true">
<property name="connectionProperties">
<props merge="default">
<prop key="AutoCommit">false</prop>
</props>
</property>
</bean>
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate"
p:dataSource-ref="oracleDataSource" />
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager"
p:dataSource-ref="oracleDataSource">
</bean>
The SQL that returned the 100+ active connections is ,
select username, terminal,schemaname, osuser,program from v$session where username = 'grduser'

You should configure connection cache, the default value of max connections for implicit connection cache is the max number of database sessions configured for the database.

Thanks to #Evgeniy Dorofeev.
Solution in detail :
The connectionCache was enabled, but the properties were not set.
Set the properties as written below,
`
<bean id="oracleDataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close"
p:URL="${url}"
p:user="${username}"
p:password="${password}"
p:connectionCachingEnabled="true">
<property name="connectionProperties">
<props merge="default">
<prop key="AutoCommit">false</prop>
</props>
</property>
<property name="connectionCacheProperties">
<props>
<prop key="MinLimit">5</prop>
<prop key="MaxLimit">10</prop>
<prop key="InactivityTimeout">2</prop>
</props>
</property>
</bean>
`
Now, for every operation in the application that requires a connection, it will try to get from the pool if available and ready to use, but is guaranteed that the database will have maximum of only 10 active physical connections. Any attempt to get a extra physical connection will lead to a database error at the application side.
Even if you have set the connectionCache, make sure your application is not explicitly trying to get a connection, like
Connection connection = getJdbcTemplate().getDataSource().getConnection();
This is alarming, JDBCTemplate doesnt manage the closing of this connection. Hence you have to close by yourself after use, otherwise the physical connection will still be active and unclosed even after use. So next time you call this again, it try to get a new physical connection , and remain unclosed, resulting in piling up of active connections until the maxLimit is reached.
The connection might be explicity needed when you want to pass it as a parameter to some other function, say in the case of an ArrayDescriptor [if you talk to PLSQL Stored procedures that has IN parameter to accept an array of values , an array of Varchar or array of RAW]. If you need to create a ArrayDescriptor,
ArrayDescriptor arrayDescriptor = ArrayDescriptor.createDescriptor(
"SOME_TYPE_NAME", connection );
ARRAY SQLArray= new ARRAY(arrayDescriptor, connection , arrayString);
Hence do a connection.close() explicity here.
Additional info:
Connection connection = getJdbcTemplate().getDataSource().getConnection()
This attempts to establish a connection with the data source that this DataSource object represents.
Calling this line of code - once, will attempt to establish a new connection.
Calling again, will establish a second connection. For each request, it will create a new connection!.So If your maxLimit is 10,
Until there are 10 active physical connections in the database, the call will be successful, but
note that all the connections are active [not closed].
So lets say now there are 10 active db connections, as maxLimit is set to 10.
So any requests that requires a database operation, that would go through the
normal route of accessing a connection via the JDBCTemplate will be picking up the already established connection [from the 10 connections]
However any request that calls this code getJdbcTemplate().getDataSource().getConnection() to access a connection
will attempt to establish a NEW connection, and will fail, resulting in exception.
The only way to resolve this is to explicitly close the connection when we explicitly create the connection.
ie calling connection.close()
When we don't explicitly create the connection, and when it is managed by Spring, then Spring will take care of closing
the connections too. In the case of using Oracle Data Source pooling along with JDBCTemplate, closing the connection[returning the
connections to the pool] is managed by Spring.

mybatis setting for mysql time-out on tomcat server

using java7 tomcat7 and mybatis as ORM
config.xml is like this
<transactionManager type="JDBC" />
<dataSource type="POOLED">
<property name="driver" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://localhost:3306/xxxxdb" />
<property name="username" value="xxxxxxx" />
<property name="password" value="xxxxxxx" />
<property name="poolPingEnabled" value="true" />
<property name="poolPingQuery" value="SELECT 1 " />
</dataSource>
</environment>
mysql settings are all default set.
Hence interactive_timeout is 28800.
When I login my service, it fails for the first time, then it succeeds for the second time.
Above error sometimes happens even though re-login within 28800 seconds.
I paste the error message in server
2015 10:03:49 org.apache.ibatis.datasource.pooled.PooledDataSource warn
WARN: Execution of ping query 'SELECT 1' failed: Communications link failure
The last packet successfully received from the server was 30,572,290 milliseconds ago. The last packet sent successfully to the server was 1 milliseconds ago.
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 36,001,604 milliseconds ago. The last packet sent successfully to the server was 36,001,633 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
### The error may exist in sql.xml
### The error may involve com.xxx.isRegistered-Inline
### The error occurred while setting parameters
### SQL: [query for login];
### Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 36,001,604 milliseconds ago. The last packet sent successfully to the server was 36,001,633 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
I tried to add "autoReconnect=true" to the end of connection url, but it doesn't solve the problem

Possible duplicate of Java app handling for connections getting dropped
Reporting here BalusC's answer:
This exception suggests that you're opening the connection only once during application's startup and keeping forever open during the application's lifetime. This is bad. The DB will reclaim the connection sooner or later because it's been open for too long. You should close connections properly in the finally block of the very same try block as you're opening it and executing the query on it.
E.g.
public Entity find(Long id) throws SQLException {
Connection connection = null;
// ...
try {
connection = database.getConnection();
// ...
} finally {
// ...
if (connection != null) try { connection.close(); } catch (SQLException logOrIgnore) {}
}
return entity;
}
If you have a performance concern regarding this (which is very reasonable as connecting is the most expensive task), then you should be using a connection pool. It also transparently handles this kind of "connection dropped" problems. For example, BoneCP. Please note that also in case of a connection pool, you should still be closing the connections in the finally block as per the above JDBC code idiom. It will namely make them available for reuse.

How to catch when JMS connection is established?

I have message producers that are sending JMS messages about some events using ActiveMQ.
However, connection to ActiveMQ might not be up all the time. Thus, events are stored and when connection is established they are suppose to be read and sent over. Here is my code:
private void sendAndSave(MyEvent event) {
boolean sent = sendMessage(event);
event.setProcessed(sent);
boolean saved = repository.saveEvent(event);
if (!sent && !saved) {
logger.error("Change event lost for Id = {}", event.getId());
}
}
private boolean sendMessage(MyEvent event) {
try {
messenger.publishEvent(event);
return true;
} catch (JmsException ex) {
return false;
}
}
I'd like to create some kind of ApplicationEventListener that will be invoked when connection is established and process unsent events.
I went through JMS, Spring framework and ActiveMQ documentation but couldn't find any clues how to hook up my listener with ConnectionFactory.
If someone can help me out, I'll appreciate it greatly.
Here is what my app Spring context says about JMS:
<!-- Connection factory to the ActiveMQ broker instance. -->
<!-- The URI and credentials must match the values in activemq.xml -->
<!-- These credentials are shared by ALL producers. -->
<bean id="jmsTransportListener" class="com.rhd.ams.service.common.JmsTransportListener"
init-method="init" destroy-method="cleanup"/>
<bean id="amqJmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="${jms.publisher.broker.url}"/>
<property name="userName" value="${jms.publisher.username}"/>
<property name="password" value="${jms.publisher.password}"/>
<property name="transportListener" ref="jmsTransportListener"/>
</bean>
<!-- JmsTemplate, by default, will create a new connection, session, producer for -->
<!-- each message sent, then close them all down again. This is very inefficient! -->
<!-- PooledConnectionFactory will pool the JMS resources. It can't be used with consumers.-->
<bean id="pooledAmqJmsConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" destroy-method="stop">
<property name="connectionFactory" ref="amqJmsConnectionFactory" />
</bean>
<!-- Although JmsTemplate instance is unique for each message, it is -->
<!-- thread-safe and therefore can be injected into referenced obj's. -->
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<constructor-arg ref="pooledAmqJmsConnectionFactory"/>
</bean>

The way you describe the issue, it sure sounds like an open-and-shut case of JMS Durable Subscriptions. You might want to consider a more traditional implementation before going down this road. Caveats aside, ActiveMQ provides Advisory Messages which you can listen for and which will be sent for various events including new connections.
=========
Shoot, sorry... I did not understand what the issue was. I don't think Advisories are the solution at all.... after all, you need to be connected to the broker to get them, but being connected is what you know about.
So if I understand it correctly (prepare for retry #2....), what you need is a client connection which, when it fails, attempts to reconnect indefinitely. When it does reconnect, you want to trigger an event (or more) that flushes pending messages to the broker.
So detecting the lost connection is easy. You just register a JMS ExceptionListener. As far as detecting a reconnect, the simplest way I can think of is to start a reconnect thread. When it connects, stop the reconnect thread and notify interested parties using Observer/Observable or JMX notifications or the like. You could use the ActiveMQ Failover Transport which will do a connection retry loop for you, even if you only have one broker. At least, it is supposed to, but it's not doing that much for you that would not be done by your own reconnect thread... but if you're willing to delegate some control to it, it will cache your unflushed messages (see the trackMessages option), and then send them when it reconnects, which is sort of all of what you're trying to do.
I guess if your broker is down for a few minutes, that's not a bad way to go, but if you're talking hours, or you might accumulate 10k+ messages in the downtime, I just don't know if that cache mechanism is as reliable as you would need it to be.
==================
Mobile app ... right. Not really appropriate for the failover transport. Then I would implement a timer that periodically connects (might be a good idea to use the http transport, but not relevant). When it does connect, if there's nothing to flush, then see you in x minutes. If there is, send each message, wait for a handshake and purge the message from you mobile store. Then see you again in x minutes.
I assume this is Android ? If not, stop reading here. We actually implemented this some time ago. I only did the server side, but if I remember correctly, the connection timer/poller spun every n minutes (variable frequencies, I think, because getting too aggressive was draining the battery). Once a successful connection was made, I believe they used an intent broadcast to nudge the message pushers to do their thing. The thinking was that even though there was only one message pusher, we might add more.

spring freezes after number of frequent refresh / ajax calls

I have a spring application that seems to work fine, aside from the fact that after 5-6 requests, it halts and fails to handle any new incoming requests.
My page has a select dropdown, and onChange of that dropdown, makes an ajax call to the spring server. After about 5 or 6 of these, no more are accepted, and refreshing the page hangs indefinitely.
Any idea what may be causing this? Let me know if you need more information as far as configuration files and the like, but I was hoping this was a common enough problem where I could be pointed in the right direction.
Thanks
EDIT
here is my ajax code called onChange
$.ajax({
url: "./service.go?data="+data+",
dataType:"json",
timeout:15000,
cache: false,
success: function(data){
...
},
error: function(request,cause,data){
if (cause ==="timeout"){
alert("Request timed out!");
}
else{
alert("ERROR: " + data.responseText);
}
}
});
However, I don't think this is the issue, because even if i dont use ajax at all and jus hit refresh over and over in the browser, it fails.
With further testing, the problem does not occur when i hit a mapping that doesn't require database connectivity, so maybe it has something to do with my hibernate pool configuration? if I refresh a page that requires database connectivity the problem occurs on the 10th request, consistently. Here is my hibernate c3p0 configuration
driverClassName=com.sybase.jdbc3.jdbc.SybDriver
url=jdbc: HIDDEN
username=
password=
# Number of Connections a pool will try to acquire upon startup
initialPoolSize=5
# Minimum number of Connections a pool will maintain at any given time
minPoolSize=1
# Maximum number of Connections a pool will maintain at any given time
maxPoolSize=20
# Connections to acquire when the pool is exhausted
acquireIncrement=5
# Seconds a Connection can remain pooled but unused before being discarded. 30 Min Check
maxIdleTime=1800
#Test all idle, pooled but unchecked-out connections, every this number of seconds
idleConnectionTestPeriod=300
and using these properties i define my pool bean as follows
<bean id="dsrc" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="driverClass" value="${driverClassName}" />
<property name="jdbcUrl" value="${url}" />
<property name="user" value="${username}" />
<property name="password" value="${password}" />
<property name="initialPoolSize" value="${initialPoolSize}" />
<property name="minPoolSize" value="${minPoolSize}" />
<property name="maxPoolSize" value="${maxPoolSize}" />
<property name="acquireIncrement" value="${acquireIncrement}" />
<property name="maxIdleTime" value="${maxIdleTime}" />
<property name="idleConnectionTestPeriod" value="${idleConnectionTestPeriod}" />
</bean>
Here is the simple function in my controller that is hit, until he 10th time
#RequestMapping(method = RequestMethod.GET, value = "/test")
public #ResponseBody String test(){
System.out.println("Hello");
//List<Object> objects = objectService.getObjects(station); // calls hibernate DAO and when this is used instead of system.out, halts after 10th call.
return "";
}
So I know it is hitting the controller due to the printout, until the 10th time. I am unsure of how to tell thereafter if the request is hitting the server since I know at the least the mapping is not hit.

Probably something is wrong with your db connections handling. Might be they are not being returned to the pool. Make sure that you are closing all your connections.