Develop Reference

How to set timeout for getConnection using commons-dbcp BasicDataSource?

I am using commons-dbcp2 for creating a connection pool to the database. When the database is down dataSource.getConnection() method takes 20 seconds then throws an exception. I want to configure DataSource to dynamically change the timeout e.g. 5 seconds.
I tried dataSource.setLoginTimeout(), but it is not supported for BasicDataSource
BasicDataSource dataSource = new BasicDataSource();
dataSource.setDriverClassName(driverName);
dataSource.setUrl(url);
dataSource.setUsername(username);
dataSource.setPassword(password);
dataSource.setInitialSize(3);
dataSource.setMaxTotal(100);
dataSource.setValidationQuery(validationquery);
dataSource.setTestOnBorrow(true);
dataSource.setRemoveAbandonedOnBorrow(true);
try (Connection connection = dataSource.getConnection()) {
} catch (Exception e) {
}
I want after 5 seconds (as I configured) it throws the exception.

You can try validationQueryTimeout parameter which lets you time out the validation query after X seconds:
dataSource.setValidationQueryTimeout(5);
dataSource.setTestOnBorrow(true);
You don't have to set a validation SQL query, modern JDBC driver has Connection.isValid().
Unfortunately DBCP pool has issues as per Bad Behavior: Handling Database Down due to operating system TCP timeout limit. When the test was done in 2017:
Dbcp2 did not return a connection, and also did not timeout. The execution of the validation query is stuck due to unacknowledged TCP traffic. Subsequently, the SQL Statement run on the (bad) connection by the test harness hangs (due to unacknowledged TCP). setMaxWait(5000) is seemingly useless for handling network outages. There are no other meaningful timeout settings that apply to a network failure.

You can set datasource.setValidationQueryTimeout(), but keep in mind that this is for query execution. If you have network issues, you might still be stuck. For this, you need to set also setSoTimeout(), which is for the socket. The default value is 0, which means infinity.

Oracle data source connection pooling not working used with Spring and JDBCTemplate

Question: Lot of active unclosed physical connections with database even with connection pooling. Can someone tell me why is it so?
I configured the connection pool settings using oracle.jdbc.pool.OracleDataSource. However it seems the physical connections are not getting closed after use.
I thought, Since it is connection pooling, the connections will be reused from the pool, so so many physical connections will not be made,
but thats not what is happening now!
There are 100+ active physical connections in the database generating from the application [not from plsql developer or any such client tools],
due to which it kicks off TNS error while trying to do write operations on database,
where as read operations are fine even with large number of active connections.
Here is the Spring configuration,
<bean id="oracleDataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close"
p:URL="${url}"
p:user="${username}"
p:password="${password}"
p:connectionCachingEnabled="true">
<property name="connectionProperties">
<props merge="default">
<prop key="AutoCommit">false</prop>
</props>
</property>
</bean>
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate"
p:dataSource-ref="oracleDataSource" />
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager"
p:dataSource-ref="oracleDataSource">
</bean>
The SQL that returned the 100+ active connections is ,
select username, terminal,schemaname, osuser,program from v$session where username = 'grduser'

You should configure connection cache, the default value of max connections for implicit connection cache is the max number of database sessions configured for the database.

Thanks to #Evgeniy Dorofeev.
Solution in detail :
The connectionCache was enabled, but the properties were not set.
Set the properties as written below,
`
<bean id="oracleDataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close"
p:URL="${url}"
p:user="${username}"
p:password="${password}"
p:connectionCachingEnabled="true">
<property name="connectionProperties">
<props merge="default">
<prop key="AutoCommit">false</prop>
</props>
</property>
<property name="connectionCacheProperties">
<props>
<prop key="MinLimit">5</prop>
<prop key="MaxLimit">10</prop>
<prop key="InactivityTimeout">2</prop>
</props>
</property>
</bean>
`
Now, for every operation in the application that requires a connection, it will try to get from the pool if available and ready to use, but is guaranteed that the database will have maximum of only 10 active physical connections. Any attempt to get a extra physical connection will lead to a database error at the application side.
Even if you have set the connectionCache, make sure your application is not explicitly trying to get a connection, like
Connection connection = getJdbcTemplate().getDataSource().getConnection();
This is alarming, JDBCTemplate doesnt manage the closing of this connection. Hence you have to close by yourself after use, otherwise the physical connection will still be active and unclosed even after use. So next time you call this again, it try to get a new physical connection , and remain unclosed, resulting in piling up of active connections until the maxLimit is reached.
The connection might be explicity needed when you want to pass it as a parameter to some other function, say in the case of an ArrayDescriptor [if you talk to PLSQL Stored procedures that has IN parameter to accept an array of values , an array of Varchar or array of RAW]. If you need to create a ArrayDescriptor,
ArrayDescriptor arrayDescriptor = ArrayDescriptor.createDescriptor(
"SOME_TYPE_NAME", connection );
ARRAY SQLArray= new ARRAY(arrayDescriptor, connection , arrayString);
Hence do a connection.close() explicity here.
Additional info:
Connection connection = getJdbcTemplate().getDataSource().getConnection()
This attempts to establish a connection with the data source that this DataSource object represents.
Calling this line of code - once, will attempt to establish a new connection.
Calling again, will establish a second connection. For each request, it will create a new connection!.So If your maxLimit is 10,
Until there are 10 active physical connections in the database, the call will be successful, but
note that all the connections are active [not closed].
So lets say now there are 10 active db connections, as maxLimit is set to 10.
So any requests that requires a database operation, that would go through the
normal route of accessing a connection via the JDBCTemplate will be picking up the already established connection [from the 10 connections]
However any request that calls this code getJdbcTemplate().getDataSource().getConnection() to access a connection
will attempt to establish a NEW connection, and will fail, resulting in exception.
The only way to resolve this is to explicitly close the connection when we explicitly create the connection.
ie calling connection.close()
When we don't explicitly create the connection, and when it is managed by Spring, then Spring will take care of closing
the connections too. In the case of using Oracle Data Source pooling along with JDBCTemplate, closing the connection[returning the
connections to the pool] is managed by Spring.

Error: System Resource Exceeded

A console application executing under:
1). Multiple threads
2). Connection Pooling (as the database connections range could be 5 to 30) of type Microsoft Access using DBCP.
While executing this application at my end (not tested the database limit) it works fine. And whenever I try to introduce the same application on one of other machines it generates an error.
I'm wondering why this is happening as there is only the difference of machines here. So, it works perfectly at my end.
I don't know much about connection pooling but it seems whatever I have understood I have implemented as:
public class TestDatabases implements Runnable{
public static Map<String, Connection> correctDatabases;
#Override
public void run() {
// validating the databases using DBCP
datasource.getConnection(); // Obtaining the java.sql.Connection from DataSource
// if validated successfully °º¤ø,¸¸,ø¤º°`°º¤ø,¸,ø¤°º¤ø,¸¸,ø¤º°`°º¤ø,¸ putting them in correctDatabases
}
}
The above case is implemented using ExecutorService = Number of databases.
Finally, I'm trying to put them in a static Collection of Type
Map<String, Connection> and making use of it throughout the application. In other words: I'm trying to collect the connectionString along with the Connection in a Map.
In other parts of my application I'm simply dealing with multiple threads coming along with the Connection URL. So, to perform any database operations I'm calling the
Connection con = TestDatabases.correctDatases.get(connectUrl);
For that machine, this application works fine for around ~5 databases. And the error is always getting generated when I'm trying to fire the query using above Connection (con) as stmt.executeQuery(query);
As, I'm not able to reproduce this issue at my end, it seems something is going-on wrong with the Connection Pooling or I have not configured my application to deal with Connection Pooling correctly.
Just for your information, I'm correctly performing Connection close in finally block where my application terminates and this Application is using Quartz Scheduler as well. For Connection Pooling, a call to the following from TestDatabases class is done for setUp as:
public synchronized DataSource setUp() throws Exception {
Class.forName(RestConnectionValidator.prop.getProperty("driverClass")).newInstance();
log.debug("Class Loaded.");
connectionPool = new GenericObjectPool();
log.debug("Connection pool made.");
connectionPool.setMaxActive(100);
ConnectionFactory cf = new DriverManagerConnectionFactory(
RestConnectionValidator.prop.getProperty("connectionUrl")+new String(get().toString().trim()),
"","");
PoolableConnectionFactory pcf =
new PoolableConnectionFactory(cf, connectionPool,
null, null, false, true);
return new PoolingDataSource(connectionPool);
}
Following is the error I'm getting (at the other machine)
java.sql.SQLException: [Microsoft][ODBC Microsoft Access Driver] System resource exceeded.
Following is the Database Path:
jdbc:odbc:DRIVER= {Microsoft Access Driver (*.mdb, *.accdb)};DBQ=D:\\DataSources\\PR01.mdb
Each of those database seems to be not much heavy (its ~ 5 to 15 MB of total size).
So, I'm left with the following solutions:
1). Correction of Connection Pooling or migrate to the newer one's like c3p0 or DBPool or BoneCP.
2). Introducing batch concept - in which I will schedule my application for each group of 4 databases. It could be very expensive to deal with as any time the other schedule may also collapse.
I’m pretty sure that this is Java related error but I can’t fathom out why.

Just done the migration to BoneCP which solved my problem. I guess due to multi-threaded environment the dpcp was not providing the connection from pool rather it was trying to hit the database again and again. Maybe I could have solved the dpcp issue but migrating to BoneCP also provides advantage of performance.

How do I recreate a mysql data source connection, if the connection fails, in java?

I have created a mysqlDatasource connection using the following code:
MysqlDataSource d = new MysqlDataSource();
d.setUser("user");
d.setPassword("pass");
d.setServerName("hostname.com");
d.setDatabaseName("db");
Connection c = d.getConnection();
If Im running my application and the connections are disconnected because mysql restarted or for some other reason, the remaining operations will fail even if the mysql server instance is running.
In that case I want to recreate a connection? Is this possible? How do I go about doing this?

In most cases when you're creating multiple connections and juggling them, it's better to use a connection pool, where the Connection objects are just that, objects, and are multiplexed to actual socket connections handled by an underlying implementation. This means that connections are created automatically when you need them and you don't need to worry about reclaiming resources and creating an appropriate number of connections.
Two prominent examples are BoneCP and C3P0.

The other option, in addition to Mr Mao's, is to define the url explicitly using setURL to allow for automatic reconnection, using the autoReconnect parameter, do note, though, that this approach is not recommended. The answer is provided here only for completeness.

Just take an help of DBCP CONNECTION and create your Connection pool like below code. You can see there is validation query which pings database at regular interval and refreshes the pool. There are other property like validateConnection you can set this property to true.
BasicDataSource dataSource = new BasicDataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUsername("username");
dataSource.setPassword("password");
dataSource.setUrl("jdbc:mysql://<host>:<port>/<database>");
dataSource.setMaxActive(50);
dataSource.setMaxIdle(5);
dataSource.setInitialSize(5);
dataSource.setValidationQuery("SELECT 1");

Try this.
String driver = "com.microsoft.jdbc.sqlserver.SQLServerDriver";
Class.forName(driver);
String url = "jdbc:microsoft:sqlserver://host:1433/database";
Connection conn = DriverManager.getConnection(url, "username", "password");

Data source rejected establishment of connection, message from server: "Too many connections"

I am using hibenate and spring and getting below exception ween we hit from jmeter with 250 users
"Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections"
hibernate_cfg.xml
<property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/my_db</property>
<property name="hibernate.connection.username">user1</property>
<property name="hibernate.connection.pool_size">1</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">50</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">500</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.hbm2ddl.auto">update</property>`
Spring
<bean id="dataSource" scope="prototype" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName">
<value>${dbDriver}</value>
</property>
<property name="url">
<value>${dbURL}</value>
</property>
<property name="username">
<value>${dbUsername}</value>
</property>
<property name="password">
<value>${dbPassword}</value>
</property>
</bean>

This is a message from the server, so, I'd check what's the number of connected clients that the server is reporting. If this is an expected number, like 500 or so, then I'd increase this limit on the server, if you really expect that level of concurrency for your application. Otherwise, reduce the number of clients.
A bit of background on how it works: each client is a thread on the server, and each thread will consume at least one connection. If you are doing it right, the connection will return to the pool once the thread finishes (ie: once the response is sent to the client). So, in the best case, you'd have 500 connections if you have around 500 users connected. If are seeing a number close to a multiple of the number of concurrent users (ie: 2 users, 4 connections), then you might be consuming more than one connection per thread (that's the price you pay for not using the data source provided by your application server, if you are using one). If you are seeing a really high number (like, 10 times the number of users), then you might have a connection leak somewhere. This might happen if you forget to close the connection.
I'd really suggest to use an EntityManager managed by your application server, and using a DataSource also provided by it. Then, you'd not have to worry about managing the connection pooling.

I think your problem is that your datasource doesn't have a connections pool.
from org.springframework.jdbc.datasource.DriverManagerDataSource javadocs:
NOTE: This class is not an actual connection pool; it does not actually
* pool Connections.
It is also say in the javadocs of that class to use Apache's Jakarta Commons DBCP in case u do need connection pool.
If you need a "real" connection pool outside of a J2EE container,
consider Apache's
Jakarta Commons DBCP or C3P0. Commons DBCP's
BasicDataSource and C3P0's ComboPooledDataSource are full connection
pool beans, supporting the same basic properties as this class plus
specific settings (such as minimal/maximal pool size etc).
I used it and it worked like a charm :)
Hoped I helped.

if connection created every time it may caused this problem. Solution is simple. Single on. connection for each sessin. So that I am publishing some code on bottom of my post. Check incorrect and correct coding. 1-incorrect,2-correct
private EntityManagerFactory emf = null;
#SuppressWarnings("unchecked")
public BaseDAO() {
emf = Persistence.createEntityManagerFactory("aaHIBERNATE");
persistentClass = (Class<T>) ((ParameterizedType) getClass()
.getGenericSuperclass()).getActualTypeArguments()[0];
}
private static EntityManagerFactory emf = null;
#SuppressWarnings("unchecked")
public BaseDAO() {
if (emf == null) {
emf = Persistence.createEntityManagerFactory("aaHIBERNATE");}
persistentClass = (Class<T>) ((ParameterizedType) getClass()
.getGenericSuperclass()).getActualTypeArguments()[0];
}

You need to trace it in application and database server both.
Check the database configuration for possible maximum open connections.
If you are using the database connections from another clients, Check for the current opened connections in database server.
Check your application if you are opening too many connections and not closing them.
You need to increase the value of max open connections in database server settings.
To change max open connections you need to edit max_connections and max_user_connections in my.cnf file under database server.
You can also grant/edit max number of connections with per user. more info available here