Develop Reference

Java OJDBC Oracle SQL query hangs once every few months

We have a multi-threaded java application with a Web UI and REST API that is compiled using java 6 and runs in tomcat 6. During operations it uses OJDBC to access its Oracle DB millions of times a day. Once every two or three months one of the DB queries hangs and never returns, which causes part of the application to stop processing and a backlog to get created. Other threads are able to communicate with the DB and do their work, only one thread gets hung, which unfortunately stops file processing.
A thread dump shows that the thread is reading from a socket which never times out nor gets closed:
"FileUpload" daemon prio=10 tid=0x00002b8e60617800 nid=0xf9e runnable [0x00002b8e5e10b000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:311)
at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:312)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:257)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:182)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:99)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:121)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:77)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1173)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:309)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:200)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:543)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:238)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:1244)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1492)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1710)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4372)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4453)
- locked <0x00002b8e1c2d7010> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:6270)
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at xxx.OracleFileInfoDAO.getFilesToUpload(OracleFileInfoDAO.java:874)
When this happens the DBAs have looked on the DB server and do not see a long running query. The solution is to recycle tomcat which resolves the issue but I like to find out if there is a programmatic way to handle this. I have seen appends that refer to similar issues that are resolved by recycling the LINUX box that the DB server is running on, but that will not be an option for us; I need a fix at the application level.
The DB resource is defined using:
<Resource auth="Container" description="Oracle Datasource" name="xxx" scope="shareable" type="javax.sql.DataSource" url="jdbc:oracle:thin:#xxx:1521/xxx" driverClassName="oracle.jdbc.driver.OracleDriver" username="xxx" password="xxx" maxWait="5000" maxActive="100" maxIdle="20" removeAbandoned="true" testOnReturn="true" testOnBorrow="true" validationQuery="select 1 from dual" />
OJDBC driver used is: ojdbc6_g-11.2.0.4.0.jar
The DB version is: 11.2.0.3.0
The java code performing the query is:
con = CSAConnectionManager.getConnection();
StringBuilder strBuf = new StringBuilder(SQL_SELECT_FILE_INFO_TO_UPLOAD);
ps = con.prepareStatement(strBuf.toString());
ps.setString( 1, hostname );
ps.setString( 2, containerId );
ps.setMaxRows( maxRows );
Date before = new Date();
ResultSet rs = ps.executeQuery();
This is the source for getConnection():
public static Connection getConnection() throws Exception
{
return instance.getInstanceConnection();
}
public Connection getInstanceConnection() throws Exception
{
Connection con = null;
if(ds != null)
{
con = ds.getConnection();
}
else
{
String dburl = wrapper.getDBUrl();
String username = wrapper.getDBUserName();
String password = wrapper.getDBPassword();
String driverClass = wrapper.getDBDriverClass();
Class.forName(driverClass).newInstance();
con = DriverManager.getConnection(dburl,username,password);
}
con.setAutoCommit(false);
return con;
}
“ds” is defined as: private static DataSource ds = null;
And is initialized using:
Context initContext = new InitialContext();
ds = (DataSource)initContext.lookup(wrapper.getCSADBJNDIName());

In my experience, this is typically a network error. Your query has completed, but your client is still blocking on a network response it will never receive. This is why bouncing the app server works as it resets everything in the app server, but bouncing the DB server makes no sense since it is not a DB issue. Take a look this question/answer on this site...
Question on network timeouts

Check to see if the session is:
inactive - if inactive, what is the wait state, it may be waiting for client or network.
there are any blocking locks involving the idle session (e.g. it's holding some locks on uncommitted transactions).
Also check for connection storms (i.e. too many sessions hitting the instance can cause serious CPU issues)

Tomcat JDBC connection pool (releasing connection)

Referring to Tomcat JBDC connection pool, I see in the standalone java example given there, one gets the connection using datasource.getConnection()which is cool. But in the finally block, it says con.close().
Question: When I implement this, it seems obvious that the con I get from datasource will be closed every time in the finally. When this is closed, will the connection pooling mechanism acquire a new connection and adds it to the pool?
I presume there should be a method call like releaseConnection() that will let the pool take its own decision whether to close it or let it be open for some other use.
I've also tried doing this ConnectionPool aPool = datasource.createPool();
But I see there is nothing like release connection on this aPool.
I think I'm missing something here?
Appreciate your help.
Code snippet from Tomcat JBDC connection pool:
DataSource datasource = new DataSource();
datasource.setPoolProperties(p);
Connection con = null;
try {
con = datasource.getConnection();
Statement st = con.createStatement();
ResultSet rs = st.executeQuery("select * from user");
int cnt = 1;
while (rs.next()) {
System.out.println((cnt++)+". Host:" +rs.getString("Host")+
" User:"+rs.getString("User")+" Password:"+rs.getString("Password"));
}
rs.close();
st.close();
} finally {
if (con!=null) try {con.close();}catch (Exception ignore) {}
}

Since you call the close() on a method obtained by the pool it is up to the pool what to do inside this method call. It does not neccessarily have to close the pooled database connection - it may do some cleanup and then add the connetion back to the pool.
This is already answered in Closing JDBC Connections in Pool

OK, my bad, that I did not see the implementation of DataSource.
It extends DataSourceProxy that internally creates a pool before returning a Connectionbased on the PoolProperties
I understand, its upto this DataSource to handle the connections, even though I close the con in finally, DataSource may take necessary action.
Do add a comment/reply if anybody thinks otherwise.

That example only shows how to create and use a data source. For connection pool on Tomcat you may configure JNDI.
// Sample
public static Connection getConnectionFromPool() {
Context initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");
DataSource ds = (DataSource) envCtx.lookup("jdbc/TestDB");
return ds.getConnection();
...
Quote from How connection pooling works in Java and JDBC:
A connection pool operates by performing the work of creating
connections ahead of time, In the case of a JDBC connection pool, a
pool of Connection objects is created at the time the application
server (or some other server) starts. These objects are then managed
by a pool manager that disperses connections as they are requested by
clients and returns them to the pool when it determines the client is
finished with the Connection object. A great deal of housekeeping is
involved in managing these connections.
When the connection pool server starts, it creates a predetermined
number of Connection objects. A client application would then perform
a JNDI lookup to retrieve a reference to a DataSource object that
implements the ConnectionPoolDataSource interface. The client
application would not need make any special provisions to use the
pooled data source; the code would be no different from code written
for a nonpooled DataSource.

PreparedStatement and Axis2

I'm using Axis2 with Apache Tomcat. I need the clients of the web service to be able to query a database, so I found on the internet that, by adding the following resource to the Tomcat's context, it will automatically create a connection pool:
Resource name="jdbc/TestDB" auth="Container"
type="javax.sql.DataSource"
removeAbandoned="true"
removeAbandonedTimeout="30"
maxActive="80"
maxIdle="30"
maxWait="10000"
username="user"
password=""
driverClassName="org.postgresql.Driver"
url = "jdbc:postgresql://localhost:5432/mydb"
useUnicode="true"
characterEncoding="utf-8"
characterSetResults="utf8"
validationQuery="/* ping */ SELECT 1"
It seems to be working, but now what I want to do is reuse the same PreparedStatement, so it won't be parsed every time a client makes a request. So, I made the PreparedStatement static for all client connections and when I create it, i invoke statement.setPoolable(true), which, from what I understand, is redundant (a PreparedStatement is already poolable). I hoped that this way, the PreparedStatement won't be tied to a single connection. Still, I get the error:
java.sql.SQLException: org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement with address * is closed.
The weird thing is, if i call statement.isClosed() before i make any set, it returns false. Then, I set something and it throws that exception.
The code is:
try {
Connection conn;
Context envCtx = (Context) new InitialContext().lookup("java:comp/env");
DataSource ds = (DataSource) envCtx.lookup("jdbc/TestDB");
conn = ds.getConnection();
if(statement == null){
//statement is static, so it won't be null at the second call
statement = conn.prepareStatement(query);
statement.setPoolable(true);
}
if(statement.isClosed()){
statement = conn.prepareStatement(query);
//it never gets here
}
else{
Logger.getLogger(HelloAxisWorld.class.getName()).log(Level.INFO, "STATEMENT IS NOT CLOSED", new Object());
//it always gets here
}
statement.setString(1, par1); //here, the second call throws an exception
ResultSet rs = statement.executeQuery();
while (rs.next()) {
cur_value = rs.getInt("cur_value");
}
rs.close();
conn.close();
}catch (Exception ex) {
Logger.getLogger(HelloAxisWorld.class.getName()).log(Level.SEVERE, null, ex);
}
I don't understand why does statement.isClosed return false, but then the exception says it is closed. Maybe this is not the way to reuse a preparedStatement, but then how can i do it? I read that, if I call conn.prepareStatement on the same query, jdbc will return the PreparedStatement from cache (it will not be parsed again), but I'm not sure if it's true.

There is a project called c3p0 which was made specifically to handle one of the cases you are having. The website is here https://sourceforge.net/projects/c3p0/ and here http://www.mchange.com/projects/c3p0/ It handles data-source JNDI binding, connection pooling and statement pooling. There also is a reference on stackoverflow on someone using this library with servlets what-is-a-good-strategy-for-caching-prepared-statements-in-tomcat

Why the Database connection is closed after every operation

I am using Connection pooling in TOmcat 6 and i have configued this way inside context.xml file
<Resource name="jdbc/myoracle" auth="Container"
type="javax.sql.DataSource" driverClassName="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:#127.0.0.1:1521:ORCLE"
username="scott" password="tiger" maxActive="20" maxIdle="10"
maxWait="-1"/>
And this is my Factory class to obtain the Connection using DataSource
public class ConnPoolFactory {
private static DataSource dataSource;
private static Connection connection;
private ConnPoolFactory() {
}
public static synchronized Connection getConnection() throws SQLException {
try {
if (connection == null) {
Context initContext = new InitialContext();
Context envContext = (Context) initContext
.lookup("java:/comp/env");
dataSource = (DataSource) envContext.lookup("jdbc/myoracle");
connection = dataSource.getConnection();
} else {
return connection;
}
} catch (NamingException e) {
e.printStackTrace();
}
return connection;
}
}
And from my servlet inside finally block , i am closing it this way
try {
connection = ConnPoolFactory.getConnection();
finally
{
if(conn!=null)
con.close();
}
From my User Interface i can give different commands (Button press) like Insert , Update , Delete , Select --.
The issue i am facing is that , the application runs only for one command , that is for example if i clicked Insert Button , it is inserting the record fine and after that , if once again i give any command there is a Exception on the Server console saying The Connection is closed .
If i remove that finally block code inside my servlet , then the application runs fine for any number of commands
Could anybody please let me know whats wrong with that finnaly block ??

The Connection variable in ConnPoolFactory should be method-local, not static. The variable you should be testing for null is not connection but dataSource. Once you have got a non-null value of that, you then return dataSource.getConnection(). The caller should then close that connection when he's finished with it.

Your factory is not aware that the connection is closed, and keeps handing it out. I'm assuming that's what you had in mind designing it.
You should either have your servlet turn back the connection after use, or the factory create a new connection every time.
Edit: trying to be more explicit:
Your servlet code will be called once per request. That's what servlet do. It seems to me that you close the connection after any request. Fine. However, your connection factory uses a static to store the connection it created. So on the second call, it will hand out a connection which has already been closed (in effect, it is a connection pool with a single connection rather than a factory).
BTW, you might encounter another nastier bug if you don't close your connection: you'll be sharing a connection between servlet threads if two requests come simultaneously, which might or might not work depending which DB operations you perform.

Java ConnectionPool connection not closing, stuck in 'sleep'

I have a webapp that uses JNDI lookups to get a connection to the database.
The connection works fine and returns the query no problems. The issue us that the connection does not close properly and is stuck in the 'sleep' mode (according to mysql administrator). This means that they become unusable nad then I run out of connections.
Can someone give me a few pointers as to what I can do to make the connection return to the pool successfully.
public class DatabaseBean {
private static final Logger logger = Logger.getLogger(DatabaseBean.class);
private Connection conn;
private PreparedStatement prepStmt;
/**
* Zero argument constructor
* Setup generic databse connection in here to avoid redundancy
* The connection details are in /META-INF/context.xml
*/
public DatabaseBean() {
try {
InitialContext initContext = new InitialContext();
DataSource ds = (DataSource) initContext.lookup("java:/comp/env/jdbc/mysite");
conn = ds.getConnection();
}
catch (SQLException SQLEx) {
logger.fatal("There was a problem with the database connection.");
logger.fatal(SQLEx);
logger.fatal(SQLEx.getCause());
}
catch (NamingException nameEx) {
logger.fatal("There was a naming exception");
logger.fatal(nameEx);
logger.fatal(nameEx.getCause());
}
}
/**
* Execute a query. Do not use for statements (update delete insert etc).
*
* #return A ResultSet of the execute query. A set of size zero if no results were returned. It is never null.
* #see #executeUpdate() for running update, insert delete etc.
*/
public ResultSet executeQuery() {
ResultSet result = null;
try {
result = prepStmt.executeQuery();
logger.debug(prepStmt.toString());
}
catch (SQLException SQLEx) {
logger.fatal("There was an error running a query");
logger.fatal(SQLEx);
}
return result;
}
SNIP
public void close() {
try {
prepStmt.close();
prepStmt = null;
conn.close();
conn = null;
} catch (SQLException SQLEx) {
logger.warn("There was an error closing the database connection.");
}
}
}
This is inside a javabean that uses the database connection.
public LinkedList<ImportantNoticeBean> getImportantNotices() {
DatabaseBean noticesDBBean = new DatabaseBean();
LinkedList<ImportantNoticeBean> listOfNotices = new LinkedList<ImportantNoticeBean>();
try {
PreparedStatement preStmt = noticesDBBean.getConn().prepareStatement("SELECT pseudonym, message, date_to, date_from " +
"FROM importantnotices, users " +
"WHERE importantnotices.username = users.username " +
"AND NOW() >= date_from AND NOW() <= date_to;");
noticesDBBean.setPrepStmt(preStmt);
ResultSet result = noticesDBBean.executeQuery();
while (result.next()) {
ImportantNoticeBean noticeBean = new ImportantNoticeBean();
noticeBean.setAuthor(result.getString("pseudonym"));
noticeBean.setMessage(result.getString("message"));
noticeBean.setDateTo(result.getDate("date_to"));
noticeBean.setDateFrom(result.getDate("date_from"));
listOfNotices.add(noticeBean);
}
result.close();
} catch (SQLException SQLEx) {
logger.error("There was an error in ImportantNoticesBean.getImportantNotices()");
logger.error(SQLEx);
} finally {
noticesDBBean.close();
}
return listOfNotices;
}
<Context reloadable="true">
<Resource name="jdbc/mysite"
auth="Container"
type="javax.sql.DataSource"
username="user"
password="password"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/mysite"
maxActive="10"
maxIdle="5"
maxWait="6000"
removeAbandoned="true"
logAbandoned="false"
removeAbandonedTimeout="20"
/>
</Context>

You seem to be closing the connection properly - except for the case where prepStmt.close() throws a SQLException, I can't find a connection leak.
What pool implementation are you using? When you close a connection, the pool need not close the underlying MySQL connection immediately - after all that is the point of a connection pool! So from MySQL side, the connections would look alive, although your app is not using any; they might simply be held by the TC connection pool.
You might want to experiment with the settings of the connection pool.Ask it to shrink the pool when the system is idle. Or, ask it to refresh all connections periodically. Or, have a strict upper bound on the number of concurrent connections it ever gets from MySQL etc.
One way to check if your code has a connection leak is to force the ds.getConnection() to always open a new physical connection and conn.close() to release the connection (if your connection pool has settings for those). Then if you watch the connections on MySQL side, you might be able to figure out if the code really has a connection leak or not.

This is a similar question - Connection Pool Settings for Tomcat
This is my response to that question and it fixed the problem for the other guy. It may help you out too.
Tomcat Documentation
DBCP uses the Jakarta-Commons Database Connection Pool. It relies on number of Jakarta-Commons components:
* Jakarta-Commons DBCP
* Jakarta-Commons Collections
* Jakarta-Commons Pool
I'm using the same connection pooling stuff and I'm setting these properties to prevent the same thing it's just not configured through tomcat.
But if the first thing doesn't work try these.
testWhileIdle=true
timeBetweenEvictionRunsMillis=300000

Ok I might have this sorted. I have changed the database config resource to the following:
*SNIP*
maxActive="10"
maxIdle="5"
maxWait="7000"
removeAbandoned="true"
logAbandoned="false"
removeAbandonedTimeout="3"
*SNIP*
This works well enough for now. What is happening, afaik, is that once I reach the ten connections then Tomcat is checking for abandoned connections (idle time > 3). It does this in a batch job each time that max connections is reached. The potential issue with this is if i need more than 10 queries run at the same time (not unique to me). The important thing is that removeAbandonedTimeout is less than maxWait.
Is this what should be happening? ie Is this the way that the pool should operate? If it is is seems, at least to me, that you would wait until something (the connection) is broken before fixing rather than not letting it 'break' in the first place. Maybe I am still not getting it.

The issue us that the connection does not close properly and is stuck in the 'sleep' mode
This was actually only half right.
The problem I ran into was actually that each app was defining a new connection to the database sever. So each time I closed all the connections App A would make a bunch of new connections as per it's WEB.xml config file and run happily. App B would do the same. The problem is that they are independent pools which try to grab up to the server defined limit. It is a kind of race condition I guess. So when App A has finished with the connections it sits waiting to to use them again until the timeout has passed while App B who needs the connection now is denied the resources even though App A has finished with the and should be back in the pool. Once the timeout has passed, the connection is freed up and B (or C etc) can get at it again.
e.g. if the limit is 10 (mySQL profile limit) and each app has been configured to use a max of 10 the there will be 20 attempts at connections. Obviously this is a bad situation.
The solution is to RTFM and put the connection details in the right place. This does make shared posting a pain but there are ways around it (such as linking to other xml files from the context).
Just to be explicit: I put the connection details in the WEB.xml for each app and the had a fight about it.

One thing that #binil missed, you are not closing the result set in the case of an exception. Depending on the driver implementation this may cause the connection to stay open. Move the result.close() call to the finally block.

I am using the same configuration as you are. If the connection in mysql administrator(windows) shows that it is in sleep mode it only means that is pooled but not in use. I checked this running a test program program with multiple threads making random queries to Mysql. if it helps here is my configuration:
defaultAutoCommit="false"
defaultTransactionIsolation="REPEATABLE_READ"
auth="Container"
type="javax.sql.DataSource"
logAbandoned="true"
removeAbandoned="true"
removeAbandonedTimeout="300"
maxActive="-1"
initialSize="15"
maxIdle="10"
maxWait="10000"
username="youruser"
password="youruserpassword"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://yourhost/yourdatabase"/>