Develop Reference

Java OJDBC Oracle SQL query hangs once every few months

We have a multi-threaded java application with a Web UI and REST API that is compiled using java 6 and runs in tomcat 6. During operations it uses OJDBC to access its Oracle DB millions of times a day. Once every two or three months one of the DB queries hangs and never returns, which causes part of the application to stop processing and a backlog to get created. Other threads are able to communicate with the DB and do their work, only one thread gets hung, which unfortunately stops file processing.
A thread dump shows that the thread is reading from a socket which never times out nor gets closed:
"FileUpload" daemon prio=10 tid=0x00002b8e60617800 nid=0xf9e runnable [0x00002b8e5e10b000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:311)
at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:312)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:257)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:182)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:99)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:121)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:77)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1173)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:309)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:200)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:543)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:238)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:1244)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1492)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1710)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4372)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4453)
- locked <0x00002b8e1c2d7010> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:6270)
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at xxx.OracleFileInfoDAO.getFilesToUpload(OracleFileInfoDAO.java:874)
When this happens the DBAs have looked on the DB server and do not see a long running query. The solution is to recycle tomcat which resolves the issue but I like to find out if there is a programmatic way to handle this. I have seen appends that refer to similar issues that are resolved by recycling the LINUX box that the DB server is running on, but that will not be an option for us; I need a fix at the application level.
The DB resource is defined using:
<Resource auth="Container" description="Oracle Datasource" name="xxx" scope="shareable" type="javax.sql.DataSource" url="jdbc:oracle:thin:#xxx:1521/xxx" driverClassName="oracle.jdbc.driver.OracleDriver" username="xxx" password="xxx" maxWait="5000" maxActive="100" maxIdle="20" removeAbandoned="true" testOnReturn="true" testOnBorrow="true" validationQuery="select 1 from dual" />
OJDBC driver used is: ojdbc6_g-11.2.0.4.0.jar
The DB version is: 11.2.0.3.0
The java code performing the query is:
con = CSAConnectionManager.getConnection();
StringBuilder strBuf = new StringBuilder(SQL_SELECT_FILE_INFO_TO_UPLOAD);
ps = con.prepareStatement(strBuf.toString());
ps.setString( 1, hostname );
ps.setString( 2, containerId );
ps.setMaxRows( maxRows );
Date before = new Date();
ResultSet rs = ps.executeQuery();
This is the source for getConnection():
public static Connection getConnection() throws Exception
{
return instance.getInstanceConnection();
}
public Connection getInstanceConnection() throws Exception
{
Connection con = null;
if(ds != null)
{
con = ds.getConnection();
}
else
{
String dburl = wrapper.getDBUrl();
String username = wrapper.getDBUserName();
String password = wrapper.getDBPassword();
String driverClass = wrapper.getDBDriverClass();
Class.forName(driverClass).newInstance();
con = DriverManager.getConnection(dburl,username,password);
}
con.setAutoCommit(false);
return con;
}
“ds” is defined as: private static DataSource ds = null;
And is initialized using:
Context initContext = new InitialContext();
ds = (DataSource)initContext.lookup(wrapper.getCSADBJNDIName());

In my experience, this is typically a network error. Your query has completed, but your client is still blocking on a network response it will never receive. This is why bouncing the app server works as it resets everything in the app server, but bouncing the DB server makes no sense since it is not a DB issue. Take a look this question/answer on this site...
Question on network timeouts

Check to see if the session is:
inactive - if inactive, what is the wait state, it may be waiting for client or network.
there are any blocking locks involving the idle session (e.g. it's holding some locks on uncommitted transactions).
Also check for connection storms (i.e. too many sessions hitting the instance can cause serious CPU issues)

How to properly keep a DB connection from a Connection Pool opened in JBoss

I'm using JBoss AS 7.1 as a server and I have my DataSource configured with pooling. I'm quite new to this so please excuse any rookie mistakes... after all I'm here to learn.
When a client logs-in it gets a connection to the database and I need to keep that connection(from the pool) open until the user logs-out or the HttpSession expires. This is an absolute requirement coming from our DB Admin. who says that he needs the DB session variables. I am using a servlet for all this.
Playing with the possibilities I have encountered 2 major problems:
As far as I see JBoss automatically closes unused connections => my opened connection returns to the pool. So this might not be the right path.
If I try to store/recall the Connection object like this:
private Hashtable<String, Connection> connections = new Hashtable<String, Connection>();
try {
String strDSName1 = "java:/OracleDSJNDI";
ctx = new InitialContext();
ds1 = (javax.sql.DataSource) ctx.lookup(strDSName1);
System.out.println("Got 1'st ds.");
} catch (Exception e) {
System.out.println("ERROR getting 1'st DS : " + e);
}
connection = ds1.getConnection();
connections.put(session.getId(), connection);
conn = (Connection) connections.get(sessionID);
it throws this exception:
java.sql.SQLException: Connection is not associated with a managed
connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6#dee1f37
My question is: How do I properly keep my connection opened?
Thanks

How do I properly keep my connection opened?
You must not do that, let the connection pool handle this.
Behind the scenes, the connection pool will keep a bunch of database connections to the database engine (MySQL, Oracle, SQL Server... depends how you configure it) in SLEEPING state. When you execute this code:
//avoiding all the particular exceptions just for code simplicity purposes...
//in real world applications, you must handle each of these exceptions
public Connection getConnection() throws Exception {
ctx = new InitialContext();
ds1 = (javax.sql.DataSource) ctx.lookup(strDSName1);
return ds1.getConnection();
}
You're asking to the connection pool to retrieve one of these connections available. The connection pool will give you a database connection (if available) and let you use it as long as you want. Then you use it wherever you want/need and close it:
public void foo() throws Exception {
Connection connection = getConnection();
//do what you want/need...
//in the end, you close the connection
//this is A MUST!
connection.close();
}
When executing connection.close() from a connection retrieved by the connection pool, you're not closing the physical database connection but notifying the connection pool this specific database connection must return to the SLEEPING state.
Some advices from the explanation:
You must not try to keep the connection alive, that's connection pool's job.
You must not try to store the connections in any cache-like structure, that's connection pool's job.
You must retrieve a java.sql.Connection in the shortest scope you will need it. Once you have used it, close it.

Your DBA is basically requiring you to avoid connection pooling by making the database connection equivalent to the user's session.
So one option is to not use the connection pool, and instead roll your own functionality that opens/closes the database connection around the user's session. That seems complicated and unusual though.
Another option is to examine the DBA's requirement. The DBA may have to adapt to the idea that he'll need to track state in a different way, e.g. by using a key related to the session to store the state he needs in a table, instead of storing state in the connection layer.
Generally speaking storing state in some component's session handling is adding indirect complexity, because you start having to care about how the component handles expiry and uniqueness, as you're finding here where the HTTP session state handles this differently from the database session.

Trying to use connection pooling outside servlet engine

I have a series of methods running within a servlet engine (Tomcat in this case), using connection pooling to access the database written in this way:
// Gets an RSS_Feed.
public static RSS_Feed get(int rssFeedNo) {
ConnectionPool_DB pool = ConnectionPool_DB.getInstance();
Connection connection = pool.getConnection();
PreparedStatement ps = null;
ResultSet rs = null;
String query = ("SELECT * " +
"FROM RSS_Feed " +
"WHERE RSSFeedNo = ?;");
try {
ps = connection.prepareStatement(query);
ps.setInt(1, rssFeedNo);
rs = ps.executeQuery();
if (rs.next()) {
return mapRSSFeed(rs);
}
else {
return null;
}
}
catch(Exception ex) {
logger.error("Error getting RSS_Feed " + rssFeedNo + "\n", ex);
return null;
}
finally {
Database_Utils.closeResultSet(rs);
Database_Utils.closeStatement(ps);
pool.freeConnection(connection);
}
}
Is it possible to call such a method outside of the servlet engine at all? I would like to do this in a batch process executed from the command line instead of within the servlet engine. I know I could simply rewrite the query without connection pooling but this is one of many queries involved in the process.
The connection pooling is implemented via Apache Common DBCP.
ConnectionPool_DB.getInstance(); reads:
private ConnectionPool_DB() {
try {
InitialContext ic = new InitialContext();
dataSource = (DataSource) ic.lookup(PropertiesFile.getProperty("myApp", "DATASOURCE"));
// dataSource = (DataSource) ic.lookup("java:/comp/env/jdbc/myApp");
}
catch(Exception ex) {
logger.error("Error getting a connection pool's datasource\n", ex);
}
}

I have something like this in a project:
Context ctx = new InitialContext();
DataSource ds = (DataSource) ctx.lookup("DbConnection");
ConnectionPool connectionPool = new ConnectionPool(ds)
And inside context xml I define the resouce like this
<Resource name="DbConnection"
auth="SERVLET"
type="javax.sql.DataSource"
scope="Shareable"
driverClassName="**driverClassName**"
url="**url**"
username="**username**"
password="**password**"
maxActive="10"
maxIdle="10"
maxWait="1000"
/>
So I assume you have something similar
If so you need to write code to create the DataSource yourself.
This should help you with that
http://docs.oracle.com/javase/tutorial/jdbc/basics/sqldatasources.html

Do you mean that you want to share a connection pool between your servlet engine and a batch job? Or that you want to use connection pooling within a batch job?
As to sharing a pool between Tomcat and a batch job: Hmm, I don't see how you'd do it. Tomcat and the batch job would each have their own instance of the Java Virtual Machine. They're not sharing memory, classes, etc, so I don't know where such a common pool would live.
If you mean within a batch job: Sure. I think such a thing is rarely necessary. In batch jobs I normally open a connection at the start of the program and close it at the end. There's not much value to creating a connection pool. Desktop apps are a little trickier. I often create a connection when the app starts and close it when they exit, but arguably this ties up a connection when the user is just staring blindly at the screen (like I often do for the hour or so before lunch), so other times I open a connection every time the user clicks a key that causes something to happen, then release it before going back to "wait" mode. Again, there's little point pooling because in a desktop app, there are no other users to share the pool with.
But can it be done? Sure. I've done it in desktop apps where many things could happen at various times and so it was awkward to pass a single connection around.

Sure, it could be used may be with slightly modification of the JNDI connectivity. But the Tomcat should run.

Why the Database connection is closed after every operation

I am using Connection pooling in TOmcat 6 and i have configued this way inside context.xml file
<Resource name="jdbc/myoracle" auth="Container"
type="javax.sql.DataSource" driverClassName="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:#127.0.0.1:1521:ORCLE"
username="scott" password="tiger" maxActive="20" maxIdle="10"
maxWait="-1"/>
And this is my Factory class to obtain the Connection using DataSource
public class ConnPoolFactory {
private static DataSource dataSource;
private static Connection connection;
private ConnPoolFactory() {
}
public static synchronized Connection getConnection() throws SQLException {
try {
if (connection == null) {
Context initContext = new InitialContext();
Context envContext = (Context) initContext
.lookup("java:/comp/env");
dataSource = (DataSource) envContext.lookup("jdbc/myoracle");
connection = dataSource.getConnection();
} else {
return connection;
}
} catch (NamingException e) {
e.printStackTrace();
}
return connection;
}
}
And from my servlet inside finally block , i am closing it this way
try {
connection = ConnPoolFactory.getConnection();
finally
{
if(conn!=null)
con.close();
}
From my User Interface i can give different commands (Button press) like Insert , Update , Delete , Select --.
The issue i am facing is that , the application runs only for one command , that is for example if i clicked Insert Button , it is inserting the record fine and after that , if once again i give any command there is a Exception on the Server console saying The Connection is closed .
If i remove that finally block code inside my servlet , then the application runs fine for any number of commands
Could anybody please let me know whats wrong with that finnaly block ??

The Connection variable in ConnPoolFactory should be method-local, not static. The variable you should be testing for null is not connection but dataSource. Once you have got a non-null value of that, you then return dataSource.getConnection(). The caller should then close that connection when he's finished with it.

Your factory is not aware that the connection is closed, and keeps handing it out. I'm assuming that's what you had in mind designing it.
You should either have your servlet turn back the connection after use, or the factory create a new connection every time.
Edit: trying to be more explicit:
Your servlet code will be called once per request. That's what servlet do. It seems to me that you close the connection after any request. Fine. However, your connection factory uses a static to store the connection it created. So on the second call, it will hand out a connection which has already been closed (in effect, it is a connection pool with a single connection rather than a factory).
BTW, you might encounter another nastier bug if you don't close your connection: you'll be sharing a connection between servlet threads if two requests come simultaneously, which might or might not work depending which DB operations you perform.

java jdbc mysql connector: how to resolve disconnection after a long idle time

I'm using red5 1.0.0rc1 to create an online game.
I'm connecting to a MySQL database using a jdbc mysql connector v5.1.12
it seems that after several hours of idle my application can continue running queries because the connection to the db got closed and i have to restart the application.
how can I resolve the issue ?
Kfir

The MySQL JDBC driver has an autoreconnect feature that can be helpful on occasion; see "Driver/Datasource Class Names, URL Syntax and Configuration Properties for Connector/J"1, and read the caveats.
A second option is to use a JDBC connection pool.
A third option is to perform a query to test that your connection is still alive at the start of each transaction. If the connection is not alive, close it and open a new connection. A common query is SELECT 1. See also:
Cheapest way to to determine if a MySQL connection is still alive
A simple solution is to change the MySQL configuration properties to set the session idle timeout to a really large number. However:
This doesn't help if your application is liable to be idle for a really long time.
If your application (or some other application) is leaking connections, increasing the idle timeout could mean that lost connections stay open indefinitely ... which is not good for database memory utilization.
1 - If the link breaks (again), please Google for the quoted page title then edit the answer to update it with the new URL.

Well, you reopen the connection.
Connection pools (which are highly recommended, BTW, and if you run Java EE your container - Tomcat, JBoss, etc - can provide a javax.sql.DataSource through JNDI which can handle pooling and more for you) validate connections before handing them out by running a very simple validation query (like SELECT 1 or something). If the validation query doesn't work, it throws away the connection and opens a new one.
Increasing the connection or server timeout tends to just postpone the inevitable.

I had the Same issue for my application and I have removed the idle time out tag
Thats it
It really worked fine
try this, I was using the Jboss server, in that i have made the following change in mysql-ds.xml file.
Let me know if you have any more doubts

The normal JDBC idiom is to acquire and close the Connection (and also Statement and ResultSet) in the shortest possible scope, i.e. in the very same try-finally block of the method as you're executing the query. You should not hold the connection open all the time. The DB will timeout and reclaim it sooner or later. In MySQL it's by default after 8 hours.
To improve connecting performance you should really consider using a connection pool, like c3p0 (here's a developer guide). Note that even when using a connection pool, you still have to write proper JDBC code: acquire and close all the resources in the shortest possible scope. The connection pool will in turn worry about actually closing the connection or just releasing it back to pool for further reuse.
Here's a kickoff example how your method retrieving a list of entities from the DB should look like:
public List<Entity> list() throws SQLException {
// Declare resources.
Connection connection = null;
Statement statement = null;
ResultSet resultSet = null;
List<Entity> entities = new ArrayList<Entity>();
try {
// Acquire resources.
connection = database.getConnection();
statement = connection.createStatement("SELECT id, name, value FROM entity");
resultSet = statement.executeQuery();
// Gather data.
while (resultSet.next()) {
Entity entity = new Entity();
entity.setId(resultSet.getLong("id"));
entity.setName(resultSet.getString("name"));
entity.setValue(resultSet.getInteger("value"));
entities.add(entity);
}
} finally {
// Close resources in reversed order.
if (resultSet != null) try { resultSet.close(); } catch (SQLException logOrIgnore) {}
if (statement != null) try { statement.close(); } catch (SQLException logOrIgnore) {}
if (connection != null) try { connection.close(); } catch (SQLException logOrIgnore) {}
}
// Return data.
return entities;
}
See also:
DAO tutorial - How to write proper JDBC code

Do you have a validationQuery defined (like select 1)? If not, using a validation query would help.

You can check here for a similar issue.
Append '?autoReconnect=true' to the end of your database's JDBC URL (without the quotes) worked for me.

I saw that ?autoReconnect=true wasn't working for me.
What I did, is simply creating a function called: executeQuery with:
private ResultSet executeQuery(String sql, boolean retry) {
ResultSet resultSet = null;
try {
resultSet = getConnection().createStatement().executeQuery(sql);
} catch (Exception e) {
// disconnection or timeout error
if (retry && e instanceof CommunicationsException || e instanceof MySQLNonTransientConnectionException
|| (e instanceof SQLException && e.toString().contains("Could not retrieve transation read-only status server"))) {
// connect again
connect();
// recursive, retry=false to avoid infinite loop
return executeQuery(sql,false);
}else{
throw e;
}
}
return resultSet;
}
I know, I'm using string to get the error.. need to do it better.. but it's a good start, and WORKS :-)
This will almost all reasons from a disconnect.