Develop Reference

Vert.x issue while inserting data into SQL Server

I am trying to insert data into a table in SQL Server hosted on AWS RDS.
It was working fine and suddenly I started getting an issue. It seems like an intermittent issue but I am unable to see why it is this happening
Fail to read any response from the server, the underlying connection might get lost unexpectedly.
This is how I am creating the database connection:
public static MSSQLPool createMssqlDbPool(Vertx vertx, ConfigModel configModel) {
MSSQLConnectOptions connectOptions = new MSSQLConnectOptions()
.setHost(System.getenv().getOrDefault("DB_HOST", configModel.getDbConfig().getHost()))
.setPort(Integer.parseInt(System.getenv().getOrDefault("DB_PORT", configModel.getDbConfig().getPort())))
.setDatabase(System.getenv().getOrDefault("DB_NAME", configModel.getDbConfig().getDatabase()))
.setUser(System.getenv().getOrDefault("DB_USER", configModel.getDbConfig().getUser()))
.setPassword(System.getenv().getOrDefault("DB_PASSWORD", configModel.getDbConfig().getPassword()));
// Pool options
PoolOptions poolOptions = new PoolOptions()
.setMaxSize(4);
LOG.info("DB connection : {}", connectOptions.toJson());
return MSSQLPool.pool(vertx, connectOptions, poolOptions);
}
I have read threads on GitHub about adding timeout but they are not definitive.

If you see this error it is likely a pooled connection has been idle for too long and closed by some intermediate proxy.
Change your pool options to close idle connections eagerly:
// Pool options
PoolOptions poolOptions = new PoolOptions()
.setMaxSize(4)
.setIdleTimeout(5)
.setIdleTimeoutUnit(TimeUnit.MINUTES);
5 minutes is just an example. The longer a connection lives, the better.

How to reset a session in a sql connection pool?

i have a connection pool using javax.sql.DataSource and PostgreSQL JDBC. The process is simple, I'm taking a virtual connection from connection pool, process it, then close the virtual connection (release the connection to connection pool). Everything works fine.
But when I get the same virtual connection second time the session have the same data from the previous execution. Mainly temp tables that created in the previous execution. How can I reset the session after each execution? Or can you suggest any work around to solve the issue without using creating temp table if not exists query syntax.
sample code
import java.sql.Connection;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.sql.DataSource;
public class Base {
public Connection getConnection() {
Context ctx = (Context) new InitialContext().lookup("java:/comp/env");
DataSource ds = (DataSource) ctx.lookup("database");
Connection con = ds.getConnection();
return con;
}
public Connection closeConnection( Connection con){
if (con != null || !con.isClosed()) {
con.close();
}
}
context.xml =>
<Resource
accessToUnderlyingConnectionAllowed="true"
auth="Container"
closeMethod="close"
driverClassName="org.postgresql.Driver"
maxTotal="50"
maxWaitMillis="60000"
removeAbandonedOnBorrow="true"
maxIdle="20"
name="Database"
password="p*******"
type="javax.sql.DataSource"
url="jdbc:postgresql://dbUrl/dbName"
username="dbUser"
validationQuery="select version();"
validationQueryTimeout="60"
/>
// use case
con = getconnection()
CallableStatement st = con.prepareCall("{ doSomething()}");
st.execute()
//processing
st.close()
con.close()
in doSomething() plpgsql function i have a temp table creation (with name t_table), data processing and an array output. when i repeatedly call this from java first execution is success. But from the second execution message saying that t_table already exist is thrown. I thought that when using connection pool, the connections in the pool are session independent IE. every new connections from pool will have a new session. PostgreSQL solution is to drop that particular temp table . But there is no other solution for this from the tomcat side ?

The discard temporary command can be used to clean up a session.
You can configure that as a validation query to be run when the connection is returned to the pool.
With the Tomcat JDBC pool, this would be something like this:
validationQuery="discard temporary"
testWhileIdle="false"
testOnBorrow="false"
testOnReturn="true"
If you want to clean up the session even more, you can consider using discard all instead, but that requires autocommit to be enabled, not sure if that would work for you.
If such a statement isn't allowed, you can wrap it into a function:
create function session_cleanup()
returns boolean
as
$$
discard temporary;
select true;
$$
language sql;
validationQuery="select session_cleanup()"
testWhileIdle="false"
testOnBorrow="false"
testOnReturn="true"

How to properly keep a DB connection from a Connection Pool opened in JBoss

I'm using JBoss AS 7.1 as a server and I have my DataSource configured with pooling. I'm quite new to this so please excuse any rookie mistakes... after all I'm here to learn.
When a client logs-in it gets a connection to the database and I need to keep that connection(from the pool) open until the user logs-out or the HttpSession expires. This is an absolute requirement coming from our DB Admin. who says that he needs the DB session variables. I am using a servlet for all this.
Playing with the possibilities I have encountered 2 major problems:
As far as I see JBoss automatically closes unused connections => my opened connection returns to the pool. So this might not be the right path.
If I try to store/recall the Connection object like this:
private Hashtable<String, Connection> connections = new Hashtable<String, Connection>();
try {
String strDSName1 = "java:/OracleDSJNDI";
ctx = new InitialContext();
ds1 = (javax.sql.DataSource) ctx.lookup(strDSName1);
System.out.println("Got 1'st ds.");
} catch (Exception e) {
System.out.println("ERROR getting 1'st DS : " + e);
}
connection = ds1.getConnection();
connections.put(session.getId(), connection);
conn = (Connection) connections.get(sessionID);
it throws this exception:
java.sql.SQLException: Connection is not associated with a managed
connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6#dee1f37
My question is: How do I properly keep my connection opened?
Thanks

How do I properly keep my connection opened?
You must not do that, let the connection pool handle this.
Behind the scenes, the connection pool will keep a bunch of database connections to the database engine (MySQL, Oracle, SQL Server... depends how you configure it) in SLEEPING state. When you execute this code:
//avoiding all the particular exceptions just for code simplicity purposes...
//in real world applications, you must handle each of these exceptions
public Connection getConnection() throws Exception {
ctx = new InitialContext();
ds1 = (javax.sql.DataSource) ctx.lookup(strDSName1);
return ds1.getConnection();
}
You're asking to the connection pool to retrieve one of these connections available. The connection pool will give you a database connection (if available) and let you use it as long as you want. Then you use it wherever you want/need and close it:
public void foo() throws Exception {
Connection connection = getConnection();
//do what you want/need...
//in the end, you close the connection
//this is A MUST!
connection.close();
}
When executing connection.close() from a connection retrieved by the connection pool, you're not closing the physical database connection but notifying the connection pool this specific database connection must return to the SLEEPING state.
Some advices from the explanation:
You must not try to keep the connection alive, that's connection pool's job.
You must not try to store the connections in any cache-like structure, that's connection pool's job.
You must retrieve a java.sql.Connection in the shortest scope you will need it. Once you have used it, close it.

Your DBA is basically requiring you to avoid connection pooling by making the database connection equivalent to the user's session.
So one option is to not use the connection pool, and instead roll your own functionality that opens/closes the database connection around the user's session. That seems complicated and unusual though.
Another option is to examine the DBA's requirement. The DBA may have to adapt to the idea that he'll need to track state in a different way, e.g. by using a key related to the session to store the state he needs in a table, instead of storing state in the connection layer.
Generally speaking storing state in some component's session handling is adding indirect complexity, because you start having to care about how the component handles expiry and uniqueness, as you're finding here where the HTTP session state handles this differently from the database session.

Trying to use connection pooling outside servlet engine

I have a series of methods running within a servlet engine (Tomcat in this case), using connection pooling to access the database written in this way:
// Gets an RSS_Feed.
public static RSS_Feed get(int rssFeedNo) {
ConnectionPool_DB pool = ConnectionPool_DB.getInstance();
Connection connection = pool.getConnection();
PreparedStatement ps = null;
ResultSet rs = null;
String query = ("SELECT * " +
"FROM RSS_Feed " +
"WHERE RSSFeedNo = ?;");
try {
ps = connection.prepareStatement(query);
ps.setInt(1, rssFeedNo);
rs = ps.executeQuery();
if (rs.next()) {
return mapRSSFeed(rs);
}
else {
return null;
}
}
catch(Exception ex) {
logger.error("Error getting RSS_Feed " + rssFeedNo + "\n", ex);
return null;
}
finally {
Database_Utils.closeResultSet(rs);
Database_Utils.closeStatement(ps);
pool.freeConnection(connection);
}
}
Is it possible to call such a method outside of the servlet engine at all? I would like to do this in a batch process executed from the command line instead of within the servlet engine. I know I could simply rewrite the query without connection pooling but this is one of many queries involved in the process.
The connection pooling is implemented via Apache Common DBCP.
ConnectionPool_DB.getInstance(); reads:
private ConnectionPool_DB() {
try {
InitialContext ic = new InitialContext();
dataSource = (DataSource) ic.lookup(PropertiesFile.getProperty("myApp", "DATASOURCE"));
// dataSource = (DataSource) ic.lookup("java:/comp/env/jdbc/myApp");
}
catch(Exception ex) {
logger.error("Error getting a connection pool's datasource\n", ex);
}
}

I have something like this in a project:
Context ctx = new InitialContext();
DataSource ds = (DataSource) ctx.lookup("DbConnection");
ConnectionPool connectionPool = new ConnectionPool(ds)
And inside context xml I define the resouce like this
<Resource name="DbConnection"
auth="SERVLET"
type="javax.sql.DataSource"
scope="Shareable"
driverClassName="**driverClassName**"
url="**url**"
username="**username**"
password="**password**"
maxActive="10"
maxIdle="10"
maxWait="1000"
/>
So I assume you have something similar
If so you need to write code to create the DataSource yourself.
This should help you with that
http://docs.oracle.com/javase/tutorial/jdbc/basics/sqldatasources.html

Do you mean that you want to share a connection pool between your servlet engine and a batch job? Or that you want to use connection pooling within a batch job?
As to sharing a pool between Tomcat and a batch job: Hmm, I don't see how you'd do it. Tomcat and the batch job would each have their own instance of the Java Virtual Machine. They're not sharing memory, classes, etc, so I don't know where such a common pool would live.
If you mean within a batch job: Sure. I think such a thing is rarely necessary. In batch jobs I normally open a connection at the start of the program and close it at the end. There's not much value to creating a connection pool. Desktop apps are a little trickier. I often create a connection when the app starts and close it when they exit, but arguably this ties up a connection when the user is just staring blindly at the screen (like I often do for the hour or so before lunch), so other times I open a connection every time the user clicks a key that causes something to happen, then release it before going back to "wait" mode. Again, there's little point pooling because in a desktop app, there are no other users to share the pool with.
But can it be done? Sure. I've done it in desktop apps where many things could happen at various times and so it was awkward to pass a single connection around.

Sure, it could be used may be with slightly modification of the JNDI connectivity. But the Tomcat should run.

Java ConnectionPool connection not closing, stuck in 'sleep'

I have a webapp that uses JNDI lookups to get a connection to the database.
The connection works fine and returns the query no problems. The issue us that the connection does not close properly and is stuck in the 'sleep' mode (according to mysql administrator). This means that they become unusable nad then I run out of connections.
Can someone give me a few pointers as to what I can do to make the connection return to the pool successfully.
public class DatabaseBean {
private static final Logger logger = Logger.getLogger(DatabaseBean.class);
private Connection conn;
private PreparedStatement prepStmt;
/**
* Zero argument constructor
* Setup generic databse connection in here to avoid redundancy
* The connection details are in /META-INF/context.xml
*/
public DatabaseBean() {
try {
InitialContext initContext = new InitialContext();
DataSource ds = (DataSource) initContext.lookup("java:/comp/env/jdbc/mysite");
conn = ds.getConnection();
}
catch (SQLException SQLEx) {
logger.fatal("There was a problem with the database connection.");
logger.fatal(SQLEx);
logger.fatal(SQLEx.getCause());
}
catch (NamingException nameEx) {
logger.fatal("There was a naming exception");
logger.fatal(nameEx);
logger.fatal(nameEx.getCause());
}
}
/**
* Execute a query. Do not use for statements (update delete insert etc).
*
* #return A ResultSet of the execute query. A set of size zero if no results were returned. It is never null.
* #see #executeUpdate() for running update, insert delete etc.
*/
public ResultSet executeQuery() {
ResultSet result = null;
try {
result = prepStmt.executeQuery();
logger.debug(prepStmt.toString());
}
catch (SQLException SQLEx) {
logger.fatal("There was an error running a query");
logger.fatal(SQLEx);
}
return result;
}
SNIP
public void close() {
try {
prepStmt.close();
prepStmt = null;
conn.close();
conn = null;
} catch (SQLException SQLEx) {
logger.warn("There was an error closing the database connection.");
}
}
}
This is inside a javabean that uses the database connection.
public LinkedList<ImportantNoticeBean> getImportantNotices() {
DatabaseBean noticesDBBean = new DatabaseBean();
LinkedList<ImportantNoticeBean> listOfNotices = new LinkedList<ImportantNoticeBean>();
try {
PreparedStatement preStmt = noticesDBBean.getConn().prepareStatement("SELECT pseudonym, message, date_to, date_from " +
"FROM importantnotices, users " +
"WHERE importantnotices.username = users.username " +
"AND NOW() >= date_from AND NOW() <= date_to;");
noticesDBBean.setPrepStmt(preStmt);
ResultSet result = noticesDBBean.executeQuery();
while (result.next()) {
ImportantNoticeBean noticeBean = new ImportantNoticeBean();
noticeBean.setAuthor(result.getString("pseudonym"));
noticeBean.setMessage(result.getString("message"));
noticeBean.setDateTo(result.getDate("date_to"));
noticeBean.setDateFrom(result.getDate("date_from"));
listOfNotices.add(noticeBean);
}
result.close();
} catch (SQLException SQLEx) {
logger.error("There was an error in ImportantNoticesBean.getImportantNotices()");
logger.error(SQLEx);
} finally {
noticesDBBean.close();
}
return listOfNotices;
}
<Context reloadable="true">
<Resource name="jdbc/mysite"
auth="Container"
type="javax.sql.DataSource"
username="user"
password="password"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/mysite"
maxActive="10"
maxIdle="5"
maxWait="6000"
removeAbandoned="true"
logAbandoned="false"
removeAbandonedTimeout="20"
/>
</Context>

You seem to be closing the connection properly - except for the case where prepStmt.close() throws a SQLException, I can't find a connection leak.
What pool implementation are you using? When you close a connection, the pool need not close the underlying MySQL connection immediately - after all that is the point of a connection pool! So from MySQL side, the connections would look alive, although your app is not using any; they might simply be held by the TC connection pool.
You might want to experiment with the settings of the connection pool.Ask it to shrink the pool when the system is idle. Or, ask it to refresh all connections periodically. Or, have a strict upper bound on the number of concurrent connections it ever gets from MySQL etc.
One way to check if your code has a connection leak is to force the ds.getConnection() to always open a new physical connection and conn.close() to release the connection (if your connection pool has settings for those). Then if you watch the connections on MySQL side, you might be able to figure out if the code really has a connection leak or not.

This is a similar question - Connection Pool Settings for Tomcat
This is my response to that question and it fixed the problem for the other guy. It may help you out too.
Tomcat Documentation
DBCP uses the Jakarta-Commons Database Connection Pool. It relies on number of Jakarta-Commons components:
* Jakarta-Commons DBCP
* Jakarta-Commons Collections
* Jakarta-Commons Pool
I'm using the same connection pooling stuff and I'm setting these properties to prevent the same thing it's just not configured through tomcat.
But if the first thing doesn't work try these.
testWhileIdle=true
timeBetweenEvictionRunsMillis=300000

Ok I might have this sorted. I have changed the database config resource to the following:
*SNIP*
maxActive="10"
maxIdle="5"
maxWait="7000"
removeAbandoned="true"
logAbandoned="false"
removeAbandonedTimeout="3"
*SNIP*
This works well enough for now. What is happening, afaik, is that once I reach the ten connections then Tomcat is checking for abandoned connections (idle time > 3). It does this in a batch job each time that max connections is reached. The potential issue with this is if i need more than 10 queries run at the same time (not unique to me). The important thing is that removeAbandonedTimeout is less than maxWait.
Is this what should be happening? ie Is this the way that the pool should operate? If it is is seems, at least to me, that you would wait until something (the connection) is broken before fixing rather than not letting it 'break' in the first place. Maybe I am still not getting it.

The issue us that the connection does not close properly and is stuck in the 'sleep' mode
This was actually only half right.
The problem I ran into was actually that each app was defining a new connection to the database sever. So each time I closed all the connections App A would make a bunch of new connections as per it's WEB.xml config file and run happily. App B would do the same. The problem is that they are independent pools which try to grab up to the server defined limit. It is a kind of race condition I guess. So when App A has finished with the connections it sits waiting to to use them again until the timeout has passed while App B who needs the connection now is denied the resources even though App A has finished with the and should be back in the pool. Once the timeout has passed, the connection is freed up and B (or C etc) can get at it again.
e.g. if the limit is 10 (mySQL profile limit) and each app has been configured to use a max of 10 the there will be 20 attempts at connections. Obviously this is a bad situation.
The solution is to RTFM and put the connection details in the right place. This does make shared posting a pain but there are ways around it (such as linking to other xml files from the context).
Just to be explicit: I put the connection details in the WEB.xml for each app and the had a fight about it.

One thing that #binil missed, you are not closing the result set in the case of an exception. Depending on the driver implementation this may cause the connection to stay open. Move the result.close() call to the finally block.

I am using the same configuration as you are. If the connection in mysql administrator(windows) shows that it is in sleep mode it only means that is pooled but not in use. I checked this running a test program program with multiple threads making random queries to Mysql. if it helps here is my configuration:
defaultAutoCommit="false"
defaultTransactionIsolation="REPEATABLE_READ"
auth="Container"
type="javax.sql.DataSource"
logAbandoned="true"
removeAbandoned="true"
removeAbandonedTimeout="300"
maxActive="-1"
initialSize="15"
maxIdle="10"
maxWait="10000"
username="youruser"
password="youruserpassword"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://yourhost/yourdatabase"/>