The number of connections to DB exceeds the permissible limit.
This is what I tried so far.
When the user successfully logs in I add one connection object to the session:
Connection conn = DatabaseConnectionManager.getConnection();
sessionMap.put("Connection", conn);
then, whenever I need a DB connection, I fetch it from the session:
Map<String, Object> sessionMap = (Map<String, Object>) ActionContext.getContext().get("session");
Connection conn = (Connection) sessionMap.get("Connection");
In the getConnection() method I print the number of times the method is called.
So although I fetch the Connection object from the session why does the number of connections exceed the permissible limit which is 50 ?
JNDI Code:
Connection conn = null;
try {
Context initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");
DataSource ds = (DataSource) envCtx.lookup("jdbc/MySqdb");
conn = ds.getConnection();
}
catch (NamingException ex) {
Logger.getLogger(DatabaseConnectionManager.class.getName()).log(Level.SEVERE, null, ex);
}
catch (SQLException sqle) {
sqle.printStackTrace();
}
System.out.println("connection: "+connection++);
return conn;
I have used JNDI. The number of connections is within the permissible limit. I'm not sure if this is the right way, please advise.
One connection for each user is not a good solution at all. As mentioned you must use some connection poll and if you want something simple use https://commons.apache.org/proper/commons-dbcp/ .
As mentioned there:
Creating a new connection for each user can be time consuming (often
requiring multiple seconds of clock time), in order to perform a
database transaction that might take milliseconds. Opening a
connection per user can be unfeasible in a publicly-hosted Internet
application where the number of simultaneous users can be very large.
Accordingly, developers often wish to share a "pool" of open
connections between all of the application's current users. The number
of users actually performing a request at any given time is usually a
very small percentage of the total number of active users, and during
request processing is the only time that a database connection is
required. The application itself logs into the DBMS, and handles any
user account issues internally.
You can find samples at http://svn.apache.org/viewvc/commons/proper/dbcp/trunk/doc/
Related
I'm using JBoss AS 7.1 as a server and I have my DataSource configured with pooling. I'm quite new to this so please excuse any rookie mistakes... after all I'm here to learn.
When a client logs-in it gets a connection to the database and I need to keep that connection(from the pool) open until the user logs-out or the HttpSession expires. This is an absolute requirement coming from our DB Admin. who says that he needs the DB session variables. I am using a servlet for all this.
Playing with the possibilities I have encountered 2 major problems:
As far as I see JBoss automatically closes unused connections => my opened connection returns to the pool. So this might not be the right path.
If I try to store/recall the Connection object like this:
private Hashtable<String, Connection> connections = new Hashtable<String, Connection>();
try {
String strDSName1 = "java:/OracleDSJNDI";
ctx = new InitialContext();
ds1 = (javax.sql.DataSource) ctx.lookup(strDSName1);
System.out.println("Got 1'st ds.");
} catch (Exception e) {
System.out.println("ERROR getting 1'st DS : " + e);
}
connection = ds1.getConnection();
connections.put(session.getId(), connection);
conn = (Connection) connections.get(sessionID);
it throws this exception:
java.sql.SQLException: Connection is not associated with a managed
connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6#dee1f37
My question is: How do I properly keep my connection opened?
Thanks
How do I properly keep my connection opened?
You must not do that, let the connection pool handle this.
Behind the scenes, the connection pool will keep a bunch of database connections to the database engine (MySQL, Oracle, SQL Server... depends how you configure it) in SLEEPING state. When you execute this code:
//avoiding all the particular exceptions just for code simplicity purposes...
//in real world applications, you must handle each of these exceptions
public Connection getConnection() throws Exception {
ctx = new InitialContext();
ds1 = (javax.sql.DataSource) ctx.lookup(strDSName1);
return ds1.getConnection();
}
You're asking to the connection pool to retrieve one of these connections available. The connection pool will give you a database connection (if available) and let you use it as long as you want. Then you use it wherever you want/need and close it:
public void foo() throws Exception {
Connection connection = getConnection();
//do what you want/need...
//in the end, you close the connection
//this is A MUST!
connection.close();
}
When executing connection.close() from a connection retrieved by the connection pool, you're not closing the physical database connection but notifying the connection pool this specific database connection must return to the SLEEPING state.
Some advices from the explanation:
You must not try to keep the connection alive, that's connection pool's job.
You must not try to store the connections in any cache-like structure, that's connection pool's job.
You must retrieve a java.sql.Connection in the shortest scope you will need it. Once you have used it, close it.
Your DBA is basically requiring you to avoid connection pooling by making the database connection equivalent to the user's session.
So one option is to not use the connection pool, and instead roll your own functionality that opens/closes the database connection around the user's session. That seems complicated and unusual though.
Another option is to examine the DBA's requirement. The DBA may have to adapt to the idea that he'll need to track state in a different way, e.g. by using a key related to the session to store the state he needs in a table, instead of storing state in the connection layer.
Generally speaking storing state in some component's session handling is adding indirect complexity, because you start having to care about how the component handles expiry and uniqueness, as you're finding here where the HTTP session state handles this differently from the database session.
so I am currently working on a java application that is supposed to log specific events into a database. I expect that there will be at most 15 to 20 inserts per minute, basically I was wondering if I should make a new connection for every insert statement or keep one open as long as the application is running.
What I am doing is:
public void logEvent(MyCustomEvent e) {
Connection con = null;
Statement st = null;
ResultSet rs = null;
try {
con = DriverManager.getConnection(url, user, password);
st = con.createStatement();
st.executeUpdate("INSERT INTO Table(" + e.data + ");");
} catch (SQLException ex) {
Logger lgr = Logger.getLogger(MySQLConnector.class.getName());
lgr.log(Level.SEVERE, ex.getMessage(), ex);
} finally {
try {
if (rs != null) {
rs.close();
}
if (st != null) {
st.close();
}
if (con != null) {
con.close();
}
} catch (SQLException ex) {
Logger lgr = Logger.getLogger(MySQLConnector.class.getName());
lgr.log(Level.SEVERE, ex.getMessage(), ex);
}
}
}
Is there no problem in making a new connection every time or can/should I buffer the inputs somehow?
Making connections is expensive so it's probably best not to keep making them. However, holding one connection open all the time has its own problems ( what happens if it closes for some reason? )
Why not have a look at database connection pooling - google will show up several competing connection pool implementations for you. You'll get the best of both worlds. To your application, the connection will appear to be permanently open, but if the connection fails for some reason, the connection pool will handle re-opening it for you.
You should keep the connection open and reuse it. Starting up and tearing down a JDBC connection is expensive, especially if you are securing it with SSL.
The other option is to use what is known as a connection pool, where the application doesn't create the JDBC connections directly, but acquires one from a pool of preciously opened connections. When it is done, it returns the connection to the pool.
Your database provider could provide a connection pool library, or you could use something like C3PO. This Q&A lists some other options: Java JDBC connection pool library choice in 2011/2012?
There is a gotcha with keeping connections open for a long time in MySQL. The problem is that MySQL has a default "idle connection" timeout of a few hours (10 or so I think). So if your application sits idle for long periods of time, it can find that its connection is broken. A connection pool may take care of reconnecting for you. Otherwise, the standard workaround for this problem is to increase the default timeout to something REALLY large. For example: MySQL connection timeout
Referring to Tomcat JBDC connection pool, I see in the standalone java example given there, one gets the connection using datasource.getConnection()which is cool. But in the finally block, it says con.close().
Question: When I implement this, it seems obvious that the con I get from datasource will be closed every time in the finally. When this is closed, will the connection pooling mechanism acquire a new connection and adds it to the pool?
I presume there should be a method call like releaseConnection() that will let the pool take its own decision whether to close it or let it be open for some other use.
I've also tried doing this ConnectionPool aPool = datasource.createPool();
But I see there is nothing like release connection on this aPool.
I think I'm missing something here?
Appreciate your help.
Code snippet from Tomcat JBDC connection pool:
DataSource datasource = new DataSource();
datasource.setPoolProperties(p);
Connection con = null;
try {
con = datasource.getConnection();
Statement st = con.createStatement();
ResultSet rs = st.executeQuery("select * from user");
int cnt = 1;
while (rs.next()) {
System.out.println((cnt++)+". Host:" +rs.getString("Host")+
" User:"+rs.getString("User")+" Password:"+rs.getString("Password"));
}
rs.close();
st.close();
} finally {
if (con!=null) try {con.close();}catch (Exception ignore) {}
}
Since you call the close() on a method obtained by the pool it is up to the pool what to do inside this method call. It does not neccessarily have to close the pooled database connection - it may do some cleanup and then add the connetion back to the pool.
This is already answered in Closing JDBC Connections in Pool
OK, my bad, that I did not see the implementation of DataSource.
It extends DataSourceProxy that internally creates a pool before returning a Connectionbased on the PoolProperties
I understand, its upto this DataSource to handle the connections, even though I close the con in finally, DataSource may take necessary action.
Do add a comment/reply if anybody thinks otherwise.
That example only shows how to create and use a data source. For connection pool on Tomcat you may configure JNDI.
// Sample
public static Connection getConnectionFromPool() {
Context initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");
DataSource ds = (DataSource) envCtx.lookup("jdbc/TestDB");
return ds.getConnection();
...
Quote from How connection pooling works in Java and JDBC:
A connection pool operates by performing the work of creating
connections ahead of time, In the case of a JDBC connection pool, a
pool of Connection objects is created at the time the application
server (or some other server) starts. These objects are then managed
by a pool manager that disperses connections as they are requested by
clients and returns them to the pool when it determines the client is
finished with the Connection object. A great deal of housekeeping is
involved in managing these connections.
When the connection pool server starts, it creates a predetermined
number of Connection objects. A client application would then perform
a JNDI lookup to retrieve a reference to a DataSource object that
implements the ConnectionPoolDataSource interface. The client
application would not need make any special provisions to use the
pooled data source; the code would be no different from code written
for a nonpooled DataSource.
I have a series of methods running within a servlet engine (Tomcat in this case), using connection pooling to access the database written in this way:
// Gets an RSS_Feed.
public static RSS_Feed get(int rssFeedNo) {
ConnectionPool_DB pool = ConnectionPool_DB.getInstance();
Connection connection = pool.getConnection();
PreparedStatement ps = null;
ResultSet rs = null;
String query = ("SELECT * " +
"FROM RSS_Feed " +
"WHERE RSSFeedNo = ?;");
try {
ps = connection.prepareStatement(query);
ps.setInt(1, rssFeedNo);
rs = ps.executeQuery();
if (rs.next()) {
return mapRSSFeed(rs);
}
else {
return null;
}
}
catch(Exception ex) {
logger.error("Error getting RSS_Feed " + rssFeedNo + "\n", ex);
return null;
}
finally {
Database_Utils.closeResultSet(rs);
Database_Utils.closeStatement(ps);
pool.freeConnection(connection);
}
}
Is it possible to call such a method outside of the servlet engine at all? I would like to do this in a batch process executed from the command line instead of within the servlet engine. I know I could simply rewrite the query without connection pooling but this is one of many queries involved in the process.
The connection pooling is implemented via Apache Common DBCP.
ConnectionPool_DB.getInstance(); reads:
private ConnectionPool_DB() {
try {
InitialContext ic = new InitialContext();
dataSource = (DataSource) ic.lookup(PropertiesFile.getProperty("myApp", "DATASOURCE"));
// dataSource = (DataSource) ic.lookup("java:/comp/env/jdbc/myApp");
}
catch(Exception ex) {
logger.error("Error getting a connection pool's datasource\n", ex);
}
}
I have something like this in a project:
Context ctx = new InitialContext();
DataSource ds = (DataSource) ctx.lookup("DbConnection");
ConnectionPool connectionPool = new ConnectionPool(ds)
And inside context xml I define the resouce like this
<Resource name="DbConnection"
auth="SERVLET"
type="javax.sql.DataSource"
scope="Shareable"
driverClassName="**driverClassName**"
url="**url**"
username="**username**"
password="**password**"
maxActive="10"
maxIdle="10"
maxWait="1000"
/>
So I assume you have something similar
If so you need to write code to create the DataSource yourself.
This should help you with that
http://docs.oracle.com/javase/tutorial/jdbc/basics/sqldatasources.html
Do you mean that you want to share a connection pool between your servlet engine and a batch job? Or that you want to use connection pooling within a batch job?
As to sharing a pool between Tomcat and a batch job: Hmm, I don't see how you'd do it. Tomcat and the batch job would each have their own instance of the Java Virtual Machine. They're not sharing memory, classes, etc, so I don't know where such a common pool would live.
If you mean within a batch job: Sure. I think such a thing is rarely necessary. In batch jobs I normally open a connection at the start of the program and close it at the end. There's not much value to creating a connection pool. Desktop apps are a little trickier. I often create a connection when the app starts and close it when they exit, but arguably this ties up a connection when the user is just staring blindly at the screen (like I often do for the hour or so before lunch), so other times I open a connection every time the user clicks a key that causes something to happen, then release it before going back to "wait" mode. Again, there's little point pooling because in a desktop app, there are no other users to share the pool with.
But can it be done? Sure. I've done it in desktop apps where many things could happen at various times and so it was awkward to pass a single connection around.
Sure, it could be used may be with slightly modification of the JNDI connectivity. But the Tomcat should run.
I'm using red5 1.0.0rc1 to create an online game.
I'm connecting to a MySQL database using a jdbc mysql connector v5.1.12
it seems that after several hours of idle my application can continue running queries because the connection to the db got closed and i have to restart the application.
how can I resolve the issue ?
Kfir
The MySQL JDBC driver has an autoreconnect feature that can be helpful on occasion; see "Driver/Datasource Class Names, URL Syntax and Configuration Properties for Connector/J"1, and read the caveats.
A second option is to use a JDBC connection pool.
A third option is to perform a query to test that your connection is still alive at the start of each transaction. If the connection is not alive, close it and open a new connection. A common query is SELECT 1. See also:
Cheapest way to to determine if a MySQL connection is still alive
A simple solution is to change the MySQL configuration properties to set the session idle timeout to a really large number. However:
This doesn't help if your application is liable to be idle for a really long time.
If your application (or some other application) is leaking connections, increasing the idle timeout could mean that lost connections stay open indefinitely ... which is not good for database memory utilization.
1 - If the link breaks (again), please Google for the quoted page title then edit the answer to update it with the new URL.
Well, you reopen the connection.
Connection pools (which are highly recommended, BTW, and if you run Java EE your container - Tomcat, JBoss, etc - can provide a javax.sql.DataSource through JNDI which can handle pooling and more for you) validate connections before handing them out by running a very simple validation query (like SELECT 1 or something). If the validation query doesn't work, it throws away the connection and opens a new one.
Increasing the connection or server timeout tends to just postpone the inevitable.
I had the Same issue for my application and I have removed the idle time out tag
Thats it
It really worked fine
try this, I was using the Jboss server, in that i have made the following change in mysql-ds.xml file.
Let me know if you have any more doubts
The normal JDBC idiom is to acquire and close the Connection (and also Statement and ResultSet) in the shortest possible scope, i.e. in the very same try-finally block of the method as you're executing the query. You should not hold the connection open all the time. The DB will timeout and reclaim it sooner or later. In MySQL it's by default after 8 hours.
To improve connecting performance you should really consider using a connection pool, like c3p0 (here's a developer guide). Note that even when using a connection pool, you still have to write proper JDBC code: acquire and close all the resources in the shortest possible scope. The connection pool will in turn worry about actually closing the connection or just releasing it back to pool for further reuse.
Here's a kickoff example how your method retrieving a list of entities from the DB should look like:
public List<Entity> list() throws SQLException {
// Declare resources.
Connection connection = null;
Statement statement = null;
ResultSet resultSet = null;
List<Entity> entities = new ArrayList<Entity>();
try {
// Acquire resources.
connection = database.getConnection();
statement = connection.createStatement("SELECT id, name, value FROM entity");
resultSet = statement.executeQuery();
// Gather data.
while (resultSet.next()) {
Entity entity = new Entity();
entity.setId(resultSet.getLong("id"));
entity.setName(resultSet.getString("name"));
entity.setValue(resultSet.getInteger("value"));
entities.add(entity);
}
} finally {
// Close resources in reversed order.
if (resultSet != null) try { resultSet.close(); } catch (SQLException logOrIgnore) {}
if (statement != null) try { statement.close(); } catch (SQLException logOrIgnore) {}
if (connection != null) try { connection.close(); } catch (SQLException logOrIgnore) {}
}
// Return data.
return entities;
}
See also:
DAO tutorial - How to write proper JDBC code
Do you have a validationQuery defined (like select 1)? If not, using a validation query would help.
You can check here for a similar issue.
Append '?autoReconnect=true' to the end of your database's JDBC URL (without the quotes) worked for me.
I saw that ?autoReconnect=true wasn't working for me.
What I did, is simply creating a function called: executeQuery with:
private ResultSet executeQuery(String sql, boolean retry) {
ResultSet resultSet = null;
try {
resultSet = getConnection().createStatement().executeQuery(sql);
} catch (Exception e) {
// disconnection or timeout error
if (retry && e instanceof CommunicationsException || e instanceof MySQLNonTransientConnectionException
|| (e instanceof SQLException && e.toString().contains("Could not retrieve transation read-only status server"))) {
// connect again
connect();
// recursive, retry=false to avoid infinite loop
return executeQuery(sql,false);
}else{
throw e;
}
}
return resultSet;
}
I know, I'm using string to get the error.. need to do it better.. but it's a good start, and WORKS :-)
This will almost all reasons from a disconnect.