What are best practices for closing or caching sql connections?
I see things saying to always close it in finally block.
Though I also see things talking about caching them for future use.
How expensive is it to open new connections for each task?
I just started to work on a java data warehousing app someone else programmed. (and I don't have DB experience) It is a single threaded app that loads files into the DB, and does some aggregation and analysis. We have had some issues with deadlocks, which I wouldn't think should be a problem in a single threaded app. I see exceptions are swallowed and not logged all over the class that handles the DB connections, so I am hoping adding logging to those should provide some insight.
But I am hoping for some guidance on best practices for dealing with DB connections in the meantime.
Regardless of whether or not you are using connection pooling, any database code should follow this form:
try (
Connection connection = dataSource.getConnection();
PreparedStatement preparedStatement = ...
)
{
...
}
The dataSource could be a pool or a simple connection factory, to your code it shouldn't matter, it's simply where it gets connections from.
This said, I'd use Spring's JdbcTemplate class, which does pretty much what the code above does.
Regarding the question 'how expensive is it to open new connections for each task?' - the answer is very, at least in comparison to grabbing one from a pool. You also have to consider what happens if large numbers of tasks are run - most databases will be configured to limit the number of connections.
The bottom line is, in anything but the most simple application, you should use a connection pool like c3po and size it according you your needs
Regarding your deadlock, this is most likely occurring in the database where there are a variety of locks that can be made when data is updated depending on how the database is configured.
Related
I have a severe problem with my database connection in my web application. Since I use a single database connection for the whole application from singleton Database class, if i try concurrent db operations (two users) the database rollsback the transactions.
This is my static method used:
All threads/servlets call static Database.doSomething(...) methods, which in turn call the the below method.
private static /* synchronized*/ Connection getConnection(final boolean autoCommit) throws SQLException {
if (con == null) {
con = new MyRegistrationBean().getConnection();
}
con.setAutoCommit(true); //TODO
return con;
}
What's the recommended way to manage this db connection/s I have, so that I don't incurr in the same problem.
Keeping a Connection open forever is a very bad idea. It doesn't have an endless lifetime, your application may crash whenever the DB times out the connection and closes it. Best practice is to acquire and close Connection, Statement and ResultSet in the shortest possible scope to avoid resource leaks and potential application crashes caused by the leaks and timeouts.
Since connecting the DB is an expensive task, you should consider using a connection pool to improve connecting performance. A decent applicationserver/servletcontainer usually already provides a connection pool feature in flavor of a JNDI DataSource. Consult its documentation for details how to create it. In case of for example Tomcat you can find it here.
Even when using a connection pool, you still have to write proper JDBC code: acquire and close all the resources in the shortest possible scope. The connection pool will on its turn worry about actually closing the connection or just releasing it back to pool for further reuse.
You may get some more insights out of this article how to do the JDBC basics the proper way. As a completely different alternative, learn EJB and JPA. It will abstract away all the JDBC boilerplate for you into oneliners.
Hope this helps.
See also:
Is it safe to use a static java.sql.Connection instance in a multithreaded system?
Am I Using JDBC Connection Pooling?
How should I connect to JDBC database / datasource in a servlet based application?
When is it necessary or convenient to use Spring or EJB3 or all of them together?
I've not much experience with PostgreSql, but all the web applications I've worked on have used a single connection per set of actions on a page, closing it and disposing it when finished.
This allows the server to pool connections and stops problems such as the one that you are experiencing.
Singleton should be the JNDI pool connection itself; Database class with getConnection(), query methods et al should NOT be singleton, but can be static if you prefer.
In this way the pool exists indefinitely, available to all users, while query blocks use dataSource.getConnection() to draw a connection from the pool; exec the query, and then close statement, result set, and connection (to return it to the pool).
Also, JNDI lookup is quite expensive, so it makes sense to use a singleton in this case.
I've been researching all around the web the most efficient way to design a connection pool and tried to analyze into details the available libraries (HikariCP, BoneCP, etc.).
Our application is a heavy-load consumer webapp and most of the time the users are working on similar business objects (thus the underlying SQL queries executed are the often the same, but still there are numerous).
It is designed to work with different DBMS (Oracle and MS SQL Server especially).
So a simplified use case would be :
User goes on a particular JSP page (e.g. Enterprise).
A corresponding Bean is created.
Each time it realizes an action (e.g. getEmployees(), computeTurnover()), the Bean asks the pool for a connection and returns it back when done.
If we want to take advantage of the Prepared Statement caching of the underlying JDBC driver (as PStatements are attached to a connection - jTDS doc.), from what I understand an optimal way of doing it would be :
Analyze what kind of SQL query a particular Bean want to execute before providing it an available connection from the pool.
Find a connection where the same prepared statement has already been executed if possible.
Serve the connection accordingly (and use the benefits of the cache/precompiled statement).
Return the connection to the pool and start over.
Am I missing an important point here (like JDBC drivers capable of reusing cached statements regardless of the connection) or is my analysis correct ?
The different sources I found state it is not possible, but why ?
For your scheme to work, you'd need to be able to get the connection that already has that statement prepared.
This falls foul on two points:
In JDBC you obtain the connection first,
Cached prepared statements (if a driver or connection pool even supports that) aren't exposed in a standardized way (if at all) nor would you be able to introspect them.
The performance overhead of finding the right connection (and the subsequent contention on the few connections that already have it prepared) would probably undo any benefit of reusing the prepared statement.
Also note that some database systems also have a serverside cache for prepared statements (meaning that it already has the plan etc available), limiting the overhead from a new prepare from the client.
If you really think the performance benefit is big enough, you should consider using a data source specific for this functionality (so it is almost guaranteed that the connection will have the statement in its cache).
A solution could be for a connection pool implementation to delay retrieving the connection from the pool until the Connection.prepareStatement() is called. At that time a connection pool would look up available connections by the SQL statement text and then play forward all the calls made before Connection.prepareStatement(). This way it would be possible to get a connection with a ready PreparedStatement without the issues other guys suggested.
In other words, when you request a connection from the pool, it would return a wrapper that logs everything until the first operation requiring DB access (such as prepareStatement() is requested.
You'd need to ask a vendor of your connection pool functionality to add this feature.
I've logged this request with C3P0:
https://github.com/swaldman/c3p0/issues/55
Hope this helps.
I have gone through couple of articles on singleton example. And I could see that developers sometimes make the database connection object or connection manager as singleton implementation. In some of the posts even it was advised to use Database connection pool.
Well Singleton means to create a single instance, so basically we restrict the access. e.g. Printer or hardware access, logger access in which we try to restrict the access of the user to one at a time using singleton. However what is the purpose of using singleton in DB connection objects?
If I can understand correctly creating a Database connection as singleton means that app server will have only one instance. Does this mean only one user can access the Database connection and next user has to wait until the connection is closed?
Please advise.
I think you understand correctly the implication of making the connection itself a singleton. Generally it is not a good idea (although in some very particular case it could make sense).
Making a connection manager or a connection pool a singleton is completely different. The pool itself would handle a collection of connections and it can create new as they are needed (up to a limit) or re-use the ones that have been already used and discarded.
Having several connection pools at the same time would lose the advantages of the pool:
It would be harder to control the total number of connections open
One pool could be creating connections while other could have connections available
Hope this helps to clarify the subject. You might want to read more on connection pools.
Q: "However what is the purpose of using singleton in DB connection objects?" A: There is (almost always) none. So your thinking is correct.
Q: "Does this mean only one user can access the Database connection and next user has to wait until the connection is closed?"
A: Depends (to first part) and No (to second part after "and"). In single-threaded application only one user will use the database at one time and another will wait, until dispatch of first user ends but not when the connection is closed. Once connection is closed, you need to create another connection to make use of database. In multi-threaded application many threads may be using the same connection instance and result really depends on the vendor implementation: may block dispatching (effectively transforming your app to single-threaded app) or throw exceptions or even something different. However, such design in multi-threaded app is in my opinion a programmer-error.
What is the fastest option to issue stored procedures in a threaded environment in Java? According to http://dev.mysql.com/doc/refman/5.1/en/connector-j-usagenotes-basic.html#connector-j-examples-preparecall Connection.prepareCall() is an expensive method. So what's the alternative to calling it in every thread, when synchronized access to a single CallableStatement is not an option?
The most JDBC drivers use only a single socket per connection. I think MySQL also use also a single socket. That it is a bad performance idea to share one connection between multiple threads.
If you use multiple connection between different threads then you need a CallableStatment for every connection. You need a CallabaleStatement pool for every connection. The simplest to pool it in this case is to wrap the connection class and delegate all calls to the original class. This can be create very fast with Eclipse. In the wrapped method prepareCall() you can add a simple pool. You need also a wrapped class of the CallableStatement. The close method return the CallableStatement to the pool.
But first you should check if the call is real expensive because many driver has already such poll inside. Create a loop of prepareCall() and close() and count the time.
Connection is not thread safe, so you can't share it across threads.
When you prepareCall, the JDBC driver (may) be telling the RDBMS system to do a lot of work that is stored on the server side. You may be guilty of premature optimization here.
After giving this a little thought it seems that if you are having issues with this infrastructure code then your problems are elsewhere. Most applications do not take an inordinate amount of time doing this stuff.
Make sure you are using a DataSource, most do connection caching and some even do caching of statements.
Also for this to be a performance bottle neck it would imply that you are doing many queries one after the other, or that your pool of connections is too small. Maybe you should do some benchmarking on your code to see how much time the stored proc is taking vs how much time the JDBC code is taking.
Of course I would follow the MySQL recommendation of using CallableStatement, I am sure they have benchmarked this. Most apps do not share anything between Threads and it is rarely an issue.
Encountering the following error with our J2EE application:
java.sql.SQLException: Error in allocating a connection. Cause: In-use connections equal max-pool-size and expired max-wait-time. Cannot allocate more connections.
How do I know how many connections the application is currently using and what should be the optimal connection pool settings for a heavily traffic application? Can I change it, and how can I determine what I should set it to (is it a memory issue, bandwidth, etc.)?
How to know how much connection the
application is currently using
You don't give enough information to answer that. Most appservers will have some sort of JMX reporting on things like this. Alternatively, depending on the database, you could find the number of currently open connections.
what should be the optimal connection
pool settings for a heavily traffic
applicaiton
Higher than what you've got?
The above of course assumes that you're not mishandling connections. By that I mean if you're using them directly in code you should always use this idiom:
Connection conn = null;
try {
conn = ... ; // get connection
// do stuff
} finally {
if (conn != null) try { conn.close(); } catch (Exception e) { }
}
If you're not released connections back to the pool and are waiting for the garbage collector to clean them up and release them you're going to use way more connections that you actually need.
First thing to check is whether you have resource leaks. Try surveilling with jconsole or jvisualvm to see how your application behaves and if there is anything that springs in the eye.
Then the actual jdbc pool connection is inherently Java EE container specific so you need to give more information.
Are you sure you are closing everything you need to? Take a look here.
Make sure that the close goes in the finally block. You will see this code in the link:
finally
{
closeAll(resultSet, statement, connection);
}
I helped someone else find a similar issue (theirs was a memory issue instead... but if the process had gone on longer it would have had this result) where they had not closed the result set or the connection.
I think you may need to inspect ask yourself some questions, here are some to think about:
A. Are you using youre connections correctly, are you closing them and returning them to the pool after usage?
B. Are you using long running transactions, e.g. "conversations" with users? Can they be left hanging if the user terminates his usage of the application?
C. Have you designed your data access to fit your application? E.g. are you using caching techniques in areas where you expect repeated reads frequently?
D. Are your connection pool big enough? 5 years ago I had a application with 250 simultaneous connections towards a Oracle database, a lot more than what you typically find out of the box. running the application on say 50 didn't work.
To give more detailed answer, you need to provide more info on the app.
Good luck!