I currently have a Database class, with PreparedStatement member variables that are initialized in the constructor. Something like this:
public class Database
{
private Connection connection;
private PreparedStatement statement1, statement2, ...;
public Database(String url, String user, String pass)
{
Class.forName("com.mysql.jdbc.Driver").newInstance();
connection = DriverManager.getConnection(url, user, pass);
statement1 = connection.prepareStatement("sql stuff");
statement2 = connection.prepareStatement("sql stuff");
// etc
}
public User getUser(int userId)
{
// execute getUser statement
}
// and other similar methods
}
The application will be multithreaded and I would like to use c3p0 for connection pooling. But I have no idea how to go about it.
Let's say I create a Database object for every thread, and the constructor now gets a connection from the pool. Each thread is supposed to only call one of the methods (with max 5 queries), then end. Will I have to initialize all the prepared statements every time? If yes, wouldn't it take too long?
Is there a better way I could do this?
You need to use the Datasource which provides the way to pool the connection and then you application get the connection from pool.
You can create the datasource programmatic like in this example at the startup of application or you can configure from webserver console (depending on webserver) and then get the datasource in your app through JNDI
Precompilation and DB-side caching of the Prepared Statement leads to overall faster execution and the ability to reuse the same SQL statement .
One of the advantages of a connection pool is that it re-uses existing connections, something which your current implementation does not do. So the question "wouldn't it take too long to initialize all the prepared statements each time?" is not really relevant since creating a new database connection each time will most likely take much longer than initializing the prepared statements each time. Even if the prepared statements are initialized each time and never re-used, I doubt you will notice any performance difference because executing the database statements takes a much longer time than initializing prepared statements.
That being said, most JDBC drivers will have an option to cache prepared statements (i.e. this is not strictly up to the connection pool). See for example the MySQL configuration options here (cachePrepStmts, prepStmtCacheSize and prepStmtCacheSqlLimit). But please keep in mind that these optimizations are "nice to have", first and foremost make sure your program works correctly in the multi-threaded scenario (e.g. ensure you always return a connection borrowed from the pool to the pool, even when (runtime) exceptions occur) and is maintainable.
Related
PreparedStatment ps = null;
public void executeQueries(){
try{
ps = conn.prepareStatement(Query1);
// Execute Query1 here and do the processing.
ps = conn.prepareStatement(Query2);
// Execute Query2 here and do the processing.
//... more queries
}catch(){}
finally{
ps.close(); // At this point would the caching of queries in DB be lost?
}
}
In my Application, I call the method executeQueries() frequently.
My question is, If I close the PreparedStatement in the finally block inside the method (that I use frequently), would the database system remove the caching? If YES, can I make a global PreparedStatement for the entire application as there are loads of JAVA CLASSES in my application that query the database.
Thank you!
Update : The question has been marked duplicate but the linked thread does not answer my question at all. AFAIK, the database system stores the executed queries in the cache memory. It also stores their execution plan. This is where PreparedStatement perfoms better than Statement. However, I am not very sure if the information related to the query is removed once the PreparedStatement is closed.
Specifically with regard to MySQL, according to
8.10.3 Caching of Prepared Statements and Stored Programs
The server maintains caches for prepared statements and stored programs on a per-session basis. Statements cached for one session are not accessible to other sessions. When a session ends, the server discards any statements cached for it.
So closing a PreparedStatement would not remove the statement(s) from the cache, but closing the Connection presumably would.
... unless the application uses a connection pool, in which case closing the Connection may not necessarily end the database session; it may keep the session open and just return the connection to the pool.
Then there's also the question of whether the statements are actually being PREPAREd on the server. That is controlled by the useServerPrepStmts connection string attribute. IIRC, by default, server-side prepared statements are not enabled.
I have a severe problem with my database connection in my web application. Since I use a single database connection for the whole application from singleton Database class, if i try concurrent db operations (two users) the database rollsback the transactions.
This is my static method used:
All threads/servlets call static Database.doSomething(...) methods, which in turn call the the below method.
private static /* synchronized*/ Connection getConnection(final boolean autoCommit) throws SQLException {
if (con == null) {
con = new MyRegistrationBean().getConnection();
}
con.setAutoCommit(true); //TODO
return con;
}
What's the recommended way to manage this db connection/s I have, so that I don't incurr in the same problem.
Keeping a Connection open forever is a very bad idea. It doesn't have an endless lifetime, your application may crash whenever the DB times out the connection and closes it. Best practice is to acquire and close Connection, Statement and ResultSet in the shortest possible scope to avoid resource leaks and potential application crashes caused by the leaks and timeouts.
Since connecting the DB is an expensive task, you should consider using a connection pool to improve connecting performance. A decent applicationserver/servletcontainer usually already provides a connection pool feature in flavor of a JNDI DataSource. Consult its documentation for details how to create it. In case of for example Tomcat you can find it here.
Even when using a connection pool, you still have to write proper JDBC code: acquire and close all the resources in the shortest possible scope. The connection pool will on its turn worry about actually closing the connection or just releasing it back to pool for further reuse.
You may get some more insights out of this article how to do the JDBC basics the proper way. As a completely different alternative, learn EJB and JPA. It will abstract away all the JDBC boilerplate for you into oneliners.
Hope this helps.
See also:
Is it safe to use a static java.sql.Connection instance in a multithreaded system?
Am I Using JDBC Connection Pooling?
How should I connect to JDBC database / datasource in a servlet based application?
When is it necessary or convenient to use Spring or EJB3 or all of them together?
I've not much experience with PostgreSql, but all the web applications I've worked on have used a single connection per set of actions on a page, closing it and disposing it when finished.
This allows the server to pool connections and stops problems such as the one that you are experiencing.
Singleton should be the JNDI pool connection itself; Database class with getConnection(), query methods et al should NOT be singleton, but can be static if you prefer.
In this way the pool exists indefinitely, available to all users, while query blocks use dataSource.getConnection() to draw a connection from the pool; exec the query, and then close statement, result set, and connection (to return it to the pool).
Also, JNDI lookup is quite expensive, so it makes sense to use a singleton in this case.
I'm creating a server-side Java task that executes the same SQL UPDATE every 60-seconds forever so it is ideal for using a java.sql.PreparedStatement.
I would rather re-connect to the database every 60-seconds than assume that a single connection will still be working months into the future. But if I have to re-generate a new PreparedStatement each time I open a new connection, it seems like it is defeating the purpose.
My question is: since the PreparedStatement is created from a java.sql.Connection does it mean that the connection must be maintained in order to use the PreparedStatement efficiently or is the PreparedStatement held in the database and not re-compiled with each new connection? I'm using postgresql at the present, but may not always.
I suppose I could keep the connection open and then re-open only when an exception occurs while attempting an update.
Use a database connection pool. This will maintain the connections alive in sleep mode even after closing them. This approach also saves performance for your application.
Despite the connection that created the PreparedStatement, the SQL statement will be cached by the database engine and there won't be any problems when recreating the PreparedStatement object.
Set your connection timeout to the SQL execution time+few minutes.
Now, you can take 2 different approaches here -
Check before executing the update, if false is returned then open new Connection
if( connection == null || !connection.isValid(0)) {
// open new connection and prepared statement
}
Write a stored procedure in the Db, and call it passing necessary params. This is an alternate approach.
Regarding you approach of closing and opening db connection every 60 seconds for the same prepared statement, it does not sound like a good idea.
I have been unable to find an exact answer to this question. I'm using C3P0's ComboPooledDataSource. Which of these methodologies is better practice:
dataSource = connectionClass.getDataSource();
conn = dataSource.getConnection;
executeQuery(query1, conn);
executeQuery(query2, conn);
...
executeQuery(finalQuery, conn);
conn.close();
OR
executeQuery(query1);
executeQuery(query2);
...
executeQuery(finalQuery);
where executeQuery:
conn = dataSource.getConnection;
st = conn.createStatement();
rs = executeQuery(query);
conn.closed();
In short, I have to do a decent amount of queries every so often. Is it better to go with the first design, which gets the connection once for each batch and passes it as an argument. Or is it better to go with the second approach and just get a connection each time I call my executeQuery method. If I was using DriverManager I would obviously choose the first (only get the connection once), but when using the C3P0 package I am not sure if doing that is the right way to go or not. Or does it not matter with such a package?
With a connection pool, the difference is neglectible, because even if you use the second approach, bringing back a pooled connection takes little time. Still, using the first approach is the better way to go, because
It avoids the additional (little) overhead of getting a connection from the pool.
If you later need to introduce transactions (do all of your changes or, in case of an error, conveniently and securely roll back your changes), then the first approach is your only option.
Some comments/suggestions
If you application is single threaded (unless you mention), it does not matter. It even does not matter whether you use connection pool or not. Just use a single connection and pass the same to function where you need it.
Connection pools are useful when the use case involves multiple database connections simultaneously.
Since your application is a batch and single threaded, it does not warrant use of connection pool.
Regarding your application, both the approaches are equivalent. When you call connection.close() on pooled datasource connection, its not actually closed but returned to pool.
I'm using c3p0. I set up a pooled as follows,
cpds = new ComboPooledDataSource();
cpds.setJdbcUrl(...);
/* connection setup */
spds.setMaxStatements(200);
I have an object that prepares several prepared statements on initialization. In order to do that, I grab a connection (con = getConnection()) from the PooledDataSource and then prepare a statement (e.g., PreparedStatement stmt = con.preparedStatemet(/*sql*/)). The prepared statements are stored as private variables in the object and the current connection is closed at the end of initialization (con.close()). The prepared statements are used in methods of the object.
For prepared statements that update the database, this works just fine. However, when I call a method that uses a prepared statement (stmt.executeQuery()) to query the database, I get the following SQLException
java.sql.SQLException: You can't operate on a closed Statement!!!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:127)
Did I get something wrong concerning the usage of c3p0?
Many thanks in advance!
Edit: Obviously, my question is partly based on my lack of understanding. As was pointed out in the definite answer, a PreparedStatement belongs to a connection and whenever the connection is closed, the associated statements should be closed, as well. But if that is the case, I don't understand what the use of c3p0's statement cache is.
you should get the same Exception calling executeUpdate(). JDBC Connection and Statement pooling is designed to be transparent: the same API that works for unpooled DataSources should be used for pooled versions too. There will be a dramatic difference in performance, but the code should be semantically interchangeable.
in an unpooled environment, it should be obvious why your approach fails: a Statement, prepared or otherwise, is a child of a Connection, without which it can't function. you are hoping that in the pooled environment, even though the Connection has been "closed", it should still exist in the pool, so hey, those Statements might be good. but that's a very bad idea (and if your attempts to do updates really are succeeding after the parent Connection has been close()ed, again, that'd be a bug, a bad one.) once a Connection has been "closed" it goes back in the pool, but not forever. other clients will check it out, and start performing transaction work that shouldn't be interrupted by your stale Statements. eventually Connections will be expired out of the pool. what should happened to your retained PreparedStatements then?
c3p0 pools Statements transparently, meaning you should use exactly the same API you would have used with no pooling. Call prepareStatement(...) on your Connection, every time. if you've enabled Statement pooling in c3p0 (as you have), then internally c3p0 will check to see whether the Statement has already been prepared, and if so it will quietly use the cached version rather than forwarding the request to the dbms.
i hope this helps!