I am working on an application where I need to connect N number of database systems [N ranges any where between 1 to 350].
The idea is - the user will be presented with a list of databases and will be asked to select any or all of the databases from the list.
Once the databases are selected, I need to connect to each of the database and execute a stored procedure.
I am planning to use plain old JDBC and obtain connection for each of them one a time [or by running them in multiple threads] and execute the store procedure and close the connection.
And all this should happen in a transaction. What is the best way to do this?
If not JDBC...any other efficient way?
Update -
The stored procedure is actually involved in running some sql - for example updating a column, grant permission for a user etc.
I'd create a threadpool with a reasonable maximum amount of threads, between ten and twenty threads maybe, with help of Executors#newFixedThreadPool() and invoke the separate DB connecting and SP executing tasks each as a Callable using ExecutorService#invokeAll(). You would like to play with the threadcount and profile which yields the best performance after all.
Each Callable implementation should take the connection details and SP name as constructor argument so that you can reuse the same implementation for different DB calls.
Update: OK, it's a webapplication. You don't want to waste threads. If it is supposed to be used by a single concurrent user, then you should really ensure that the threadpool is properly shutdown at end of request or at highest end of session. But if it is supposed to be used by multiple concurrent users, then you'd like to share the threadpool in the application scope. Also here, you need to ensure that it is properly shutdown when the webapp shuts down. The ServletContextListener is useful here.
If it is acceptable for you to use two connections, use connection pool c3p0 to manage them. To connect two databases I declare:
public Connection connection1;
public Connection connection2;
DataSource dataSource1;
DataSource dataSource2;
Then two similar methods:
public Connection dbConnect1() throws SQLException {
ComboPooledDataSource cpds = new ComboPooledDataSource();
try {
cpds.setDriverClass("com.mysql.jdbc.Driver");
} catch (PropertyVetoException e) {
}
cpds.setJdbcUrl("jdbc:mysql://localhost:3306/myDatabase1?autoReconnect=true");
cpds.setUser("myMYSQLServerLogin");
cpds.setPassword("myMYSQLServerPassword");
cpds.setMinPoolSize(5);
cpds.setAcquireIncrement(5);
cpds.setMaxPoolSize(20);
cpds.setMaxIdleTime(60);
cpds.setMaxStatements(100);
cpds.setPreferredTestQuery("SELECT 1");
cpds.setIdleConnectionTestPeriod(60);
dataSource1 = cpds;
connection1 = dataSource1.getConnection();
return connection1;
}
public Connection dbConnect2() throws SQLException {
ComboPooledDataSource cpds = new ComboPooledDataSource();
try {
cpds.setDriverClass("com.mysql.jdbc.Driver");
} catch (PropertyVetoException e) {
}
cpds.setJdbcUrl("jdbc:mysql://localhost:3306/myDatabase2?autoReconnect=true");
cpds.setUser("myMYSQLServerLogin");
cpds.setPassword("myMYSQLServerPassword");
cpds.setMinPoolSize(5);
cpds.setAcquireIncrement(5);
cpds.setMaxPoolSize(20);
cpds.setMaxIdleTime(60);
cpds.setMaxStatements(100);
cpds.setPreferredTestQuery("SELECT 1");
cpds.setIdleConnectionTestPeriod(60);
dataSource2 = cpds;
connection2 = dataSource2.getConnection();
return connection2;
}
As duffymo indicated in his comment, you will only be able to do transactions across multiple databases if you have a transaction coordinator and two phase commit.
For this you will need a J2EE stack that will handle JTA. If you are running in Tomcat or another container that does not have JTA, there are several options you can download and install.
Of course you will need to let the Container, not the database/stored procedure handle the transaction commits and rollbacks.
This sounds like a big mess, but it's your problem.
You need one connection pool per database. I wouldn't advise that you try to handle the connection lifecycle yourself. Let the app server do that for you.
If you want a group of databases to participate in one big transaction you'll have to use the JDBC XA drivers for all of them. You'll also need a JTA transaction manager to oversee the transaction for you.
The stored procedures cannot contain any logic to handle transactions; you have to let JTA do it.
You don't say what the stored procedure is doing. If it doesn't need to return anything, an alternative design might be JMS, a queue, and a listener pool. I'd be worried about threading if I were you. I'd find a way to let the container do that complicated stuff for meif I could.
public static Connection getconnection(String db,String host){
try {
Class.forName("com.mysql.jdbc.Driver");
Connection con = DriverManager.getConnection("jdbc:mysql://"+**Pass Your Host Here
Like Localhost**+"/"+Pass Your DB Name** +"?useUnicode=yes&characterEncoding=UTF-
8","root","root");
return con;
} catch (ClassNotFoundException | SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return null;
}
}
Related
In the JDBC driver for Postgres, is PGSimpleDataSource thread-safe?
That is, if I use a cached singleton instance of that class, can I pass it out to multiple threads? Each thread may be calling getConnection at the same moment. The documentation makes no mention of thread-safety.
I am trying to avoid both (a) making multi-threaded calls on a Connection and (b) using a connection pool, as discussed in the doc. I want a separate Connection for each servlet thread.
I'm assuming you won't be changing the data source configuration on multiple threads, because then it isn't thread-safe. You can inspect the source code yourself, on https://github.com/pgjdbc/pgjdbc, the specific code for getConnection is in BaseDataSource:
public Connection getConnection(String user, String password) throws SQLException {
try {
Connection con = DriverManager.getConnection(getUrl(), user, password);
if (LOGGER.isLoggable(Level.FINE)) {
LOGGER.log(Level.FINE, "Created a {0} for {1} at {2}", new Object[]{getDescription(), user, getUrl()});
}
return con;
} catch (SQLException e) {
LOGGER.log(Level.SEVERE, "Failed to create a {0} for {1} at {2}: {3}",
new Object[]{getDescription(), user, getUrl(), e});
throw e;
}
}
In other words, it is a thin wrapper around DriverManager. DriverManager itself is thread-safe, so then it becomes a question if org.postgresql.Driver is thread-safe. I don't have time to try to verify that, but lets just say it would be really surprising if that wasn't thread-safe (and otherwise world-wide applications would fail with all kinds of strange race-conditions, etc).
As a side note: PGSimpleDataSource does not provide connection pooling, you might want to consider whether that is right for your use case.
I have a bukkit plugin (minecraft) that requires a connection to the database.
Should a database connection stay open all the time, or be opened and closed when needed?
The database connection must be opened only when its needed and closed after doing all the necessary job with it. Code sample:
Prior to Java 7:
Connection con = null;
try {
con = ... //retrieve the database connection
//do your work...
} catch (SQLException e) {
//handle the exception
} finally {
try {
if (con != null) {
con.close();
}
} catch (SQLException shouldNotHandleMe) {
//...
}
}
Java 7:
try (Connection con = ...) {
} catch (SQLException e) {
}
//no need to call Connection#close since now Connection interface extends Autocloseable
But since manually opening a database connection is too expensive, it is highly recommended to use a database connection pool, represented in Java with DataSource interface. This will handle the physical database connections for you and when you close it (i.e. calling Connection#close), the physical database connection will just be in SLEEP mode and still be open.
Related Q/A:
Java Connection Pooling
Some tools to handle database connection pooling:
BoneCP
c3po
Apache Commons DBCP
HikariCP
Depends on what are your needs.
Creating a connection takes some time, so if you need to access database frequently it's better to keep the connection open. Also it's better to create a pool, so that many users can access database simultaneously(if it's needed).
If you need to use this connection only few times you may not keep it open, but you will have delay when you would like to access database. So i suggest you to make a timer that will keep connection open for some time(connection timeout).
You need to close your connections after each query executions.Sometimes you need to execute multiple queries at the same time because the queries are hanging from each other.Such as "first insert task then assign it to the employees".At this time execute your queries on the same transaction and commit it, if some errors occur then rollback.By default autocommit is disabled in JDBC. Example
Use connection pooling.If you are developing a webapplication then use App Server connection pooling.App server will use the same pooling for each of your applications so you can control the connection count from the one point.Highly recommend the Apache Tomcat Connection pooling.Example
As an additional info:
Connection, Statement and ResultSet.
1.If you close connection you don't need close statement or resultset.Both of them will be closed automatically
2.If you close Statement it will close ResultSet also
3.if you use try-with-resources like this:
try (Connection con = ...) {
} catch (SQLException e) {
}
it will close the connection automatically.Because try-with-resources require autoclosable objects and Connection is autocloseable.You can see the details about try-with-resources here
Actually, it's all matter on how you write your application! It's an art, but sadly everyone takes a tutorial for a good practice like Microsoft's tutorials.
If you know what you are coding, then you keep your connection open for the lifetime of the application. It's simple, not because you have to go at work in the morning that everyday we have to build a special route just for you! You take that single route or 2 or 4 like everyone does! You judge for the traffics and you build 2, 4 or 6 routes as needed. If there is traffic with these 4 or 6 routes, you wait!
Happy coding.
The Connection should be opened only when required. If it is open before the actual need, it reduces one active connection from the connection pool..so it ultimately effects the users of the application.
So,it is always a better practice to open connection only when required and closing it after completion of process.
Always try puttting you connection close logic inside the finally block that will ensure that your connection will be closed,even if any exception occurs in the application
finally
{
connection.close()
}
I'm using JBoss AS 7.1 as a server and I have my DataSource configured with pooling. I'm quite new to this so please excuse any rookie mistakes... after all I'm here to learn.
When a client logs-in it gets a connection to the database and I need to keep that connection(from the pool) open until the user logs-out or the HttpSession expires. This is an absolute requirement coming from our DB Admin. who says that he needs the DB session variables. I am using a servlet for all this.
Playing with the possibilities I have encountered 2 major problems:
As far as I see JBoss automatically closes unused connections => my opened connection returns to the pool. So this might not be the right path.
If I try to store/recall the Connection object like this:
private Hashtable<String, Connection> connections = new Hashtable<String, Connection>();
try {
String strDSName1 = "java:/OracleDSJNDI";
ctx = new InitialContext();
ds1 = (javax.sql.DataSource) ctx.lookup(strDSName1);
System.out.println("Got 1'st ds.");
} catch (Exception e) {
System.out.println("ERROR getting 1'st DS : " + e);
}
connection = ds1.getConnection();
connections.put(session.getId(), connection);
conn = (Connection) connections.get(sessionID);
it throws this exception:
java.sql.SQLException: Connection is not associated with a managed
connection.org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6#dee1f37
My question is: How do I properly keep my connection opened?
Thanks
How do I properly keep my connection opened?
You must not do that, let the connection pool handle this.
Behind the scenes, the connection pool will keep a bunch of database connections to the database engine (MySQL, Oracle, SQL Server... depends how you configure it) in SLEEPING state. When you execute this code:
//avoiding all the particular exceptions just for code simplicity purposes...
//in real world applications, you must handle each of these exceptions
public Connection getConnection() throws Exception {
ctx = new InitialContext();
ds1 = (javax.sql.DataSource) ctx.lookup(strDSName1);
return ds1.getConnection();
}
You're asking to the connection pool to retrieve one of these connections available. The connection pool will give you a database connection (if available) and let you use it as long as you want. Then you use it wherever you want/need and close it:
public void foo() throws Exception {
Connection connection = getConnection();
//do what you want/need...
//in the end, you close the connection
//this is A MUST!
connection.close();
}
When executing connection.close() from a connection retrieved by the connection pool, you're not closing the physical database connection but notifying the connection pool this specific database connection must return to the SLEEPING state.
Some advices from the explanation:
You must not try to keep the connection alive, that's connection pool's job.
You must not try to store the connections in any cache-like structure, that's connection pool's job.
You must retrieve a java.sql.Connection in the shortest scope you will need it. Once you have used it, close it.
Your DBA is basically requiring you to avoid connection pooling by making the database connection equivalent to the user's session.
So one option is to not use the connection pool, and instead roll your own functionality that opens/closes the database connection around the user's session. That seems complicated and unusual though.
Another option is to examine the DBA's requirement. The DBA may have to adapt to the idea that he'll need to track state in a different way, e.g. by using a key related to the session to store the state he needs in a table, instead of storing state in the connection layer.
Generally speaking storing state in some component's session handling is adding indirect complexity, because you start having to care about how the component handles expiry and uniqueness, as you're finding here where the HTTP session state handles this differently from the database session.
I've built an application (app1) that looks at and records certain fields in a database. This application shares the database with another application (app2) that requires a solitary connection to it when starting up but is fine to allow other connections to the DB once it (app2) is started. In my application (app1), I have made a dao object using Spring to connect to the DB and evidently, the connection is never closed which causes app2 to crash upon start up. From what I've read, Spring is supposed to automatically handle opening and closing all of the DB connections it manages. I'm not sure of any code I could share to help paint a better picture of my problem but if some is needed, I'll post what I can. Thanks for any help.
If you're using the JDBC template you don't have to worry about explicitly closing connections, Spring will take care of internally managing a connection pool and obtaining/releasing connections from that pool.
When sharing connections between applications I would recommend using a connection pool. Opening and closing of connections could be done by declarative transaction demarcation via annotations (#Transactional).
http://faheemsohail.com/2012/01/configuring-c3p0-connection-pooling-with-spring-and-hibernate/
Would this work for you?
public void closeCon() {
{
if (con != null)
try {
con.close();
} catch (SQLException e) {
e.printStackTrace();
}
What is best/good/optimal way to monitor connection to database.
Im writing swing application. What I want it to do is to check connection with database every time period. I've tried something like this.
org.hibernate.Session session = null;
try {
System.out.println("Check seesion!");
session = HibernateUtil.getSessionFactory().openSession();
} catch (HibernateException ex) {
} finally {
session.close();
}
But that dosn't work.
Second question which is comming to my mind is how this session closing will affect other queries.
Use a connection pool like c3p0 or dbcp. You can configure such pool to monitor connections in pool - before passing connection to Hibernate, after receiving it back or periodically. If the connection is broken, the pool with transparently close it, discard and open a new one - without you noticing.
Database connection pools are better suited for multi-user, database heavy application where several connections are opened at the same time, but I don't think it's an overkill. Pools should work just fine being bound to max 1 connection.
UPDATE: every time you try to access the database Hibernate will ask the DataSource (connection pool). If no active connection is available (e.g. because database is down), this will throw an exception. If you want to know in advance when database is unavailable (even when user is not doing anything), unfortunately you need a background thread checking the database once in a while.
However barely opening a session might not be enough in some configurations. You'll better run some dummy, cheap query like SELECT 1 (in raw JDBC).