I have a class which looks like that:
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
public class ConnectionPool {
private HikariDataSource hds;
private final String propertyFileName;
public ConnectionPool(String propertyFileName) {
if (propertyFileName == null) {
throw new IllegalArgumentException("propertyFileName can't be null");
}
this.propertyFileName = propertyFileName;
reloadFile();
}
public void reloadFile() {
if (hds != null) {
hds.close();
}
hds = new HikariDataSource(new HikariConfig(propertyFileName));
}
public HikariDataSource getHikariDataSource() {
return hds;
}
public String getPropertyFileName() {
return propertyFileName;
}
public void executeQuery(final String sql, final CallBack<ResultSet, SQLException> callBack) {
new Thread(new Runnable() {
#Override
public void run() {
Connection connection = null;
PreparedStatement preparedStatement = null;
ResultSet resultSet = null;
try {
connection = hds.getConnection();
preparedStatement = connection.prepareStatement(sql);
resultSet = preparedStatement.executeQuery();
callBack.call(resultSet, null);
} catch (SQLException e) {
callBack.call(null, e);
} finally {
if (resultSet != null) {
try {
resultSet.close();
} catch (SQLException ignored) {}
}
if (preparedStatement != null) {
try {
preparedStatement.close();
} catch (SQLException ignored) {}
}
if (connection != null) {
try {
connection.close();
} catch (SQLException ignored) {}
}
}
}
}).start();
}
public void executeUpdate(final String sql, final CallBack<Integer, SQLException> callBack) {
//TODO
}
public void execute(final String sql, final CallBack<Boolean, SQLException> callBack) {
//TODO
}
public void connection(final String sql, final CallBack<Connection, SQLException> callBack) {
//TODO
}
}
The problem is that the reloadFile() method can be called from a different thread as hds is used. So it's possible that hds is closed while I use a connection object of it in another thread. What's the best way to solve this problem? Should I wait a few seconds after creating the new HikariDataSource object befor closing the old one (until the queries are finished)?
Edit: Another question: Should hds be volatile, so that the changes of hds are visible for all threads?
Have had a very very quick and brief look in the source code in HikariDataSource. In its close(), it is calling its internal HikariPool's shutdown() method, for which it will try to properly close the pooled connections.
If you want to even avoid any chance that in-progress connection from force closing, one way is to make use of a ReadWriteLock:
public class ConnectionPool {
private HikariDataSource hds;
private ReentrantReadWriteLock dsLock = ....;
//....
public void reloadFile() {
dsLock.writeLock().lock();
try {
if (hds != null) {
hds.close();
}
hds = new HikariDataSource(new HikariConfig(propertyFileName));
} finally {
dsLock.writeLock().unlock();
}
}
public void executeQuery(final String sql, final CallBack<ResultSet, SQLException> callBack) {
new Thread(new Runnable() {
#Override
public void run() {
Connection connection = null;
PreparedStatement preparedStatement = null;
ResultSet resultSet = null;
dsLock.readLock().lock();
try {
connection = hds.getConnection();
// ....
} catch (SQLException e) {
callBack.call(null, e);
} finally {
// your other cleanups
dsLock.readLock().unlock();
}
}
}).start();
}
//....
}
This will make sure that
multiple thread can access your datasource (to get connection etc)
Reload of datasource needs to wait until thread using the datasource to complete
No thread is able to use the datasource to get connection when it is reloading.
Why exactly are you trying to cause HikariCP to reload? Many of the important pool parameters (minimumIdle,maximumPoolSize,connectionTimeout,etc.) are controllable at runtime through the JMX bean without restarting the pool.
Restarting the pool is a good way to "hang" your application for several seconds while connections are closed and rebuilt. If you can't do what you need through the JMX interface, Adrian's suggestion seems like quite a reasonable solution.
Other solutions are possible, but have more complexity.
EDIT: Just for my own entertainment, here is the more complex solution...
public class ConnectionPool {
private AtomicReference<HikariDataSource> hds;
public ConnectionPool(String propertyFileName) {
hds = new AtomicReference<>();
...
}
public void reloadFile() {
final HikariDataSource ds = hds.getAndSet(new HikariDataSource(new HikariConfig(propertyFileName)));
if (ds != null) {
new Thread(new Runnable() {
public void run() {
ObjectName poolName = new ObjectName("com.zaxxer.hikari:type=Pool (" + ds.getPoolName() + ")");
MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
HikariPoolMXBean poolProxy = JMX.newMXBeanProxy(mBeanServer, poolName, HikariPoolMXBean.class);
poolProxy.softEvictConnections();
do {
Thread.sleep(500);
} while (poolProxy.getActiveConnections() > 0);
ds.close();
}
}).start();
}
}
public HikariDataSource getHikariDataSource() {
return hds.get();
}
public void executeQuery(final String sql, final CallBack<ResultSet, SQLException> callBack) {
new Thread(new Runnable() {
#Override
public void run() {
...
try {
connection = getHikariDataSource().getConnection();
...
}
}
}).start();
}
}
This will swap out the pool (atomically) and will start a thread that waits until all active connections have returned before shutting down the orphaned pool instance.
This assumes that you let HikariCP generate unique pool names, i.e. do not set poolName in your properties, and that registerMbeans=true.
A few options:
Synchronize all access to the data source so that only one thread can ever be messing with it. Not scaleable, but workable.
Roll your own connection pooling, such as Apache Commons Pooling so that each access, regardless of thread, requests a data source and pooling creates one as necessary. Can mess with data ACID, just depends on whether dirty data is needed, when data is flushed, transactionality, etc.
Each thread could also have its own data source using ThreadLocal so that each thread is totally independent of each other. Again, quality of data might be an issue, resources might be an issue if you've got "lots" of threads (depends on your definition) and too many open connections cause resource issues on either the client or server.
Related
I am using following code for registering and listening to Oracle database change notifications. This code is working fine when i run it as a standalone java program. It is receiving the notification from the database and printing as expected.
public class DBChangeNotification {
static final String USERNAME = "XXX";
static final String PASSWORD = "YYY";
static String URL = "jdbc:oracle:thin:#xxxx:xxxx:xxxx";
public static void main(String[] args) {
DBChangeNotification demo = new DBChangeNotification();
try {
demo.run();
} catch (SQLException mainSQLException) {
mainSQLException.printStackTrace();
}
}
public void run() throws SQLException {
OracleConnection conn = connect();
Properties prop = new Properties();
prop.setProperty(OracleConnection.DCN_NOTIFY_ROWIDS, "true");
prop.setProperty(OracleConnection.DCN_QUERY_CHANGE_NOTIFICATION, "true");
prop.setProperty(OracleConnection.DCN_BEST_EFFORT, "true");
DatabaseChangeRegistration dcr = conn.registerDatabaseChangeNotification(prop);
try {
// add the listenerr:
DCNDemoListener list = new DCNDemoListener(this);
dcr.addListener(list);
// second step: add objects in the registration:
Statement stmt = conn.createStatement();
// associate the statement with the registration:
((OracleStatement) stmt).setDatabaseChangeRegistration(dcr);
ResultSet rs = stmt.executeQuery("select * from xxxxxxxx where yyyy='zzzzz'");
while (rs.next()) {
}
String[] tableNames = dcr.getTables();
for (int i = 0; i < tableNames.length; i++) {
System.out.println(tableNames[i] + " is part of the registration.");
}
rs.close();
stmt.close();
} catch (SQLException ex) {
// if an exception occurs, we need to close the registration in order
// to interrupt the thread otherwise it will be hanging around.
if (conn != null) {
conn.unregisterDatabaseChangeNotification(dcr);
}
ex.printStackTrace();
throw ex;
} finally {
try {
// Note that we close the connection!
conn.close();
} catch (Exception innerex) {
innerex.printStackTrace();
}
}
}
/**
* Creates a connection the database.
*/
OracleConnection connect() throws SQLException {
OracleDriver dr = new OracleDriver();
Properties prop = new Properties();
prop.setProperty("user", DBChangeNotification.USERNAME);
prop.setProperty("password", DBChangeNotification.PASSWORD);
return (OracleConnection) dr.connect(DBChangeNotification.URL, prop);
}
}
/**
* DCN listener: it prints out the event details in stdout.
*/
class DCNDemoListener implements DatabaseChangeListener {
DBChangeNotification demo;
DCNDemoListener(DBChangeNotification dem) {
System.out.println("DCNDemoListener");
demo = dem;
}
#Override
public void onDatabaseChangeNotification(DatabaseChangeEvent e) {
Thread t = Thread.currentThread();
System.out.println("DCNDemoListener: got an event (" + this + " running on thread " + t + ")");
System.out.println(e.toString());
synchronized (demo) {
demo.notify();
}
}
}
My requirement is to use this feature in a web application. Web application when started in the server, has to listen to data change notifications (may be on a separate thread) and notify the application through a websocket client. I have added the following code in contextInitialized method of servlet context listener, so that it will start as soon as the application starts.
public class MyServletContextListener implements ServletContextListener {
DBChangeNotification demo;
#Override
public void contextDestroyed(ServletContextEvent arg0) {
//Notification that the servlet context is about to be shut down.
}
#Override
public void contextInitialized(ServletContextEvent arg0) {
demo = new DBChangeNotification();
try {
demo.run();
} catch (SQLException mainSQLException) {
mainSQLException.printStackTrace();
}
}
}
I did not see any notifications received by the web application when database change event occurs in the registered table. Please help me in resolving the issue. I do not know whether this is a correct approach or not.... may please suggest any alternative except continuous polling. I need to start something in the server as soon as i receive notification from database. Thank you.
It might be that you're running your code on an Oracle instance that doesn't have the Notification API available.
Check this SO for more info
I have created a database object according to the singleton pattern. The database object contains 2 methods: connect() and update().
The update should run multithreaded, meaning that I cannot put synchronized in the update method signature (I want users to access it simultaneously not one at a time).
My problem is that I want to make sure that 2 scenarios according to this flows:
A thread 1 (user1) is the first to create an instance of the DB and thread 2 (user2) is calling the connect() and update() method to this DB - should not give NullPointerException even if by the time that user2 is doing the update() the connect from user1 is not done.
update() should not include synchronized (because of the reason I mentioned above).
Thanks for all the helpers!
SingeltonDB
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
public class SingeltonDB {
private static DBconnImpl db = null;
private static SingeltonDB singalDb = null;
Lock dbLock;
private SingeltonDB(String username, String password) {
db = new DBconnImpl();
}
public static boolean isOpen() {
return (db != null);
}
public synchronized static SingeltonDB getInstance(String username,
String password) throws Exception {
if (db != null) {
throw (new Exception("The database is open"));
} else {
System.out.println("The database is now open");
singalDb = new SingeltonDB(username, password);
}
db.connect(username, password);
System.out.println("The database was connected");
return singalDb;
}
public synchronized static SingeltonDB getInstance() throws Exception {
if (db == null) {
throw (new Exception("The database is not open"));
}
return singalDb;
}
public void create(String tableName) throws Exception {
dbLock = new ReentrantLock();
dbLock.lock();
db.create(tableName);
dbLock.unlock();
}
public User query(String tableName, int rowID) throws Exception {
if (db == null) {
System.out.println("Error: the database is not open");
return null;
}
return (db.query(tableName, rowID));
}
public void update(String tableName, User user) throws Exception {
if (db == null) {
System.out.println("Error: the database is not open");
return;
}
db.update(tableName, user);
}
}
Main
public class Main {
public static void main(String[] args) throws Exception {
Creator cr= new Creator(new UserContorller());
Thread t1 = new Thread(cr);
t1.start();
Producer pr = new Producer(new UserContorller());
Thread t2 = new Thread(pr);
t2.start();
/*
* Consumer cn = new Consumer(new UserContorller()); Thread t2 = new
* Thread(cn); t2.start();
*/
}
}
class Creator implements Runnable {
UserContorller uc;
public Creator(UserContorller uc) {
this.uc = uc;
}
#Override
public void run() {
try {
uc = new UserContorller("MyAccount", "123");
uc.createTable("table1");
} catch (Exception e) {
e.printStackTrace();
}
}
}
class Producer implements Runnable {
UserContorller uc;
public Producer(UserContorller uc) {
this.uc = uc;
}
#Override
public void run() {
for (int i = 0; i < 100; i++) {
try {
uc.saveUser("table1", i, "User", i);
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
class Consumer implements Runnable {
UserContorller uc;
public Consumer(UserContorller uc) {
this.uc = uc;
}
#Override
public void run() {
for (int i = 0; i < 100; i++) {
try {
System.out.println(uc.getUser("table1", i));
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
Note: The post below was written in the perspective of both users using the same credentials (hidden from them) to connect to the database. If users employ different credentials, the idea of a singleton db object is purposeless, each user should have their own connection object, and of course then connection details are passed on from the user to the Db via whatever represents the user in the program (here the thread instances apparently).
The main issue in the implementation you provided is that the getinstance method requires its caller to know the connection details, or assume that the connection has already been done. But neither threads could nor should know in advance if the Db has been opened already -- and design wise it's a mistake to hand them the responsibility of explicitely opening it. These threads are work threads, they shouldn't be concerned about Db configuration details.
The only sane way to handle this situation is to have these configuration parameters held by the Db object directly, or better yet another object in charge of providing it (it's the factory pattern).
However, if you want first your code to work with minimal changes, get rid of the parameter less getinstance method, have any thread requiring the Db object to use the remaining variant of that method, passing along the correct parameters, and change it to return the instance if it exists, or create it otherwise, without raising an exception. I believe that it's what #Dima has been trying to explain in his answer.
Connect once, when creating the singleton (in the constructor, perhaps).
Have a synchronized static method (getInstance or something), that checks if an instance exists, creates and connects as necessary, and returns the instance. By following this protocol, you ensure that threads always get a connected Db object ready to be used.
The users will call that method to get the singleton instance, and call update or whatever they want on it, it does not need to be synchronized.
Disclaimer- I am not a Java programmer. Odds are I'll need to do my homework on any advice given, but I will gladly do so :)
That said, I wrote a complete database-backed socket server which is working just fine for my small tests, and now I'm getting ready for initial release. Since I do not know Java/Netty/BoneCP well- I have no idea if I made a fundamental mistake somewhere that will hurt my server before it even gets out the door.
For example, I have no idea what an executor group does exactly and what number I should use. Whether it's okay to implement BoneCP as a singleton, is it really necessary to have all those try/catch's for each database query? etc.
I've tried to reduce my entire server to a basic example which operates the same way as the real thing (I stripped this all in text, did not test in java itself, so excuse any syntax errors due to that).
The basic idea is that clients can connect, exchange messages with the server, disconnect other clients, and stay connected indefinitely until they choose or are forced to disconnect. (the client will send ping messages every minute to keep the connection alive)
The only major difference, besides untesting this example, is how the clientID is set (safely assume it is truly unique per connected client) and that there is some more business logic in checking of values etc.
Bottom line- can anything be done to improve this so it can handle the most concurrent users possible? Thanks!
//MAIN
public class MainServer {
public static void main(String[] args) {
EdgeController edgeController = new EdgeController();
edgeController.connect();
}
}
//EdgeController
public class EdgeController {
public void connect() throws Exception {
ServerBootstrap b = new ServerBootstrap();
ChannelFuture f;
try {
b.group(new NioEventLoopGroup(), new NioEventLoopGroup())
.channel(NioServerSocketChannel.class)
.localAddress(9100)
.childOption(ChannelOption.TCP_NODELAY, true)
.childOption(ChannelOption.SO_KEEPALIVE, true)
.childHandler(new EdgeInitializer(new DefaultEventExecutorGroup(10)));
// Start the server.
f = b.bind().sync();
// Wait until the server socket is closed.
f.channel().closeFuture().sync();
} finally { //Not quite sure how to get here yet... but no matter
// Shut down all event loops to terminate all threads.
b.shutdown();
}
}
}
//EdgeInitializer
public class EdgeInitializer extends ChannelInitializer<SocketChannel> {
private EventExecutorGroup executorGroup;
public EdgeInitializer(EventExecutorGroup _executorGroup) {
executorGroup = _executorGroup;
}
#Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast("idleStateHandler", new IdleStateHandler(200,0,0));
pipeline.addLast("idleStateEventHandler", new EdgeIdleHandler());
pipeline.addLast("framer", new DelimiterBasedFrameDecoder(8192, Delimiters.nulDelimiter()));
pipeline.addLast("decoder", new StringDecoder(CharsetUtil.UTF_8));
pipeline.addLast("encoder", new StringEncoder(CharsetUtil.UTF_8));
pipeline.addLast(this.executorGroup, "handler", new EdgeHandler());
}
}
//EdgeIdleHandler
public class EdgeIdleHandler extends ChannelHandlerAdapter {
private static final Logger logger = Logger.getLogger( EdgeIdleHandler.class.getName());
#Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception{
if(evt instanceof IdleStateEvent) {
ctx.close();
}
}
private void trace(String msg) {
logger.log(Level.INFO, msg);
}
}
//DBController
public enum DBController {
INSTANCE;
private BoneCP connectionPool = null;
private BoneCPConfig connectionPoolConfig = null;
public boolean setupPool() {
boolean ret = true;
try {
Class.forName("com.mysql.jdbc.Driver");
connectionPoolConfig = new BoneCPConfig();
connectionPoolConfig.setJdbcUrl("jdbc:mysql://" + DB_HOST + ":" + DB_PORT + "/" + DB_NAME);
connectionPoolConfig.setUsername(DB_USER);
connectionPoolConfig.setPassword(DB_PASS);
try {
connectionPool = new BoneCP(connectionPoolConfig);
} catch(SQLException ex) {
ret = false;
}
} catch(ClassNotFoundException ex) {
ret = false;
}
return(ret);
}
public Connection getConnection() {
Connection ret;
try {
ret = connectionPool.getConnection();
} catch(SQLException ex) {
ret = null;
}
return(ret);
}
}
//EdgeHandler
public class EdgeHandler extends ChannelInboundMessageHandlerAdapter<String> {
private final Charset CHARSET_UTF8 = Charset.forName("UTF-8");
private long clientID;
static final ChannelGroup channels = new DefaultChannelGroup();
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
Connection dbConnection = null;
Statement statement = null;
ResultSet resultSet = null;
String query;
Boolean okToPlay = false;
//Check if status for ID #1 is true
try {
query = "SELECT `Status` FROM `ServerTable` WHERE `ID` = 1";
dbConnection = DBController.INSTANCE.getConnection();
statement = dbConnection.createStatement();
resultSet = statement.executeQuery(query);
if (resultSet.first()) {
if (resultSet.getInt("Status") > 0) {
okToPlay = true;
}
}
} catch (SQLException ex) {
okToPlay = false;
} finally {
if (resultSet != null) {
try {
resultSet.close();
} catch (SQLException logOrIgnore) {
}
}
if (statement != null) {
try {
statement.close();
} catch (SQLException logOrIgnore) {
}
}
if (dbConnection != null) {
try {
dbConnection.close();
} catch (SQLException logOrIgnore) {
}
}
}
if (okToPlay) {
//clientID = setClientID();
sendCommand(ctx, "HELLO", "WORLD");
} else {
sendErrorAndClose(ctx, "CLOSED");
}
}
#Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
channels.remove(ctx.channel());
}
#Override
public void messageReceived(ChannelHandlerContext ctx, String request) throws Exception {
// Generate and write a response.
String[] segments_whitespace;
String command, command_args;
if (request.length() > 0) {
segments_whitespace = request.split("\\s+");
if (segments_whitespace.length > 1) {
command = segments_whitespace[0];
command_args = segments_whitespace[1];
if (command.length() > 0 && command_args.length() > 0) {
switch (command) {
case "HOWDY": processHowdy(ctx, command_args); break;
default: break;
}
}
}
}
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
TraceUtils.severe("Unexpected exception from downstream - " + cause.toString());
ctx.close();
}
/* */
/* STATES - / CLIENT SETUP */
/* */
private void processHowdy(ChannelHandlerContext ctx, String howdyTo) {
Connection dbConnection = null;
Statement statement = null;
ResultSet resultSet = null;
String replyBack = null;
try {
dbConnection = DBController.INSTANCE.getConnection();
statement = dbConnection.createStatement();
resultSet = statement.executeQuery("SELECT `to` FROM `ServerTable` WHERE `To`='" + howdyTo + "'");
if (resultSet.first()) {
replyBack = "you!";
}
} catch (SQLException ex) {
} finally {
if (resultSet != null) {
try {
resultSet.close();
} catch (SQLException logOrIgnore) {
}
}
if (statement != null) {
try {
statement.close();
} catch (SQLException logOrIgnore) {
}
}
if (dbConnection != null) {
try {
dbConnection.close();
} catch (SQLException logOrIgnore) {
}
}
}
if (replyBack != null) {
sendCommand(ctx, "HOWDY", replyBack);
} else {
sendErrorAndClose(ctx, "ERROR");
}
}
private boolean closePeer(ChannelHandlerContext ctx, long peerClientID) {
boolean success = false;
ChannelFuture future;
for (Channel c : channels) {
if (c != ctx.channel()) {
if (c.pipeline().get(EdgeHandler.class).receiveClose(c, peerClientID)) {
success = true;
break;
}
}
}
return (success);
}
public boolean receiveClose(Channel thisChannel, long remoteClientID) {
ChannelFuture future;
boolean didclose = false;
long thisClientID = (clientID == null ? 0 : clientID);
if (remoteClientID == thisClientID) {
future = thisChannel.write("CLOSED BY PEER" + '\n');
future.addListener(ChannelFutureListener.CLOSE);
didclose = true;
}
return (didclose);
}
private ChannelFuture sendCommand(ChannelHandlerContext ctx, String cmd, String outgoingCommandArgs) {
return (ctx.write(cmd + " " + outgoingCommandArgs + '\n'));
}
private ChannelFuture sendErrorAndClose(ChannelHandlerContext ctx, String error_args) {
ChannelFuture future = sendCommand(ctx, "ERROR", error_args);
future.addListener(ChannelFutureListener.CLOSE);
return (future);
}
}
When a network message arrive at server, it will be decoded and will release a messageReceived event.
If you look at your pipeline, last added thing to pipeline is executor. Because of that executor will receive what has been decoded and will release the messageReceived event.
Executors are processor of events, server will tell which events happening through them. So how executors are being used is an important subject. If there is only one executor and because of that, all clients using this same executor, there will be a queue for usage of this same executor.
When there are many executors, processing time of events will decrease, because there will not be any waiting for free executors.
In your code
new DefaultEventExecutorGroup(10)
means this ServerBootstrap will use only 10 executors at all its lifetime.
While initializing new channels, same executor group being used:
pipeline.addLast(this.executorGroup, "handler", new EdgeHandler());
So each new client channel will use same executor group (10 executor threads).
That is efficient and enough if 10 threads are able to process incoming events properly. But if we can see messages are being decoded/encoded but not handled as events quickly, that means there is need to increase amount of them.
We can increase number of executors from 10 to 100 like that:
new DefaultEventExecutorGroup(100)
So that will process event queue faster if there is enough CPU power.
What should not be done is creating new executor for each new channel:
pipeline.addLast(new DefaultEventExecutorGroup(10), "handler", new EdgeHandler());
Above line is creating a new executor group for each new channel, that will slow down things greatly, for example if there are 3000 clients, there will be 3000 executorgroups(threads). That is removing main advantage of NIO, ability to use with low thread amounts.
Instead of creating 1 executor for each channel, we can create 3000 executors at startup and at least they will not be deleted and created each time a client connects/disconnects.
.childHandler(new EdgeInitializer(new DefaultEventExecutorGroup(3000)));
Above line is more acceptable than creating 1 executor for each client, because all clients are connected to same ExecutorGroup and when a client disconnects Executors still there even if client data is removed.
If we must speak about database requests, some database queries can take long time to being completed, so if there are 10 executorss and there are 10 jobs being processed, 11th job will have to wait until one of others complete. This is a bottleneck if server receiving more than 10 very time consuming database job at the same time. Increasing count of executors will solve bottleneck to some degree.
can you look at my connection pool if it is a possible way to implement it?
public class ConnectionPool {
private static List<DBConnection> pool = null;
private static int available = 0;
private ConnectionPool() {}
public static DBConnection getConnection() {
if (pool == null) {
pool = new ArrayList<DBConnection>();
for (int i = 0; i < 3; i++) {
try {
pool.add(new DBConnection());
available++;
} catch (SQLException e) {
e.printStackTrace();
}
}
}
if (pool.size() > 0) {
available--;
return pool.remove(available);
} else {
return null;
}
}
public static void returnConnection(DBConnection c) {
pool.add(c);
available++;
}
}
I'm using only one array and the client should ask the connection pool for a connection use it and then return it to the connection pool.
Connection connection = ConnectionPool.getConnection();
connection.execute("insert into users values('"+user.getUsername()+"', '"+user.getPassword()+"', '"+user.getGroup()+"', '"+banned+"', '"+broker+admin+sharehodler+company+"')");
ConnectionPool.returnConnection(connection);
connection = null;
Please I need feedback on this implementation. Thank you
There are some points that make this implementation very problematic.
Thread safety. What if several threads work with the pool? You are not locking the list on read/write access.
Static maximum pool size of 3 connections, also you immediately create all of them, whether they are needed or not. A common strategy is to create a bunch of connections and create some more when needed, until the allowed/configured maximum is reached.
You only have static methods. It should be possible to have several pools, meaning you need instances of ConnectionPool.
No way to pass host+dbname+user+password to the connections that are created.
You don't deal with 'broken' connections - you may have to re-create a connection if an existing one screwed up. This is far more relevant than I thought before I started using pools.
Use config values instead of static values, see point #2
Lastly: sure, it's interesting to write this stuff yourself - but if you need a pool for a project, pick an existing one, such as c3p0 or the tomcat connection pool.
I'm sure there's more to point out, but unless these are fixed, there's no use in continuing.
One big problem with your pool implementation is that you pass the naked connection to the callers of the pool. This means that someone can obtain a connection from your pool, close it, and then return it to the pool. This is bad.
The normal way around this problem is to wrap the return connection objects using delegation, and make them ignore calls to the close method (or even better, make close() safely return the underlying connection to the pool).
Other major issues:
What happens if a connection is returned in the middle of a transaction?
What happens if a connection is somehow corrupted or disconnected? Does it stay in the pool?
All in all, you should reuse an existing connection pool implementation rather than writing your own. These days, many type 4 drivers have their own connection pools included right within the driver.
Some thoughts:
Your code is NOT thread safe. Maybe work on this.
Too much code in getConnection(). Is lazy initialisation really
needed ?
available is useless, can be substitute by pool.size().
AFAIK,
your getConnection() method need to be changed to in order to retrieve Connection object only.
Preparing connection and pooling should be taken away from the getConnection() method and added in such a way that when ConnectionPool class is loaded first time.
Also you need to handle some other attributes as well like connection timeout, purging etc in order to make it work for all scenarios.
Make it thread safe as well.
Other members have already suggested lots of things. I have some model implementation, thought of sharing it for new visitors. Here is code:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Iterator;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
abstract class ObjectPool<T> {
private ConcurrentLinkedQueue<T> pool;
ScheduledExecutorService executorService;
ObjectPool(int minObjects) {
pool = new ConcurrentLinkedQueue<T>();
for (int i = 0; i < minObjects; i++) {
pool.add(createObject());
}
}
ObjectPool(final int minObjects, final int maxSize, final long interval){
pool = new ConcurrentLinkedQueue<T>();
for (int i = 0; i < minObjects; i++) {
pool.add(createObject());
}
executorService = Executors.newSingleThreadScheduledExecutor();
executorService.scheduleWithFixedDelay(new Runnable(){
public void run() {
int size = pool.size();
while(size > maxSize){
pool.remove();
}
Iterator i = pool.iterator();
while(i.hasNext()){
T t = (T) i.next();
if(checkExpiry(t)){
System.out.println("Expiry existed...");
i.remove();
}
}
while(pool.size() < minObjects){
System.out.println("Adding more objects to pool");
pool.add(createObject());
}
}
}, interval, interval, TimeUnit.MILLISECONDS);
}
public T borrowObject() {
if (pool.peek() == null)
return createObject();
return pool.remove();
}
public void addObject(T obj) {
if (obj == null)
return;
pool.add(obj);
}
public abstract T createObject();
public abstract boolean checkExpiry(T t);
}
class MultithreadQuery extends Thread{
private ObjectPool<Connection> pool;
private int threadNo;
String query;
MultithreadQuery(ObjectPool<Connection> pool,int threadNo, String query){
this.pool = pool;
this.threadNo = threadNo;
this.query = query;
}
#Override
public void run(){
Connection con = pool.borrowObject();
Statement stmt;
try {
stmt = con.createStatement();
System.out.println("Query started for thread->"+ threadNo);
ResultSet rs=stmt.executeQuery(query);
while(rs.next())
System.out.println(rs.getInt(1)+" "+rs.getString(2)+" "+rs.getString(3));
System.out.println("closing connection....");
con.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
pool.addObject(con);
System.out.println("Query ended for thread->"+ threadNo);
}
}
public class ObjectPoolPatternDemo {
ObjectPool<Connection> pool;
public void setUp(){
pool = new ObjectPool<Connection>(4, 10, 1) {
#Override
public Connection createObject() {
Connection con;
try {
con = DriverManager.getConnection("URL","Username","Password");
return con;
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
#Override
public boolean checkExpiry(Connection conn) {
boolean expiryFlag = false;
try {
if(conn.isClosed())
expiryFlag = true;
} catch (SQLException e) {
e.printStackTrace();
}
return expiryFlag;
}
};
}
public static void main(String[] args) throws SQLException {
ObjectPoolPatternDemo oppd = new ObjectPoolPatternDemo();
oppd.setUp();
ExecutorService es = Executors.newFixedThreadPool(4);
String query = "select * from TABLE";
es.execute(new MultithreadQuery(oppd.pool,1,query));
es.execute(new MultithreadQuery(oppd.pool,2,query));
es.execute(new MultithreadQuery(oppd.pool,3,query));
es.execute(new MultithreadQuery(oppd.pool,4,query));
es.execute(new MultithreadQuery(oppd.pool,5,query));
es.execute(new MultithreadQuery(oppd.pool,6,query));
es.shutdown();
try {
es.awaitTermination(1, TimeUnit.DAYS);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("finally completed...");
}
}
I am writing a webservice that allows users to post files and then retrieve them at a URL (basically think of it as the RESTful Amazon S3). The issue I came across was rather then return a byte[] from my Oracle query (Spring JDBC) I am returning an InputStream and then streaming the data back to the client in chunks. This (IMO) is a much better idea since I put no size restriction on the file and I don't want 2GB byte arrays in memory.
At first it seemed to work fine, but I ran into a case during heavy load that sometimes a Connection would get reused before the previous servlet could send the file. It seems after the JDBC call that returned the InputStream, the Connection would be returned to the pool (Spring would call conn.close(), but not clear the associated ResultSet). So if no other request was given that Connection then the InputStream would still be valid and could be read from, but if the Connection was given to a new request then the InputStream would be null and the previous request would fail.
My solution was to create a subclass of InputStream that also takes a Connection as a constructor arg, and in the overridden public close() method also close the Connection. I had to ditch the Spring JDBC and just make a normal PreparedStatement call, otherwise Spring would always return the connection to the pool.
public class ConnectionInputStream extends InputStream {
private Connection conn;
private InputStream stream;
public ConnectionInputStream(InputStream s, Connection c) {
conn = c;
stream = s;
}
// all InputStream methods call the same method on the variable stream
#Override
public void close() throws IOException {
try {
stream.close();
} catch (IOException ioex) {
//do something
} finally {
try {
conn.close();
} catch (SQLException sqlex) {
//ignore
}
}
}
}
Does anyone have a more elegant solution, or see any glaring problems with my solution? Also this code wasn't cut/paste from my actual code so if there is a typo just ignore it.
Unfortunately, my imagination went wild when you asked this question. I don't know if this solution is considered more elegant. However, these classes are simple and easily re-usable so you may find a use for them if they are not satisfactory. You will see everything coming together at the end...
public class BinaryCloseable implements Closeable {
private Closeable first;
private Closeable last;
public BinaryCloseable(Closeable first, Closeable last) {
this.first = first;
this.last = last;
}
#Override
public void close() throws IOException {
try {
first.close();
} finally {
last.close();
}
}
}
BinaryCloseable is used by CompositeCloseable:
public class CompositeCloseable implements Closeable {
private Closeable target;
public CompositeCloseable(Closeable... closeables) {
target = new Closeable() { public void close(){} };
for (Closeable closeable : closeables) {
target = new BinaryCloseable(target, closeable);
}
}
#Override
public void close() throws IOException {
target.close();
}
}
The ResultSetCloser closes ResultSet objects:
public class ResultSetCloser implements Closeable {
private ResultSet resultSet;
public ResultSetCloser(ResultSet resultSet) {
this.resultSet = resultSet;
}
#Override
public void close() throws IOException {
try {
resultSet.close();
} catch (SQLException e) {
throw new IOException("Exception encountered while closing result set", e);
}
}
}
The PreparedStatementCloser closes PreparedStatement objects:
public class PreparedStatementCloser implements Closeable {
private PreparedStatement preparedStatement;
public PreparedStatementCloser(PreparedStatement preparedStatement) {
this.preparedStatement = preparedStatement;
}
#Override
public void close() throws IOException {
try {
preparedStatement.close();
} catch (SQLException e) {
throw new IOException("Exception encountered while closing prepared statement", e);
}
}
}
The ConnectionCloser closes Connection objects:
public class ConnectionCloser implements Closeable {
private Connection connection;
public ConnectionCloser(Connection connection) {
this.connection = connection;
}
#Override
public void close() throws IOException {
try {
connection.close();
} catch (SQLException e) {
throw new IOException("Exception encountered while closing connection", e);
}
}
}
We now refactor your original InputStream idea into:
public class ClosingInputStream extends InputStream {
private InputStream stream;
private Closeable closer;
public ClosingInputStream(InputStream stream, Closeable closer) {
this.stream = stream;
this.closer = closer;
}
// The other InputStream methods...
#Override
public void close() throws IOException {
closer.close();
}
}
Finally, it all comes together as:
new ClosingInputStream(
stream,
new CompositeCloseable(
stream,
new ResultSetCloser(resultSet),
new PreparedStatementCloser(statement),
new ConnectionCloser(connection)
)
);
When this ClosingInputStream's close() method is called, this is effectively what happens (with exception handling omitted for clarity's sake):
public void close() {
try {
try {
try {
try {
// This is empty due to the first line in `CompositeCloseable`'s constructor
} finally {
stream.close();
}
} finally {
resultSet.close();
}
} finally {
preparedStatement.close();
}
} finally {
connection.close();
}
}
You're now free to close as many Closeable objects as you like.
Why not read the entire InputStream/byte[]/whatever from the query before releasing the query yourself? It sounds like you are trying to return data from the query after your code has told Spring / the pool that you are done with the connection.
An alternative approach is to use a callback. Below is kind of the idea.
class MyDao
{
public boolean getData(Function<InputStream, Boolean> processData) {
// Do your SQL stuff to get a ResultSet
InputStream input = resultSet.getBinaryStream(0);
processData.apply(input);
// Do your cleanup if any
}
}