I've been using several connection pooling api's before but I thought its time to do something on my own. Iam currently searching for a very basic MySQL Connection Pooling function on which I can build on. I really would appreciate if you guys could give me a honest opinion!
Here's the code:
package temp.jdbc;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;
public class MySQL {
private static final String driver = "com.mysql.jdbc.Driver";
private static final String url = "jdbc:mysql://localhost:3306/beta";
private static final String username = "root";
private static final String password = "mypassword";
private static List<Connection> pool = new ArrayList<Connection>();
public static void setup() {
try {
for(int i = 0; i < 5; i++) {
Class.forName(driver);
Connection connection = DriverManager.getConnection(url, username, password);
pool.add(connection);
}
}
catch (ClassNotFoundException e) { e.printStackTrace(); }
catch (SQLException e) { e.printStackTrace(); }
}
public static void addConnection(Connection connection) {
pool.add(connection);
}
public static Connection getConnection() {
Connection connection = null;
if(pool.size() <= 0) return connection;
connection = pool.get(0); pool.remove(0);
return connection;
}
}
Thanks in advance!
Sincerely,
Gemaken
There's some major issues with your implementation of a connection pool - connections aren't returned to the pool, no thread-safety, no blocking when all connections are being used, it's static, etc.
Given this, I'll start with the basics.
First, the whole point of any pool is to hold an expensive resource, in this case a javax.sql.Connection, so that it may be reused in the future avoiding the cost of creating that resource again.
With most pools there's a min and max - when created the pool will create the min number of resources and can expand upto the max as the resources are consumed.
When a resource is retrieved from a pool, the retrieving thread should be guaranteed exclusive use of that resource, until the resource is returned to the pool.
Should all resources be being used, any further attempts to obtain a resource should block (with some sort of timeout) until a resource is available.
The use of a pool implies that the application is multi-threaded, so it is also important that the implementation is thread-safe.
As the JDBC API is well defined connection pools typically implement javax.sql.DataSource allowing connection pools to be switched in and out as needed. Given this javax.sql.Connection.close() should be used to return the connection to the pool.
Related
I tried to find the connection pool which allows to get connection by jdbc url but failed. Hikari connection pool doesn't allow do it, the same situation in c3po.
My use case is:
ConnectionPool.getConnection(jdbcUrl);
Does anybody know such connection pool in java world?
A Simple Guide to Connection Pooling in Java
Author - Baeldung
About Author
A Simple Implementation
To better understand the underlying logic of connection pooling, let's
create a simple implementation.
Let's start out with a loosely-coupled design, based on just one
single interface:
public interface ConnectionPool {
Connection getConnection();
boolean releaseConnection(Connection connection);
String getUrl();
String getUser();
String getPassword();
}
The ConnectionPool interface defines the public API of a basic
connection pool.
Now, let's create an implementation, which provides some basic
functionality, including getting and releasing a pooled connection:
public class BasicConnectionPool
implements ConnectionPool {
private String url;
private String user;
private String password;
private List<Connection> connectionPool;
private List<Connection> usedConnections = new ArrayList<>();
private static int INITIAL_POOL_SIZE = 10;
public static BasicConnectionPool create(
String url, String user,
String password) throws SQLException {
List<Connection> pool = new ArrayList<>(INITIAL_POOL_SIZE);
for (int i = 0; i < INITIAL_POOL_SIZE; i++) {
pool.add(createConnection(url, user, password));
}
return new BasicConnectionPool(url, user, password, pool);
}
// standard constructors
#Override
public Connection getConnection() {
Connection connection = connectionPool
.remove(connectionPool.size() - 1);
usedConnections.add(connection);
return connection;
}
#Override
public boolean releaseConnection(Connection connection) {
connectionPool.add(connection);
return usedConnections.remove(connection);
}
private static Connection createConnection(
String url, String user, String password)
throws SQLException {
return DriverManager.getConnection(url, user, password);
}
public int getSize() {
return connectionPool.size() + usedConnections.size();
}
// standard getters
}
Connection pooling is a well-known data access pattern, whose main
purpose is to reduce the overhead involved in performing database
connections and read/write database operations.
In a nutshell, a connection pool is, at the most basic level, a
database connection cache implementation, which can be configured to
suit specific requirements.
In this tutorial, we'll make a quick roundup of a few popular
connection pooling frameworks, and we'll learn how to implement from
scratch our own connection pool.
Why Connection Pooling?
The question is rhetorical, of course.
If we analyze the sequence of steps involved in a typical database
connection life cycle, we'll understand why:
Opening a connection to the database using the database driver Opening
a TCP socket for reading/writing data Reading / writing data over the
socket Closing the connection Closing the socket It becomes evident
that database connections are fairly expensive operations, and as
such, should be reduced to a minimum in every possible use case (in
edge cases, just avoided).
Here's where connection pooling implementations come into play.
By just simply implementing a database connection container, which
allows us to reuse a number of existing connections, we can
effectively save the cost of performing a huge number of expensive
database trips, hence boosting the overall performance of our
database-driven applications.
JDBC Connection Pooling Frameworks
From a pragmatic perspective, implementing a connection pool from the
ground up is just pointless, considering the number of
“enterprise-ready” connection pooling frameworks available out there.
From a didactic one, which is the goal of this article, it's not.
Even so, before we learn how to implement a basic connection pool,
let's first showcase a few popular connection pooling frameworks.
Apache Commons DBCP
public class DBCPDataSource {
private static BasicDataSource ds = new BasicDataSource();
static {
ds.setUrl("jdbc:h2:mem:test");
ds.setUsername("user");
ds.setPassword("password");
ds.setMinIdle(5);
ds.setMaxIdle(10);
ds.setMaxOpenPreparedStatements(100);
}
public static Connection getConnection() throws SQLException {
return ds.getConnection();
}
private DBCPDataSource(){ }
}
In this case, we've used a wrapper class with a static block to easily
configure DBCP's properties.
Here's how to get a pooled connection with the DBCPDataSource class:
connection con = DBCPDataSource.getConnection();
HikariCP
public class HikariCPDataSource {
private static HikariConfig config = new HikariConfig();
private static HikariDataSource ds;
static {
config.setJdbcUrl("jdbc:h2:mem:test");
config.setUsername("user");
config.setPassword("password");
config.addDataSourceProperty("cachePrepStmts", "true");
config.addDataSourceProperty("prepStmtCacheSize", "250");
config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048");
ds = new HikariDataSource(config);
}
public static Connection getConnection() throws SQLException {
return ds.getConnection();
}
private HikariCPDataSource(){}
}
Similarly, here's how to get a pooled connection with the
HikariCPDataSource class:
Connection con = HikariCPDataSource.getConnection();
I need to use a connection pool in a standalone (as in non-web) Java application. Where I work, we are not allowed to use APIs without going through layers of security, and the job needs to be completed soon. Below is my attempt at creating this connection pool.
I have unit tested this code and tested it within the context of the overall application a hundred times and in all cases the tests passed with zero errors, and in addition the performance of each run is just under three thousand times faster than a simple connect, retrieve data, disconnect in serial approach; however, I still have nagging concerns that there could be issues with this approach that I simply haven't unearthed yet. I would appreciate any advice anyone has concerning the below code. This is my first post on this site; please let me know if I've made any errors in etiquette. I did search this site about this problem before posting. Please see below the code for an invocation example. Thanks. --JR
package mypackage;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.Map;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.TimeUnit;
/**
* Note: This class is only instantiated once per application run.
* Multiple instantiations, as specified in the release notes,
* are not supported.
*/
public class ConnectionManager {
// Use a blocking queue to store the database connections.
// The application will only be called once, by a single user,
// but within the application many threads will require
// a connection.
private BlockingQueue<Connection> connectionQueue = null;
// Load the connection queue with a user-defined number of connections.
// Params contains a map of all non hard-coded variables in the
// application.
public ConnectionManager(int howMany, Map<String, Object> params) {
Database database = new Database();
connectionQueue = new ArrayBlockingQueue<Connection>(howMany);
for(int i = 0; i < howMany; i++) {
connectionQueue.add(database.getConn(params));
}
}
// Return a connection from the queue, waiting up to 15 minutes to do so.
// 15 minutes is hard-coded because it is the standard time-out for all
// processes at our agency. This application must complete in less
// than fifteen minutes (is currently completing in thirty five seconds).
public Connection getConnection() {
Connection conn = null;
try {
conn = connectionQueue.poll(15, TimeUnit.MINUTES);
}
catch(InterruptedException e) {
e.printStackTrace();
}
catch(SQLException e) {
e.printStackTrace();
}
return conn;
}
// Returns a connection to the connection queue.
public void returnConnectionToManager(Connection conn) {
connectionQueue.add(conn);
}
// Called on the last line of the application program's dispatcher.
// Closes all active connections (which will only exist if there
// was a failure within one of the worker threads).
public void closeAllConnections() {
for(Connection conn : connectionQueue) {
try {
conn.close();
}
catch(SQLException e) {
e.printStackTrace();
}
}
}
}
Invocation example:
...
private ConnectionManager cm;
...
public Table(Map<String, Object> params, String method) {
...
cm = (ConnectionManager) params.get("cm");
}
// Execute a chunk of SQL code without requiring processing of a
// result set. Acquires connection from pool via cm.getConnection
// and releases connection via cm.returnConnectionToManager.
// (Database is just a helper class with simple methods for
// closing prepared statement, result sets, etc.)
private void execute(String sql) {
PreparedStatement ps = null;
Connection conn = null;
try {
conn = cm.getConnection();
ps = conn.prepareStatement(sql);
ps.execute();
}
catch (SQLException e) {
e.printStackTrace();
}
finally {
database.closePreparedStatement(ps);
cm.returnConnectionToManager(conn);
}
}
Your code looks good, but there is one serious problem, that clients of your API needs to take care of getting and releasing connection, one of them forget, and memory/resource leak is ready.
Make a one place in which you posts your queries to execute, in this place take connection, execute query and return the connection to the pool. It will secure you that the connections are returned. If you need to invoke multiple queries one after another in a single connection make the method accept an array or list of SQL queries to execute in order. The idea is to encapsulate each request to the db, so you manage all connections. It could be donethat you write an interface that has en execute(Connection conn) which you need to implement, and you could have then some Service that takes such object gives it a connection and then releases the resources back to connection pool.
Something like:
interface SqlWork {
execute(Connection conn);
}
SqlWork myWork = new SqlWork () {
execute(Connection conn) {
// do you work with the conn here
}
}
class SqlExecutionService {
ConnectionManager cm = ...;
public void execute(SqlWork sqlWork) {
Connection conn = null;
try {
conn = cm.getConnection();
sqlWork.execute(conn);
} catch (Your exceptions here) {
//serve or rethrow them
}
finally
{
if (conn!=null) {
cm.returnConnectionToManager(conn);
}
}
}
}
Example of use:
SqlExecutionService sqlExecService = ...;
sqlExecService.execute(myWork);
I am new in connection pooling.I have a created a connection pool in mysql that adds five connections.Now i want to know what is the application of connection pooling,i.e after creating that pool how to use that.. i am pasting my code here
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Vector;
import com.mysql.jdbc.Connection;
class ConnectionPoolManager {
String databaseUrl = "jdbc:mysql://localhost:3306/homeland";
String userName = "root";
String password = "root";
Vector connectionPool = new Vector();
public ConnectionPoolManager() {
initialize();
}
public ConnectionPoolManager(
// String databaseName,
String databaseUrl, String userName, String password) {
this.databaseUrl = databaseUrl;
this.userName = userName;
this.password = password;
initialize();
}
private void initialize() {
// Here we can initialize all the information that we need
initializeConnectionPool();
}
private void initializeConnectionPool() {
while (!checkIfConnectionPoolIsFull()) {
System.out
.println("Connection Pool is NOT full. Proceeding with adding new connections");
// Adding new connection instance until the pool is full
connectionPool.addElement(createNewConnectionForPool());
}
System.out.println("Connection Pool is full.");
}
private synchronized boolean checkIfConnectionPoolIsFull() {
final int MAX_POOL_SIZE = 5;
// Check if the pool size
if (connectionPool.size() < 5) {
return false;
}
return true;
}
// Creating a connection
private Connection createNewConnectionForPool() {
Connection connection = null;
try {
Class.forName("com.mysql.jdbc.Driver");
connection = (Connection) DriverManager.getConnection(databaseUrl,
userName, password);
System.out.println("Connection: " + connection);
} catch (SQLException sqle) {
System.err.println("SQLException: " + sqle);
return null;
} catch (ClassNotFoundException cnfe) {
System.err.println("ClassNotFoundException: " + cnfe);
return null;
}
return connection;
}
public synchronized Connection getConnectionFromPool() {
Connection connection = null;
// Check if there is a connection available. There are times when all
// the connections in the pool may be used up
if (connectionPool.size() > 0) {
connection = (Connection) connectionPool.firstElement();
connectionPool.removeElementAt(0);
}
// Giving away the connection from the connection pool
return connection;
}
public synchronized void returnConnectionToPool(Connection connection) {
// Adding the connection from the client back to the connection pool
connectionPool.addElement(connection);
}
public static void main(String args[]) {
new ConnectionPoolManager();
}
}
can any one help?
The purpose of connection pooling is to maintain a number of open connections to a database so that when your application requires a connection it does not have to go through the potentially resource and time intensive process of opening a new connection.
When an application requires a database connection it 'borrows' one from the pool. When it's done, it gives it back and that connection may be reused at some later point.
Once you have obtained a connection, you use it in your application through the JDBC (Java Database Connectivity) API.
Oracle's basic tutorial for using JDBC can be found at http://docs.oracle.com/javase/tutorial/jdbc/basics/index.html
Another thing to keep in mind is that alot of work has gone into developing connection pools already, and it probably is not necessary to reinvent the wheel, except perhaps as a learning excercise. Apache Tomcat's connection pool implementation can be used outside of Tomcat (for example, in a standalone Java application) and is fairly flexible and easy to configure. It can be found at https://people.apache.org/~fhanik/jdbc-pool/jdbc-pool.html
I would say the code is pretty self explanatory.
You create an instance of the pool, personally, I prefer to use a singleton, but that's another topic
ConnectionPoolManager connectionPoolManager = new ConnectionPoolManager();
Now, every body that wants a connection, is going to need a reference to this manager. When you need to, you request a free connection from the pool...
public void queryDatabaseForStuff(ConnectionPoolManager cpm) throws SQLException {
Connection con = cpm.getConnectionFromPool();
//....
Once you're finished with the connection, you pass it back to the manager...
try {
//...
} finally {
cmp.returnConnectionToPool(con);
}
Now. You might like to investigating a blocking process that will block the current call to getConnectionFromPool while the pool is empty, meaning that it will either throw an exception (if you want to include a time out feature) or a valid connection.
When re-pooling a Connection, you might like to check to see if the Connection has been closed or not and have some kind of revival process to ensure that the pool is awlays close to capcaity...
Please check this link for getting detailed answer - https://examples.javacodegeeks.com/core-java/sql/jdbc-connection-pool-example/
You don't need to recreate your Connection object pool , instead please use the libraries provided by Apache . Please be clear of the following :
1 - Why and what made you think of connection pool ?
2 - Use the following Apache commons-dbcp lib in your Maven project and then use the classes as per documentation .
3. Does this solve all your problems ?
ITs Better to perform the connection pooling via in built API
Like
DBCP or this.
Its always better let these API perform the connection pooling and programmatically creating and maintaining the connection pooling always painful activity.
I am using Java 1.7 and Postgres via the Postgres JDBC drivers. The database connection will be used from a Web Service. In testing, I got the following error:
FATAL: connection limit exceeded for non-superusers
I solved the error by making my connection static, and, only creating once. My question is, is a static connection safe? Is this the right way to do this?
I am using the connection via a ConnectionFactory that looks something like this:
public class ConnectionFactory
{
String driverClassName = "org.postgresql.Driver";
String connectionUrl = "jdbc:postgresql://localhost:5432/dbName";
String dbUser = "user";
String dbPwd = "password";
private static ConnectionFactory connectionFactory = null;
private static Connection conn = null;
private ConnectionFactory()
{
try
{
Class.forName(driverClassName);
}
catch (ClassNotFoundException e)
{
e.printStackTrace();
}
}
public Connection getConnection() throws SQLException
{
if (conn == null)
{
conn = DriverManager.getConnection(connectionUrl, dbUser, dbPwd);
}
return conn;
}
public static ConnectionFactory getInstance()
{
if (connectionFactory == null)
{
connectionFactory = new ConnectionFactory();
}
return connectionFactory;
}
}
Postgres JDBC driver is documented as thread safe and the connection can be used by multiple threads if so required. If a thread attempts to use the connection while another one is using it, it will wait until the other thread has finished its current operation.
Connection pooling may be used anyway for performance reasons.
Sorry, original post didn't look at your code carefully. Wow. I still can't read. Anyway, third time's the charm. If your code is single threaded - then your fine. If it's multi-threaded, use something like the Commons connection pools to manage your connections. It looks like the driver is thread safe but the connection shouldn't be viewed as thread safe. So, once the driver is loaded you can safely call getConnection on the driver from multiple threads, but the connection shouldn't be shared across threads.
In one of the interviews that I faced,I was asked to implement connection pooling.
So approach was this:
Create a List or HashMap
Create predefined number of connections
Add them to the collection.
Now when the ConnectionImpl getConnection() method of ConnectionPoolingImpl class is invoked return a connection reference.
Now when someone returns the connection (releaseConnection(ConnectionImpl O)) how can I ensure that when the same application again tries to reuse the connection object, my implementation throws an exception?
The same connection object might have been returned to a new application and that should be able to use it.
My point of view would be to maintain a flag variable in another array kind of structure for each Connectionimpl object and set that variable to a valid value. When user returns the connection object I would make that some invalid value. For every operation in my ConnectionImpl, I will have to verify if the user had a valid flag.
What would you say to that approach?
I would not return the "real" connection object from the pool, but a wrapper which gives the pool control of connection life cycle, instead of the client.
Assume you have a really simple connection, which you can read int values from:
interface Connection {
int read(); // reads an int from the connection
void close(); // closes the connection
}
An implementation reading from a stream could look like this (ignoring exceptions, EOF handling, etc):
class StreamConnection implements Connection {
private final InputStream input;
int read(){ return input.read(); }
void close(){ input.close(); }
}
Furthermore, let's assume you have a pool for StreamConnections that looks like this (again, ignoring exceptions, concurrency, etc):
class StreamConnectionPool {
List<StreamConnection> freeConnections = openSomeConnectionsSomehow();
StreamConnection borrowConnection(){
if (freeConnections.isEmpty()) throw new IllegalStateException("No free connections");
return freeConnections.remove(0);
}
void returnConnection(StreamConnection conn){
freeConnections.add(conn);
}
}
The basic idea here is OK, but we can't be sure the connections are returned, and we can't be sure they aren't closed and then returned, or that you don't return a connection which came from another source altogether.
The solution is (of course) another layer of indirection: Make a pool which returns a wrapper Connection which, instead of closing the underlying connection when close() is called, returns it to the pool:
class ConnectionPool {
private final StreamConnectionPool streamPool = ...;
Connection getConnection() {
final StreamConnection realConnection = streamPool.borrowConnection();
return new Connection(){
private boolean closed = false;
int read () {
if (closed) throw new IllegalStateException("Connection closed");
return realConnection.read();
}
void close() {
if (!closed) {
closed = true;
streamPool.returnConnection(realConnection);
}
}
protected void finalize() throws Throwable {
try {
close();
} finally {
super.finalize();
}
}
};
}
}
This ConnectionPool would be the only thing the client code ever sees. Assuming it is the sole owner of the StreamConnectionPool, this approach has several advantages:
Reduced complexity and minimal impact on client code - the only difference between opening connections yourself and using the pool is that you use a factory to get hold of Connections (which you might already do, if you're using dependency injection). Most importantly, you always clean up your resources in the same way, i.e., by calling close(). Just like you don't care what read does, as long as it gives you the data you need, you don't care what close() does, as long as it releases the resources you've claimed. You shouldn't have to think whether this connection is from a pool or not.
Protection against malicious/incorrect usage - clients can only return resources they've retrieved from the pool; they can't close the underlying connections; they can't use connections they've already returned... etc.
"Guaranteed" returning of resources - thanks to our finalize implementation, even if all references to a borrowed Connection is lost, it is still returned to the pool (or does at least stand a chance to be returned). The connection will of course be held longer than necessary this way - possibly indefinitely, since finalization isn't guaranteed to ever run - but it's a small improvement.
I'd just tell them I'd use the JdbcConnectionPool class (here) that comes with H2 (you can probably copy it out). Screw trying to implement one :) It could be a trick question.
ConnectionPool implemenation
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;
/** A Connection Pool with 5 Available Connections **/
class ConnectionPool {
private List<Connection>availableConnections =
new ArrayList<Connection>();
private List<Connection>usedConnections = new ArrayList<Connection>();
private final int MAX_CONNECTIONS = 5;
private String URL;
private String USERID;
private String PASSWORD;
/** Initialize all 5 Connections and put them in the Pool **/
public ConnectionPool(String Url, String UserId, String password)
throws SQLException {
this.URL = Url;
this.USERID = UserId;
this.PASSWORD = password;
for (int count = 0; count <MAX_CONNECTIONS; count++) {
availableConnections.add(this.createConnection());
}
}
/** Private function,
used by the Pool to create new connection internally **/
private Connection createConnection() throws SQLException {
return DriverManager
.getConnection(this.URL, this.USERID, this.PASSWORD);
}
/** Public function, used by us to get connection from Pool **/
public Connection getConnection() {
if (availableConnections.size() == 0) {
System.out.println("All connections are Used !!");
return null;
} else {
Connection con =
availableConnections.remove(
availableConnections.size() - 1);
usedConnections.add(con);
return con;
}
}
/** Public function, to return connection back to the Pool **/
public boolean releaseConnection(Connection con) {
if (null != con) {
usedConnections.remove(con);
availableConnections.add(con);
return true;
}
return false;
}
/** Utility function to check the number of Available Connections **/
public int getFreeConnectionCount() {
return availableConnections.size();
}
}
OK, so if I understand correctly, your question is basically "how can we can ensure that a thread doesn't return a connection to the pool and then carry on using it?". Provided you don't pass back the "raw" Connection object to the caller, then the answer is essentially "you can put some control in somewhere if you want".
The actual check could involve marking each connection with which Thread "owns" it at a given moment, then making sure this is always Thread.currentThread() during any call to use the connection.
It doesn't matter terribly much what object you do pass back to user of the connection to represent the connection: it could be your own wrapper implementation of Connection, or just some other wrapper object with your methods for executing queries. Whichever you use, you just need to make the abovementioned check before executing any query. Bear in mind that for security you generally shouldn't be allowing "raw" arbitrary SQL to be executed, but that all queries should be based on a well-defined PreparedStatement. So there's no particular compulsion to return an actual implementation of Connection, other than this might in some circumstances help you migrate existing code (and/or if you've decided you really do want to permit execution of arbitrary SQL).
In many circumstances, you could also not bother making this check. You're passing a caller a means to access your database, so it's a bit like trying to stop pilots from crashing planes into buildings by scanning them for explosives at airports: they all ready have a means of messing up your system whether or not you make the additional check.