Develop Reference

Using ThreadPoolExecutor and connection pool without random blocking method call

I've been using StackOverFlow for a long time now and always found existing answers, but this time I couldn't find any information about what I'm trying to do.
Using java, I have a process composed of about 10 different tasks that gather distinct data from the database using pure jdbc (no ejb/jpa here). Each task (callable) can actually be run concurrently and is responsible for obtaining a connection, which is what we are doing. However we're randomly experiencing trouble with the connection pool (accessed via jndi), sometimes we're blocked because the connection pool doesn't have any available connection.
To solve this problem, I thought we could change the way we're obtaining the connections, instead of letting each callable opening and closing a connection ( following the number of tasks to execute and the number of threads to use in the ThreadPoolExecutor), I would like to create some kind of local connections pool dedicated to this process, so that we're sure nothing will block later (eventually if we can't acquire all the requested connections, we would then adapt the number of threads to launch with a minimum of 1)
My colleagues approve this idea, but what surprises me is that I can't found any similar approaches or discussion on the web (maybe I'm not using the right keywords).
I would like to know what you think about this idea, whether you already tried something similar or if I'm missing something important.
In advance, thank you.

You have not mentioned which connection pool is used. If it is not HikariCP and you are allowed to switch, having contributed there I recommend it.

HikariCP seems rather interesting finally, i'll have to check this further. But this isn't directly related to the question :)

Just a little return of experience, my idea is working, with one caveat, I couldn't get rid of one downcast from a runnable to my implementation on which I can do .setConnection() during the before() of my ExecutorService. And all tasks must have been given to the executor with the execute() method, otherwise the runnable is autolatically wrapped in a FutureTask without the ability to access the inner runnable. Maybe one of you know of to do this correctly ?

How to properly handle long-lived MySQL connections in Java when using Guice Injections?

I hate stating questions that apparently seem to have a lot of solutions online, but we really cannot seem to find any valid best-practice solution for our case, and therefore felt we had no choice.
We are building an RESTful server application in which the periods between use may differ from a couple of hours to multiple months.
The server is hosted by Jetty. We are not using any ORM, but the application is layered into three layers (WebService- , Business- and Data Layer). The Data layer exist of one class injected through the Guice framework. The JDBC (MySQL connection) is instantiated within the constructor of this class. At first, we had a lot of trouble with too many connections before we understood that Guice by default creates a new instance on each request(ref). To get rid of this problem, and because our Data layer class is stateful, we made the class injected as Singleton.
Now we've foreseen that we might run into trouble when our REST application is not used for some time, since the connection will time out, and no new connection will be instantiated, as the constructor will only be called once.
We now have multiple solutions, but we cannot seem to figure out the best way to solve this, as none of them really seems to be that good. Any input or suggestions to other solutions would be well appreciated.
1. Extend the configured mysql timeout interval
We really do not want this, as we think it's really not best practice. We should of course not have any leaking connection objects, but if we have, they would fill up the free space of connections available.
2. Instantiate a new connection at the beginning of each method, and close it at the end
This is, as far as we understand, not best practice at all, as it would cause a lot of overhead, and should be avoided if possible?
3. Change the injections back to "per-request", and close the pool at the end of each method
This would be even worse than #2, as we would not only instantiate a new connection, but also instantiate a new object on each request?
4. Check the status of the connection at the beginning of each method, and instantiate a new connection if it's closed
An example would be to ping (example) the mysql, and instantiate a new connection if it throws an exceptions. This would work, but it would create some overhead. Any ideas of whether this input actually would make any difference to the performance?
5. Explicitly catch any exceptions being thrown in the methods indicating that the connection is down, and if so - instantiate a new connection
This way, we would get rid of the ping overhead, but it would complicate our code remarkably, as we would have to figure out a way to make sure that the methods will return what they would have returned if the connection where already alive.
6. Use a connection pool
We are not familiar with connection pools, other than when using an application server (i.e Glassfish). We're also wondering whether this actually would solve our problem? And if so; any suggestions on any framework providing us with connection pools? Here they suggest using PLUS with Jetty.
Please ask if there's anything unclear. I might have forgotten to add some vital information. This is to me more of a design question, but I'd be glad to provide any code if anyone thinks that would help.
Thanks in advance!

Connection pools are the way to go.
They have a number of advantages:
They check your connections for you - this deals with timeouts
They control the number of connections
You can simply close the connection when your done - you don't need to keep references
You should certainly keep connections in some sort of pool, and in fact you will almost certainly end up writing one yourself eventually if you don't bite the bullet.
By the time you have implemented connection checking so that they don't go stale, some sort of connection holder so that you don't need to re-open them each time, some sort of exception handling code...you get my drift.
I have used dbcp and boneCP and both are very easy to use and configure and will save you hours and hours of frustration dealing with JDBC connection issues.
I am not overly familiar with Guice but I assume it has some way to provide your own factory method for Object, so you can use that to get connections from your pool and then simple call close() when you're done to return them to the pool.
If you're using a webserver you can always use an interceptor or filter to bind connections to the work thread and discard them after processing in which case your connection provider would only need to yank the one tied to the current thread.

Inject a Provider<Connection> instead and have the provider give out connections (EDIT: at the time you need it) from a connection pool which can detect stale entries.
Unreturned connections should be discarded from the pool.

Database Pooler

Hello i am trying to implement a database-object("connection") pooler for BerkeleyDB...
I decided to use a singleton EJB propably or ENUM singleton implementation for this..
A final concurrenthash map would store database objects with a timestamp...
the method getConnection() would use double check locking as long as the value from map is volatile. - No performance issues i believe..(Java Connection Pooler getConnection is synchronized!!)
The database is spread into 100 files + the daily ones.. (application designed in mid seventies 1976)..
So far everything is fine... But i want to close daily unused handles.
So i decided to use a Timer to run every 24 hours a cleanup routine..
The problem is that how can i ensure that during cleanup a connection to be closed isnt requested ?
Pseudo algorithm
cleanup(){
for(Database db in map){
if(db.getLastAccess - now >24hours) {
res=map.remove("key",db);
db.close();
}
}
}
i know that the above isnt thread safe..How could i block getconnection ? Because many things could go wrong... "If condition" may be true but before removing db obj getLastAccess could be changed! Cleanup would be called by single thread though..
Is there any solution to block getconnection somehow so cleanup to work or anyother solution?

I am not sure if you currently do this, but if you have a way to determine if a connection is in use this would make this slightly easier. One thing that you can do, is iterate over the connections in your pool. When you find one that matches your criteria for being closed, try to mark it as being in use (assuming that a connection that is in use will not be returned as a open connection). If you succeed, close it. Otherwise, check it until it becomes free and you are able to mark it as being in use. Once you have been able to do this, you should be able to close it.

Each connection would have a lock associated with it, in order for the connection to be returned by the getConnection method, the correct lock would have to be acquired. The cleanup method would also need to acquire the lock before closing a connection. Take a look at the java.util.concurrent.lock package.
Maybe a Semaphore is a better solution. Follow the link for an example.
I've never worked with BerkeleyDB, but I assume it has a JDBC interface. Can't you use an out of the box solution like DBCP or c3p0? Also check the Pool Component, it is a generic pool interface.

Fast way to execute MySQL stored procedures in Java within multiple threads?

What is the fastest option to issue stored procedures in a threaded environment in Java? According to http://dev.mysql.com/doc/refman/5.1/en/connector-j-usagenotes-basic.html#connector-j-examples-preparecall Connection.prepareCall() is an expensive method. So what's the alternative to calling it in every thread, when synchronized access to a single CallableStatement is not an option?

The most JDBC drivers use only a single socket per connection. I think MySQL also use also a single socket. That it is a bad performance idea to share one connection between multiple threads.
If you use multiple connection between different threads then you need a CallableStatment for every connection. You need a CallabaleStatement pool for every connection. The simplest to pool it in this case is to wrap the connection class and delegate all calls to the original class. This can be create very fast with Eclipse. In the wrapped method prepareCall() you can add a simple pool. You need also a wrapped class of the CallableStatement. The close method return the CallableStatement to the pool.
But first you should check if the call is real expensive because many driver has already such poll inside. Create a loop of prepareCall() and close() and count the time.

Connection is not thread safe, so you can't share it across threads.
When you prepareCall, the JDBC driver (may) be telling the RDBMS system to do a lot of work that is stored on the server side. You may be guilty of premature optimization here.

After giving this a little thought it seems that if you are having issues with this infrastructure code then your problems are elsewhere. Most applications do not take an inordinate amount of time doing this stuff.
Make sure you are using a DataSource, most do connection caching and some even do caching of statements.
Also for this to be a performance bottle neck it would imply that you are doing many queries one after the other, or that your pool of connections is too small. Maybe you should do some benchmarking on your code to see how much time the stored proc is taking vs how much time the JDBC code is taking.
Of course I would follow the MySQL recommendation of using CallableStatement, I am sure they have benchmarked this. Most apps do not share anything between Threads and it is rarely an issue.

Accessing variables and methods across threads

I am using java to create an interface to connect to a database. Each time I want to make a call to the database I need to create new connections to the database, which would make calling the database say 10 times slow.
To avoid having to create new connections each time I want to call the database I have a java thread running that holds all of the connection information.
To write/read from the database I want to create a thread that uses the connection information stored in the thread that's already running, use it to execute specified read/write functions, and then exit.
However I am having trouble accessing this information from the thread which is already running. What would be the best way to accomplish this?

This is a terrible idea, because java.sql.Connection is not thread-safe.
A better idea would be to use a connection pool. Let each thread check out a connection, use it, and put it back.

best way is not to re-invent the wheel. there are good open spource implementations of the connection pooling and i suggest you use them.
if you are already running in a container then use DataSource. look into c3p0 (http://sourceforge.net/projects/c3p0/) and commons-dbcp (http://commons.apache.org/dbcp/)

Why do you need a thread running to keep your connection open, just store it somewhere and execute queries as soon as you need it.. should it work?
In any case if you really want a thread you should care about having a synchronized collection (check Collections.asSynchronizedList) that can be accessed and managed from your thread and others too.
To overcome visibility problems just declare it as a static final variable, so you won't have any problems in accessing it from outside the thread you declared it into.
Another easy solution (since connection seems to be not thread-safe) is not to use a thread but use just a monitor: you can easily manage a wait()/notify() mechanism for which a thread that wants to execute a query checks if connection is "free". if it is occupies the monitor and do whatever it wants before notifying all waiting threads.

why are you doing this? There are frameworks, like Spring or equivalent, which will manage your connections for you. Don't reinvent the wheel man....

I would recommend to use a generic object pool instead of building your own solution and suggest to check Commons Pool from Apache Commons (this is an API for generic Object pooling, this isn't DBCP).