Hello i am trying to implement a database-object("connection") pooler for BerkeleyDB...
I decided to use a singleton EJB propably or ENUM singleton implementation for this..
A final concurrenthash map would store database objects with a timestamp...
the method getConnection() would use double check locking as long as the value from map is volatile. - No performance issues i believe..(Java Connection Pooler getConnection is synchronized!!)
The database is spread into 100 files + the daily ones.. (application designed in mid seventies 1976)..
So far everything is fine... But i want to close daily unused handles.
So i decided to use a Timer to run every 24 hours a cleanup routine..
The problem is that how can i ensure that during cleanup a connection to be closed isnt requested ?
Pseudo algorithm
cleanup(){
for(Database db in map){
if(db.getLastAccess - now >24hours) {
res=map.remove("key",db);
db.close();
}
}
}
i know that the above isnt thread safe..How could i block getconnection ? Because many things could go wrong... "If condition" may be true but before removing db obj getLastAccess could be changed! Cleanup would be called by single thread though..
Is there any solution to block getconnection somehow so cleanup to work or anyother solution?
I am not sure if you currently do this, but if you have a way to determine if a connection is in use this would make this slightly easier. One thing that you can do, is iterate over the connections in your pool. When you find one that matches your criteria for being closed, try to mark it as being in use (assuming that a connection that is in use will not be returned as a open connection). If you succeed, close it. Otherwise, check it until it becomes free and you are able to mark it as being in use. Once you have been able to do this, you should be able to close it.
Each connection would have a lock associated with it, in order for the connection to be returned by the getConnection method, the correct lock would have to be acquired. The cleanup method would also need to acquire the lock before closing a connection. Take a look at the java.util.concurrent.lock package.
Maybe a Semaphore is a better solution. Follow the link for an example.
I've never worked with BerkeleyDB, but I assume it has a JDBC interface. Can't you use an out of the box solution like DBCP or c3p0? Also check the Pool Component, it is a generic pool interface.
Related
I've been using StackOverFlow for a long time now and always found existing answers, but this time I couldn't find any information about what I'm trying to do.
Using java, I have a process composed of about 10 different tasks that gather distinct data from the database using pure jdbc (no ejb/jpa here). Each task (callable) can actually be run concurrently and is responsible for obtaining a connection, which is what we are doing. However we're randomly experiencing trouble with the connection pool (accessed via jndi), sometimes we're blocked because the connection pool doesn't have any available connection.
To solve this problem, I thought we could change the way we're obtaining the connections, instead of letting each callable opening and closing a connection ( following the number of tasks to execute and the number of threads to use in the ThreadPoolExecutor), I would like to create some kind of local connections pool dedicated to this process, so that we're sure nothing will block later (eventually if we can't acquire all the requested connections, we would then adapt the number of threads to launch with a minimum of 1)
My colleagues approve this idea, but what surprises me is that I can't found any similar approaches or discussion on the web (maybe I'm not using the right keywords).
I would like to know what you think about this idea, whether you already tried something similar or if I'm missing something important.
In advance, thank you.
You have not mentioned which connection pool is used. If it is not HikariCP and you are allowed to switch, having contributed there I recommend it.
HikariCP seems rather interesting finally, i'll have to check this further. But this isn't directly related to the question :)
Just a little return of experience, my idea is working, with one caveat, I couldn't get rid of one downcast from a runnable to my implementation on which I can do .setConnection() during the before() of my ExecutorService. And all tasks must have been given to the executor with the execute() method, otherwise the runnable is autolatically wrapped in a FutureTask without the ability to access the inner runnable. Maybe one of you know of to do this correctly ?
I want to implement mechanism which will be closing connections if there are not used by specific period of time. This time is constant for all of the connections. Opened connections can be used many times, so I need to update usage time and always compute difference between current time and usage time. I also need to close connections which excess my timeout.
My opened connections are in Map. (Map<Id, Connection>) where Id is an Integer.
I thought about resolving my problem with DelayQueue, but there is no possible to update usage (in this case delay) time in this type of Queue.
I also know that this mechanism should work in separate thread.
Please, give me some tip about the best way of implementation or example. What kind of data structure should I use?
I can use Spring also (maybe there is some good mechanism and I don't know about it).
If you're speaking about database connectivity then just use connection poolers such as c3po, hikariCP, BoneCP and so on. Don't reinvent the wheel.
Take a look at the HikariCP code. Specifically, look at:
ConcurrentBag
PoolBagEntry
BaseDataSource.getConnection()
BaseDataSource.releaseConnection()
HouseKeeper inner class
While HikariCP is a database connection pool, you can use ConcurrentBag as is, use the HouseKeeper basically as is, slightly modify PoolBagEntry, and lift the basic gist from getConnection() and releaseConnection(), to create a generic pool.
I have a Data base connection pool. There are consumers who take connection from that pool.
But i cant trust those consumers because lot of them are not returning my connection back. Hence the pool starves and many consumers are forced too wait for infinite.
for example:
class Consumer{
void someMethod(){
Connection con=Pool.getConnection();
//some bloody steps which throws exception
con.goodBye();//giving back the connection to the pool
}
}
Because of the exception and may be because of arrogance the connection is not given back always. I have no way to restrict the usage of Pool api in the consumers' class.
How can i get my connection back?.(I have no way to force the Consumer)
I believe there is no fool proof solution for this(May be im not that smart). Still can any one can come up with a pretty good solution.
One solution which i got is checking whether any exception occurs in the Consumer class, if Exception occurs then take back the connection force fully.
Or is there any new revolutionary DBPool design pattern which are not very popular for this type of typical scenarios(even though i think that my case is very generic, any one can forget to give back the connection back to the pool.)
That's bad client code. The code should handle the case of an exception and close the connection when done.
There's no way for you to know from your code if that's not being done, though. It's the client code's fault and problem if it doesn't do that.
Having a sane timeout is the only way to limit this, but it still does not "solve" it, ultimately.
--
You mention in comments that this pool is shared among multiple clients. That shifts the responsibility back to you, of course.
Can you limit each client to only using X connections at once? This way, at least they can only tie up so many at one time.
Otherwise, you could create separate pools per client. That sort of just moves the problem down the stack, but might be appropriate, depending on the logistics involved.
Do not return connection objects, return proxy objects representing connections instead. These proxy objects, when finalized, should say goodbye to the connection they stand for. If a proxy is not properly closed, it will eventually be garbage-collected, and adjust the connection state at this time.
Two issues here. First, the time before GC is unpredictable. Better than forever, but still can be very long. Second, be aware of side effects of complex calls in the finalizer, object resurrection in particular. There are some rare but ugly scenarios that prevent objects to be collected at all.
why don't you look at using WeakReference, here you can adjust your code to return a weak reference to a connection, when the thread using the connection dies, the object will have no reference (except from your WeakHashMap), you can then periodically identify these objects and call the goodBye method using a thread.
here is an article which can help you understand this better.
.net also has a WeakReference class which behaves very similar to this.
Have you tried catching the exception and closing the connection in the catch clause?
class Consumer{
void someMethod(){
Connection con=Pool.getConnection();
try{
//some bloody steps which throws exception
}catch(Exception e){
con.goodBye();
}
con.goodBye();//giving back the connection to the pool
}
}
Edit: you could also use a finally block to remove the redundant code and make sure your connection gets closed in every case. I am assuming this is java code, no experience with C#.
The only way I can think of is not to give the consumers access to the connection pool directly, but have your own list of connections. Then reclaim the connection after a timeout, say 60 seconds.
I am using java to create an interface to connect to a database. Each time I want to make a call to the database I need to create new connections to the database, which would make calling the database say 10 times slow.
To avoid having to create new connections each time I want to call the database I have a java thread running that holds all of the connection information.
To write/read from the database I want to create a thread that uses the connection information stored in the thread that's already running, use it to execute specified read/write functions, and then exit.
However I am having trouble accessing this information from the thread which is already running. What would be the best way to accomplish this?
This is a terrible idea, because java.sql.Connection is not thread-safe.
A better idea would be to use a connection pool. Let each thread check out a connection, use it, and put it back.
best way is not to re-invent the wheel. there are good open spource implementations of the connection pooling and i suggest you use them.
if you are already running in a container then use DataSource. look into c3p0 (http://sourceforge.net/projects/c3p0/) and commons-dbcp (http://commons.apache.org/dbcp/)
Why do you need a thread running to keep your connection open, just store it somewhere and execute queries as soon as you need it.. should it work?
In any case if you really want a thread you should care about having a synchronized collection (check Collections.asSynchronizedList) that can be accessed and managed from your thread and others too.
To overcome visibility problems just declare it as a static final variable, so you won't have any problems in accessing it from outside the thread you declared it into.
Another easy solution (since connection seems to be not thread-safe) is not to use a thread but use just a monitor: you can easily manage a wait()/notify() mechanism for which a thread that wants to execute a query checks if connection is "free". if it is occupies the monitor and do whatever it wants before notifying all waiting threads.
why are you doing this? There are frameworks, like Spring or equivalent, which will manage your connections for you. Don't reinvent the wheel man....
I would recommend to use a generic object pool instead of building your own solution and suggest to check Commons Pool from Apache Commons (this is an API for generic Object pooling, this isn't DBCP).
I have lots of method in my database class that all of them have one statement, when should I close the connection? In each method or at the end of database class?
You should close the connection when you are finished your transaction. Since we don't know the contents of the class or its usage, it's impossible to tell when your access of the connection begins or ends.
Actually, that may not even be true if the connection is dedicated for a specific usage and not in a pool. You may want to keep it open for the duration of your application.
We've found that the best policy is to get a connection from the connection pool, execute a single transaction, and then put the connection back into the pool immediately. This way you don't have a connection being held onto for long blocks of logic, thus preventing other threads from using it - which is an issue for scalability.
As a best practice, you should close the connection in the logical place after you are done - right after all of database activity for that task is done.
Generally, you should close a connection in the same method that opens it. Closing and opening connections isn't an arduous task, since modern DB servers keep even closed connections on "hot standby", so they are quickly accessed through a connection pool. Leaving them open though...that can get you in trouble and can be a nightmare to debug.
use lombok and it will handle both try/catch and conn.close() for you
public void doSomething() throws SQLException {
#Cleanup Connection connection = database.getConnection();
}
lombok
It depends when and how repeatedly you are using these methods. If they are sequential you should close the connection only in the end, instead of open and close often
This largely depends on what your Database class does. If your methods are called individually at various times then the methods should be responsible for opening and closing the connection. However, if the class does some kind of big processing operation that calls many methods then you may want to open and close the connection outside of the individual methods.
The most important thing is that wherever you open the connection, you also close the connection. Otherwise you get into the business of making assumptions about the state of the connection which can get you into trouble.
Close the connection (and statement and resultset!) in the same method block as you've acquired it, in the finally block of the try block where they are opened.
The general idiom is:
public void doSomething() throws SQLException {
Connection connection = null;
try {
connection = database.getConnection();
} finally {
if (connection != null) try { connection.close(); } catch (SQLException logOrIgnore) {}
}
}
Closing of connection, statement and resultset should happen in reversed order as they're opened.
If your actual concern is performance, then consider using a connection pool to improve connecting performance, for example C3P0. This by the way does not change the general idiom. Just continue writing the same idiom, the connection pooling implementation will worry itself further under the hoods.
Also see this article for more practices/examples.
Always close the connection when you finish executing your transaction. It is a good practice to obtain and close your connection in the same method. If you have separate transactions that are tightly coupled you might execute them with the same connection, but for best practices, I try to execute one transaction per connection.
If it is an option, don't explicitly mess with connections at all. Use a full blown orm framework like hibernate or use something significantly more lightweight like spring jdbc templates.
It depends somewhat on the context your application operates in. If it's a web app you need to be careful to open your connection, do whatever work is needed, and close the connection quickly. However, in a C/S or batch environment it may be better to acquire the connection and hold onto it as long as the user is interacting "frequently" (for whatever value of "frequently" you choose), especially if the user has expectations of rapid response time and it's expensive (in terms of time or resources) to acquire the connection to your particular variety of database.
I like to set a timer every time the user has the app go to the database. If/when the timer expires, close the connection, then re-open the next time he/she/it wants to hit the database again. Timer expiration can be somewhere between 1 and 20 minutes. Just so long as it's less than the database's "inactivity disconnect" time.
Share and enjoy.