I am using java to create an interface to connect to a database. Each time I want to make a call to the database I need to create new connections to the database, which would make calling the database say 10 times slow.
To avoid having to create new connections each time I want to call the database I have a java thread running that holds all of the connection information.
To write/read from the database I want to create a thread that uses the connection information stored in the thread that's already running, use it to execute specified read/write functions, and then exit.
However I am having trouble accessing this information from the thread which is already running. What would be the best way to accomplish this?
This is a terrible idea, because java.sql.Connection is not thread-safe.
A better idea would be to use a connection pool. Let each thread check out a connection, use it, and put it back.
best way is not to re-invent the wheel. there are good open spource implementations of the connection pooling and i suggest you use them.
if you are already running in a container then use DataSource. look into c3p0 (http://sourceforge.net/projects/c3p0/) and commons-dbcp (http://commons.apache.org/dbcp/)
Why do you need a thread running to keep your connection open, just store it somewhere and execute queries as soon as you need it.. should it work?
In any case if you really want a thread you should care about having a synchronized collection (check Collections.asSynchronizedList) that can be accessed and managed from your thread and others too.
To overcome visibility problems just declare it as a static final variable, so you won't have any problems in accessing it from outside the thread you declared it into.
Another easy solution (since connection seems to be not thread-safe) is not to use a thread but use just a monitor: you can easily manage a wait()/notify() mechanism for which a thread that wants to execute a query checks if connection is "free". if it is occupies the monitor and do whatever it wants before notifying all waiting threads.
why are you doing this? There are frameworks, like Spring or equivalent, which will manage your connections for you. Don't reinvent the wheel man....
I would recommend to use a generic object pool instead of building your own solution and suggest to check Commons Pool from Apache Commons (this is an API for generic Object pooling, this isn't DBCP).
Related
I've been using StackOverFlow for a long time now and always found existing answers, but this time I couldn't find any information about what I'm trying to do.
Using java, I have a process composed of about 10 different tasks that gather distinct data from the database using pure jdbc (no ejb/jpa here). Each task (callable) can actually be run concurrently and is responsible for obtaining a connection, which is what we are doing. However we're randomly experiencing trouble with the connection pool (accessed via jndi), sometimes we're blocked because the connection pool doesn't have any available connection.
To solve this problem, I thought we could change the way we're obtaining the connections, instead of letting each callable opening and closing a connection ( following the number of tasks to execute and the number of threads to use in the ThreadPoolExecutor), I would like to create some kind of local connections pool dedicated to this process, so that we're sure nothing will block later (eventually if we can't acquire all the requested connections, we would then adapt the number of threads to launch with a minimum of 1)
My colleagues approve this idea, but what surprises me is that I can't found any similar approaches or discussion on the web (maybe I'm not using the right keywords).
I would like to know what you think about this idea, whether you already tried something similar or if I'm missing something important.
In advance, thank you.
You have not mentioned which connection pool is used. If it is not HikariCP and you are allowed to switch, having contributed there I recommend it.
HikariCP seems rather interesting finally, i'll have to check this further. But this isn't directly related to the question :)
Just a little return of experience, my idea is working, with one caveat, I couldn't get rid of one downcast from a runnable to my implementation on which I can do .setConnection() during the before() of my ExecutorService. And all tasks must have been given to the executor with the execute() method, otherwise the runnable is autolatically wrapped in a FutureTask without the ability to access the inner runnable. Maybe one of you know of to do this correctly ?
I came across this interview question : How will you manage a db connection pool?
My thought was:
I will create an ArrayBlockingQueue<Connection>, create come connection objects and put them in the queue when ajvm starts up. Then wrap this in some form of an enum singleton so there is only one such queue and it stays alive for the life of the JVM.
Then use some kind of utility/driver class that will take connections from the queue and return them back to the queue.
I am thinking what else i need to say to this? Do i need to make the queue thread safe so that multiple requests dont have the same connection?
In my opninion you're missing several points here:
Connections should be returned to the initial state when returning back to the pool. For instance, connection.setAutocommit(...); should definitely be reverted
I't a good idea to wrap native connection into your own implementation of javax.sql.Connection interface to control and monitor actions performed on the connection. With this pattern you can also implement a valuable feature: return the connection back to pool on close(); call
You need some menans to control the number of the connections in a pool based on the actual pool utilization. Take a look at how "capacity" and "load factor" are implemented in Java collection to get a rough implementation idea
Connections should be monitored if they're alive. It's not that easy to archive for all possible databases.
I ran into the exact issue that setNetworkTimeout is supposed to solve according to Oracle. A query got stuck in socket.read() for several minutes.
But I have little idea what the first parameter for this method needs to be. Submitting a null causes an AbstractMethodError exception, so... does the implementation actually need some sort of thread pool just to set a network timeout?
Is there some way to achieve the same effect without running a thread pool just for this one condition?
It seems like the documentation explains this horribly, but without looking at any code behind the class my guess would be that you are expected to pass an Executor instance to the method so that implementations can spawn jobs/threads in order to check on the status of the connection.
Since connection reads will block, in order to implement any sort of timeout logic it's necessary to have another thread besides the reading one which can check on the status of the connection.
It sounds like a design decision was made that instead of the JDBC driver implementing the logic internally, of how/when to spawn threads to handle this, the API wants you as the client to pass in an Executor that will be used to check on the timeouts. This way you as the client can control things like how often the check executes, preventing it from spawning more threads in your container than you like, etc.
If you don't already have an Executor instance around you can just create a default one:
conn.setNetworkTimeout(Executors.newFixedThreadPool(numThreads), yourTimeout);
As far as Postgres JDBC driver is concerned (postgresql-42.2.2.jar), the setNetworkTimeout implementation does not make use of the Executor parameter. It simply sets the specified timeout as the underlying socket's timeout using the Socket.setSoTimeout method.
It looks like the java.sql.Connection interface is trying not to make any assumptions about the implementation and provides for an executor that may be used if the implementation needs it.
Hello i am trying to implement a database-object("connection") pooler for BerkeleyDB...
I decided to use a singleton EJB propably or ENUM singleton implementation for this..
A final concurrenthash map would store database objects with a timestamp...
the method getConnection() would use double check locking as long as the value from map is volatile. - No performance issues i believe..(Java Connection Pooler getConnection is synchronized!!)
The database is spread into 100 files + the daily ones.. (application designed in mid seventies 1976)..
So far everything is fine... But i want to close daily unused handles.
So i decided to use a Timer to run every 24 hours a cleanup routine..
The problem is that how can i ensure that during cleanup a connection to be closed isnt requested ?
Pseudo algorithm
cleanup(){
for(Database db in map){
if(db.getLastAccess - now >24hours) {
res=map.remove("key",db);
db.close();
}
}
}
i know that the above isnt thread safe..How could i block getconnection ? Because many things could go wrong... "If condition" may be true but before removing db obj getLastAccess could be changed! Cleanup would be called by single thread though..
Is there any solution to block getconnection somehow so cleanup to work or anyother solution?
I am not sure if you currently do this, but if you have a way to determine if a connection is in use this would make this slightly easier. One thing that you can do, is iterate over the connections in your pool. When you find one that matches your criteria for being closed, try to mark it as being in use (assuming that a connection that is in use will not be returned as a open connection). If you succeed, close it. Otherwise, check it until it becomes free and you are able to mark it as being in use. Once you have been able to do this, you should be able to close it.
Each connection would have a lock associated with it, in order for the connection to be returned by the getConnection method, the correct lock would have to be acquired. The cleanup method would also need to acquire the lock before closing a connection. Take a look at the java.util.concurrent.lock package.
Maybe a Semaphore is a better solution. Follow the link for an example.
I've never worked with BerkeleyDB, but I assume it has a JDBC interface. Can't you use an out of the box solution like DBCP or c3p0? Also check the Pool Component, it is a generic pool interface.
What is the fastest option to issue stored procedures in a threaded environment in Java? According to http://dev.mysql.com/doc/refman/5.1/en/connector-j-usagenotes-basic.html#connector-j-examples-preparecall Connection.prepareCall() is an expensive method. So what's the alternative to calling it in every thread, when synchronized access to a single CallableStatement is not an option?
The most JDBC drivers use only a single socket per connection. I think MySQL also use also a single socket. That it is a bad performance idea to share one connection between multiple threads.
If you use multiple connection between different threads then you need a CallableStatment for every connection. You need a CallabaleStatement pool for every connection. The simplest to pool it in this case is to wrap the connection class and delegate all calls to the original class. This can be create very fast with Eclipse. In the wrapped method prepareCall() you can add a simple pool. You need also a wrapped class of the CallableStatement. The close method return the CallableStatement to the pool.
But first you should check if the call is real expensive because many driver has already such poll inside. Create a loop of prepareCall() and close() and count the time.
Connection is not thread safe, so you can't share it across threads.
When you prepareCall, the JDBC driver (may) be telling the RDBMS system to do a lot of work that is stored on the server side. You may be guilty of premature optimization here.
After giving this a little thought it seems that if you are having issues with this infrastructure code then your problems are elsewhere. Most applications do not take an inordinate amount of time doing this stuff.
Make sure you are using a DataSource, most do connection caching and some even do caching of statements.
Also for this to be a performance bottle neck it would imply that you are doing many queries one after the other, or that your pool of connections is too small. Maybe you should do some benchmarking on your code to see how much time the stored proc is taking vs how much time the JDBC code is taking.
Of course I would follow the MySQL recommendation of using CallableStatement, I am sure they have benchmarked this. Most apps do not share anything between Threads and it is rarely an issue.