In my standalone app, generating over 1000 threads in single time and each thread has it's own Hibernate session. But in this case, count of sessions hits over database max connections restriction that throws an error.
I've tried to set .getCurrentSession() instead of .openSession(), but it brought no effect, because Hibernate opens new own session for each new thread anyway.
How can i round this problem? Can i set somehow the count of slots for concurrent connections? For e.g. pass 100 connections and let another 900 to wait till these 100 will be closed to process this further?
Using a connection pool can help solve this.
Here is a post about setting up a connection pool Can you only have one hibernate session per thread in java?
Most people utilize a connection pool, like C3P0 that can be used to
ensure session reuse and speed up your code.
The best structure I use is to create a SessionFactory once in
application launch, as this sets up the connection pool to the
database. Then, utilizing maybe a singleton pattern to keep a single
SessionFactory, request new sessions for each transaction you perform
from the single SessionFactory. Hibernate will utilize the underlying
connection pool to handle session reuse for speed and optimization.
Here is another post on various connection pooling libraries available.
By default, Hibernate ships with the ability to obtain a data source
implementation ( javax.sql.DataSource ) from JNDI by setting the
properties appropriately
hibernate default connection pooling
You can Use 'ExecutorService' to manage the 1000 threads, limit to make only 100 threads running and the others will stay in the queue.
However, I think it's not good to have so many threads in you app. I assuming you are working on a web service app, and may have 1000 requests coming in at the same time, and the old school solution is to create 1000 threads to handle that. But epoll will just be better, you can consider about some NIO framework such as MINA/Netty.
Related
I have a basic requirement :
1) Need a hibernate session manager which is called each time I open and close or do anything else using session.
2) It should work as a wrapper around the hibernate session.
3) For example : I create a session utility , which will allow at a time creating 5 sessions only , and if 5 of them are already in used , my request has to wait till I get one of the 5 session back.
There really isn't a need to write a special Session Manager to do this as you can exploit a connection pool to easily handle this behavior for you.
What you first need to do is define your connection pool with a maximum number of available connections, which in your case would be 5. Then you want to configure the connection pool so that it has a wait timeout with something reasonable. Just understand that this timeout can be problematic in a denial of service attack scenario, so use at your own risk.
With the connection pool properly configured, if the first 5 are doing some long-running task that takes several seconds and the 6th request comes in, it will wait to acquire a connection with the database from the pool before it continues, so you'll only ever have 5 concurrent connections happening with your database at any point.
I have a scenario in production for a web app, where when a form is submitted the data gets stored in 3 tables in Oracle DB through JDBC. Sometimes I am seeing connection time out errors in logs while the app is trying to connect to Oracle DB through Java code. This is intermittent.
Below is the exception:
SQL exception while storing data in table
java.sql.SQLRecoverableException: IO Error: Connection timed out
Most of the times the web app is able to connect to data base and insert values in it but some times and I am getting this time out error and unable to insert data in it. I am not sure why am I getting this intermittent issue. When I checked the connections pool config in my application, I noticed the following things:
Pool Size (Maximum number of Connections that this pool can open) : 10
Pool wait (Maximum wait time, in milliseconds, before throwing an Exception if all pooled Connections are in use) : 1000
Since the pool size is just 10 and if there are multiple users trying to connect to data base will this connection time out issue occur ?
Also since there are 3 tables where the data insertion occurs we are doing the whole insertion in just one connection itself. We are not opneing each DB connection for each individual table.
NOTE: This application is deployed on AEM (Content Management system) server and connections pool config is provided by them.
Update: I tried setting the validation query in the connections pool but still I am getting the connection time out error. I am not sure whether the connections pool has checked the validation query or not. I have attached the connections pool above for reference.
I would try two things:
Try setting a validation query so each time the pool leases a connection, you're sure it's actually available. select 1 from dual should work. On recent JDBC drivers that should not be required but you might give it a go.
Estimate the concurrency of your form. A 10 connections pool is not too small depending on the complexity of your work on DB. It seems you're saving a form so it should not be THAT complex. How many users per day do you expect? Then, on peak time, how many users do you expect to be using the form at the same time? A 10 connections pool often leases and retrieves connections quite fast so it can handle several transactions per second. If you expect more, increase the size slightly (more than 25-30 actually degrades DB performance as more queries compete for resources there).
If nothing seems to work, it would be good to check what's happening on your DB. If possible, use Enterprise Manager to see if there are latches while doing stuff on those three tables.
I give this answer from programming point of view. There are multiple possibilities for this problem. These are following and i have added appropriate solution for it. As connection timeout occurs, means your new thread do not getting database access within mentioned time and it is due to:
Possibility I: Not closing connection, there should be connection leakage somewhere in your application Solution
You need to ensure this thing and need to check for this leakage and close the connection after use.
Possibility II: Big Transaction Solution
i. Is these insertion synchronized, if it is so then use it very carefully. Use it at block level not method level. And your synchronized block size should be minimum as much as possible.
What happen is if we have big synchronized block, we give connection, but it will be in waiting state as this synchronized block needs too much time for execution. so other thread waiting time increases. Suppose we have 100 users, each have 100 threads for that operation. 1st is executing and it takes too long time. and others are waiting. So there may be a case where 80th 90th,etc thread throw timeout. And For some thread this issue occurs.
So you must need to reduce size of the synchronized block.
ii. And also for this case also check If the transaction is big, then try to cut the transaction into smaller ones if possible:-
For an example here, for one insertion one small transaction. for second other small transaction, like this. And these three small transaction completes operation.
Possibility III: Pool size is not enough if usability of application is too high Solution
Need to increase the pool size. (It is applicable if you properly closes all the connection after use)
You can use Java Executor service in this case .One thread One connection , all asynchronous .Once transaction completed , release the connection back to pool.That way , you can get rid of this timeout issue.
If one connection is inserting the data in 3 tables and other threads trying to make connection are waiting, timeout is bound to happen.
I use thousands of H2 databases via TCP using Hikari Connection Pools. In a period of 1-30min a lot of queries will be performed on about five of the databases. There will be some queries to some of the other databases too but it is not predictable which and how many databases are affected by those side queries.
I store the already used HikariDataSources in a HashMap for the subsequent queries but I fear that the HashMap (contained objects) is getting too large and therefore I want to create a cleanup thread that closes HikariDataSources and removes them from the HashMap after they weren't used for a certain period of time.
To remove the right connection pools I need to know if a pool was not used for a defined period of time. How do I get that information?
And is there a better way to handle the number of Connection Pools? Maybe there is something that is made for Connection Pools similar as HikariCP is made for Connections. Is there a Pool for Connection Pools? :D
Why should I prefer using a connection pool instead of static variable from a static class in Tomcat to hold the database connection?
This seems to me equivalent of using a connection pool having the capacity to store just one connection. So, a related question is: why the capacity of a connection pool needs to be bigger than one connection?
Thank's in advance.
With a pool, you can have multiple threads using different connections. Do you really want to limit your web application to handling one db-related request at a time? :) (And adding the complication of synchronizing to make sure that one thread doesn't try to use that single connection while another request is doing so...)
It would be generally a very bad idea to have a connection pool with a capacity of 1 - but at least if you did so, you could later increase the capacity without changing any other code, of course.
(EDIT: And as noted in other answers, connections in a pool can be closed and reopened if they become stale or broken in some way.)
The reason is to increase scalability, robustness and speed.
If you're creating a web application, there can be many concurrent HTTP requests coming in, each served by a different thread.
If you have only one static connection to the database, you need synchronization around that connection. You can't share a connection between several threads, That means each HTTP request have to wait for someone else using the database. And you need to fix up/reconnect that connection if something goes wrong with it at one point or another.
You could open a connection at the start of each HTTP request - however opening a new database connection can be expensive, and you don't get much control over how many database connections you have. Databases can be overwhelmed by having too many connections.
A connection pool solves this, as you have a pool of already opened connections that can be handed out to serve an HTTP request, or to different parts of the applications that needs to do database operations, and is returned to the pool when the database operation is finished, ready to be used again by something else.
A connection pool of just 1 connection rarely makes sense though - however connection pools take care of many other things as well, such a closing the connection and opening a new one if a connection goes stale or is otherwise in a bad state, as well as it takes care of the synchronization when there is are no more connections to hand out at a particular time.
If you're using a connection pool with only one connection it is equivalent to have one static connection - like you mentioned and there's no advantage for connection pool in regards.
The strength of a connection pool is when you're using multiple connections (multiple threads) - it saves you the effort of managing the connections (open/close connections, boilerplate-code, smart resource handling etc).
Using a connection pool for one connection only is kind of like paving a 10-lane road that will be used by one car only - lot of overhead with (almost) no gain.
Using a connection pool is not just about sharing connections: it is about leveraging years of experience with broken JDBC drivers and all the weird ways in which a Connection can become unusable. Using a single static connection is not only a bottleneck, but a very fragile solution. It will cause your application to break on a regular basis, and even restarting the application will not clean up the problems: you may get a connection leak.
Let's say we've got a web application or web service on an application server supporting JDBC connection pooling.
Should I be grabbing a new Connection on a per-thread or per-query basis?
Thanks!
Hopefully you are grabbing them on a per-transactional-unit-of-work basis.
Per query implies that you never have any logical unit of work in your system that spans more than a single query. (Maybe that's true, but you still might want to think about the future!)
Per-thread (which I assume to mean request-scoped, rather than for the entire life of the thread?) will probably result in holding them for longer than absolutely necessary, but it does allow you to manage transactions much better. (and it's how plenty of leading frameworks have worked or did work for a long time. A pattern known as Open Entity Manager In View, if you'd like to do some google-fu on it)
Assigning it indefinitely to a single thread means your max number of active request processors is capped at the max size of your database pool, which is a definite failure in scalability.
per-thread
Each new request will grab a new connection (new thread = new request). There is no need for getting a new connection for each query as after each query the connection can be reused.