I am working on mongodb connection pooling & I came across this option we can set in mongo client : "MaxConnectionIdleTime".
It basically means that connection will die after this time when sitting idle.
The default value is zero & it's stated that in case of zero, there's no limit.
Does it mean that once a connection has been created it won't die at all & will be kept in pool forever ?
Assuming there's space for new connections to be created in the pool. like min connections=10 & max connections =1000. & also, the max connection time isn't set.
If you can suggest a method to test it out on my own, that'll be really helpful too.
Please let me know if there's any way I can improve the question.
Thanks!
"Die" is a sloppy term and it is not clear what you mean by it.
If no idle time is set, the connection will not be proactively closed by the driver upon that idle time elapsing. That's all.
Connections may be closed by the driver in the following other circumstances:
It experiences a network error or a timeout.
Another connection to the server that this connection is associated with experiences a network error, causing the server to be marked unusable by the driver.
A connection may become unusable because of a network error. The connection may be unusable without the driver knowing about it. The driver often detects unusability of a connection when trying to use it (i.e. write or read something) which means a connection may be unusable (one might say dead) while the driver thinks it is perfectly fine, for a long time.
Related
I will try to explain what I want to know.
I have a Java web app that makes connections to a MySQL database.
If I execute SHOW PROCESSLIST in MySQL I have rows like this:
id: xxx
User: xxx
Host: XXX
db: XXX
Command: Sleep
Time: 12352
State:
Info: NULL
I understand that each process is an open connection to the database.
To manage the connections I use a pool like this
http://www.chuidiang.com/java/mysql/BasicDataSource-Pool-Conexiones.php
(It's in Spanish but I think you see the idea)
So to "open" the connection I make something like this:
DataSource ds = ...;
con = ds.getConnection();
Internally, my class is not opening the connection; it's only getting a connection already opened by the pool.
The question is:
Can I know which class is holding (or has made con = ds.getConnection()) a particular connection shown in SHOW PROCESSLIST?
MySQL doesn't know, so it can't tell you.
Your Java code could tell you, by running the query SELECT CONNECTION_ID() AS connection_id and logging the output. This will be the same ID shown in SHOW [FULL] PROCESSLIST:.
Note, of course, that this doesn't mean the class still holds the connection, only that it obtained it. Sleeping threads are idle and if you're pooling, nobody may be holding them. Whoever held the connection last may have already released it to the pool.
Also, remember, that sleeping threads are sleeping, so they generally aren't hurting anything, sitting there sleeping like that. The time is time since last used for a query.
You may also find that, in the presence of a a stateful firewall between MySQL and the application, a connection is idled out of the firewall's memory due to excessive idleness (sometimes with crazy low times like 15 minutes). The application tends to initiate most interactions with MySQL, so the application will figure out that the connection is useless (thanks to the firewall), abandon it, and get a new one when it la attempts to communicate with the server after such an idle period are rebuffed by the firewall, but the server will sit happily waiting for requests that never come. These connections are wasting server resources when this happens. Enabling TCP keepalives in the kernels of the servers will mitigate this, if it's happening. The firewall could be part of your infrastructure that you don't realize is doing this.
So I have a Java process that runs indefinitely as a TCP server (receives messages from another process, and has onMsg handlers).
One of the things I want to do with the message in my Java program is to write it to disk using a database connection to postgres. Right now, I have one single static connection object which I call every time a message comes in. I do NOT close and reopen the connection for each message.
I am still a bit new to Java, I wanted to know 1) whether there are any pitfalls or dangers with using one connection object open indefinitely, and 2) Are there performance benefits to never closing the connection, as opposed to reopening/closing every time I want to hit the database?
Thanks for the help!
I do NOT close and reopen the connection for each message.
Yes you do... at least as far as the plain Connection object is concerned. Otherwise, if you ever end up with a broken connection, it'll be broken forever, and if you ever need to perform multiple operations concurrently, you'll have problems.
What you want is a connection pool to manage the "real" connections to the database, and you just ask for a connection from the pool for each operation and close it when you're done with it. Closing the "logical" connection just returns the "real" connection to the pool for another operation. (The pool can handle keeping the connection alive with a heartbeat, retiring connections over time etc.)
There are lots of connection pool technologies available, and it's been a long time since I've used "plain" JDBC so I wouldn't like to say where the state of the art is at the moment - but that's research you can do for yourself :)
Creating a database connection is always a performance hit. Only a very naive implementation would create and close a connection for each operation. If you only needed to do something once an hour, then it would be acceptable.
However, if you have a program that performs several database accesses per minute (or even per second for larger apps), you don't want to actually close the connection.
So when do you close the connection? Easy answer: let a connection pool handle that for you. You ask the pool for a connection, it'll give you an open connection (that it either has cached, or if it really needs to, a brand new connection). When you're finished with your queries, you close() the connection, but it actually just returns the connection to the pool.
For very simple programs setting up a connection pool might be extra work, but it's not very difficult and definitely something you'll want to get the hang of. There are several open source connection pools, such as DBCP from Apache and 3CPO.
We have seen connection droughts in our system every once in a while, and the problem seems to be that Sessions are not being returned to the connection pool quick enough. I wrote a test that seems to confirm using Session.disconnect() on the sessions (after being done with one) will solve this problem. However, I also timed these calls, and it seems like using disconnect is increasing running time by 3 times.
According to the docs (http://docs.jboss.org/hibernate/core/3.5/api/org/hibernate/Session.html#disconnect() ), disconnect should be returning it to the connection pool. However, the doc also says it "closes" the connection. I'm not sure what it means because I know for a fact that Session.close() does more than disconnect, and what good would a connection pool be if you close the connection before returning it?
In any case, I'm wondering why a method that returns the session to the connection pool would be anything but instantaneous and essentially free. Surely thats the whole point of a connection pool, right?
Any ideas would be appreciated.
I am using a connection pool in my project. I want to know how many connections are opened?
I have given 20 connections as max no of connections. What will happen if it exceeds 20? Will the pool manage this or it will through error?
Its more of a configuration related problem. But in general the pool will throw exception when the new request for a connection results in the number of connection exceeding the max connection setting.
Must depend on the particular connection pool implementation.
Conceptually it would be possible to wait, fail or victimise.
Waiting indefinitely is potentially a very bad thing, in a badly configured system with antisocial clients keeping connections for too long the number of waiters could grow very large.
Arbitrarily victimising some existing client and taking their connection is usually not reasonable (or indeed possible) for DB connections. So that's not likely to be seen.
Which leaves a failure - you'll get an exception on the lines of "No Connections Available". Some connection pools actually wait a while (for a configurable period of time) before throwing that exception - this deals better with cases of occasional peaks in demand.
To know the number of connections opened in Oracle you can write a query like this.
SELECT s.program, s.server, p.spid, s.username FROM v$session s, v$process p
WHERE s.type = 'USER'
and s.username != 'sys';
The query assumes that your connected to Oracle in a dedicated server mode also the query counts connections made other than the user sys.
Then you can use the result in your application. And the second question, "What will happen if it exceeds 20?" depends on your implementation of connection pool and I won't say more since it has been answered.
Hope this is helpful.
I not very good in java.
I have made a website for a client but am continuously getting an error like Server connection failure during transaction. Due to underlying exception: 'com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: User root already has more than 'max_user_connections' active connections'.
The website hosting that I am using provides only 10 max_user_connections. But if I continuously use that site, I get this error because of continuously hits on the webserver.
What can be the reason behind this?
Am I not closing the connections right?
I have closed all the connections using con.close().
Please help
Regards Apurv
To open the connection I have used
Connection con=null;
Statement st=null;
Class.forName("com.mysql.jdbc.Driver").newInstance();
String useq="root";
String paq="manager";
String url="jdbc:mysql://localhost:3306/jayna?autoReconnect=true";
con=DriverManager.getConnection(url,useq,paq);
st=con.createStatement();
To close the connection I have used
if(rs!=null){
rs.close();
}
if(st!=null){
st.close();
}
if(con!=null){
con.close();
}
I haven't used a database pool but what can be the use of that when I am closing each of the connection properly??
This seems like a case of connection leak.
Are you sure, you have closed all the connections?
Conneciton.close() throws IOException. Check if it is successfully able to close connections.
Use netstat or other tools to find whether connections are really closed or are in WAITING state or something.
I think you should increase number of connections. Because if there are 10 slow query and you get 11 requests to your site 11 request couldn't be processed and you get this error.
So:
Try increase a number of connections.
Try to find slow queries(using slow query log) and optimize them
Connection pooling is a technique to provide a set of ready-to-use connections, one of the advantages being that you save the creation/opening time on each call. Another advantage is that the connection pooler can help detect abandoned connections, ie connections that the application forgot to close.
There's a standard connection pool in Tomcat, for 5.x version Tomcats look here for some info, for version 6 look here and for version 7 info can be found here. Its removeAbandoned and logabandoned features can help you determine whether your app really forgets to close connections, or 10 just isn't enough - see Andrej's suggestion, you should profile your queries.
As others have said, you either are not getting connections closed or your application simply needs more than 10 concurrent connections under some load conditions.
If the error always occurs on the 11th request, it's likely you're never getting the connections closed.
If it occurs sometime later, unpredictably, and goes away on its own, it's more likely 10 simply isn't sufficient for certain load scenarios.
If it occurs later, unpredictably, but never goes away on its own, it's possible you're failing to close connections only in specific cases that aren't hit every time.
If 10 is too small for some load scenarios (option 2), you should both check your queries and code logic to ensure you're not holding connections way longer than necessary and you should probably try to move to a Connection Pool, as others have suggested. Among other things, creating new Connections from scratch has more overhead than reusing them from a pool, so that could be causing individual accesses to take much longer than necessary.
This problem can be solved by using a Singleton class structure for initializing connection objects.
Using the Singleton pattern, whenever a connection object is initialized, rather than creating a new object, it will look for existing instance of connection object and use that one, if it exists.