I develop an web application using Spring MVC + MySQL. To manage JDBC connection I used c3p0 for connection pooling.
If I am using c3p0, What I suppose if, 5 connections are open in pool and all 5 connections are in sleep mode and if I request getConnection() from java then one of them should be returned as Connection. Am I correct?
If Yes Then consider following load testing scenario where I am getting Exception Connections could not be acquired from the underlying database!. Using Jmeter I started load test for 250 users in 15 seconds. I m continuously observing DB connections using SHOE FULL PROCESSLIST for first some minutes all is going well but as all the 250 users thrown to web server connections count is reached to 500 which is our maxPoolSize. So after this we are getting Connections could not be acquired from the underlying database!.
At this point if I execute SHOE FULL PROCESSLIST then I can see all 500 connections are in sleep mode If I am correct in above statement that any open connection that is in sleep mode will be returned by c3p0. Then why I am getting this exception.?
Here is my c3p0 properties
MINPOOLSIZE=10
ACQUIREINCREMENT=1
MAXPOOLSIZE=500
INITIALPOOLSIZE=10
NUMBERHELPERTHREAD=100
MAXIDLETIME=10
MAXSTATEMENT=20
MAXSTATEMENTPERCONNECTION=5
IDLECONNECTIONTESTPERIOD=120
ACQUIRERETRYATTEMPT=10
ACQUIRERETRYDELAY=100
AUTOCOMMITONCLOSE=false
BREAKAFTERACQUIREFAILURE=false
TESTCONNECTIONONCHECKOUT=true
TESTCONNECTIONONCHECKIN=true
Update
I found this warning before Connections could not be acquired from the underlying database! Exception
WARN : 30 Jun 2014 10:40:12.078 com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1839) - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#e49561 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (10). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too many connections
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:927)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1709)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1252)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2483)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2516)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2301)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:183)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:172)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:188)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1074)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1061)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1798)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:636)
WARN : 30 Jun 2014 10:40:12.078 com.mchange.v2.resourcepool.BasicResourcePool.forceKillAcquires(BasicResourcePool.java:882) - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool#1035ff9 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
Related
Facing one strange issue, we are connecting from oracle database to Oracle Essbase Server using JAPI connection (its http connection and no jdbc).
We are able to execute the program till 300 seconds then we get below error: -
invokeMethod localException11111 ::: java.net.SocketTimeoutException: recv() timed out
Resource temporarily unavailable
java.net.SocketTimeoutException: recv() timed out
Resource temporarily unavailable
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:740)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:683)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1280)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at com.essbase.api.session.EssOrbPluginHttp.invokeMethod(EssOrbPluginHttp.java:139)
at com.essbase.api.session.EssOrbPluginHttp.invokeMethod(EssOrbPluginHttp.java:109)
at com.essbase.api.session.EssOrbPlugin.essMainCalc(Unknown Source)
at com.essbase.api.datasource.EssCube.calcWithRunTimeSubVars(Unknown Source)
at com.essbase.api.datasource.EssCube.calculate(Unknown Source)
at com.ing.mass.essconnect.Database.calculate(Database.java:314)
at com.ing.mass.essconnect.Database.calculate(Database.java:300)
at com.ing.mass.services.Cubes.calculateNow(Cubes.java:1704)
at com.ing.mass.services.Cubes.calculate(Cubes.java:1510)
invokeMethod finally :::
TESTING .....com.essbase.api.base.EssException: Cannot connect to Provider Server. java.net.SocketTimeoutException: recv() timed out
Resource temporarily unavailable
at com.essbase.api.session.EssOrbPl uginHttp.invokeMethod(EssOrbPluginHttp.java:244)
at com.essbase.api.session.EssOrbPluginHttp.invokeMethod(EssOrbPluginHttp.java:109)
at com.essbase.api.session.EssOrbPlugin.essMainCalc(Unknown Source)
at com.essbase.api.datasource.EssCube.calcWithRunTimeSubVars(Unknown Source)
at com.essbase.api.datasource.EssCube.calculate(Unknown Source)
at com.ing.mass.essconnect.Database.calculate(Database.java:314)
at com.ing.mass.essconnect.Database.calculate(Database.java:300)
at com.ing.mass.services.Cubes.calculateNow(Cubes.java:1704)
at com.ing.mass.services.Cubes.calculate(Cubes.java:1510)
Calculation completed
Calculation completed 111
Cubes calculateNow method finallly
It has been a week scratching our heads to resolve this issue but no luck so far.
Looking forward to hear back with your expert advise.
thanks in advance
Pankaj
this issue is now resolved -
problem was with new java code which required to add a timeout parameter like this
sun.net.client.defaultReadTimeout (set in miliseconds)
I'm using HIkariCP, Hibernate and MySQL.
The problem is: When application starts, everything works fine. But when I leave it idle for some time and then get back and try to get connection from Hikari, it throws exception:
Caused by: java.sql.SQLTransientConnectionException: Main DB Pool - Connection is not available, request timed out after 60072ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:115)
... 59 common frames omitted
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 820 887 milliseconds ago. The last packet sent successfully to the server was 60 060 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3459)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3900)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2486)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2444)
at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:157)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:169)
... 64 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:101)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:144)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:174)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3469)
... 74 common frames omitted
MySQL database is located in external server and it has some timeouts/session expiration features and I cannot change it. I suppose the problem is caused exactly by some kind of session expiration due to too big inactivity time (because problem happens only after application remains idle for some time). It may by similar problem, I don't know what is the exact reason why connection is lost.
And the question is: how to force Hikari to refresh the connection in such situation instead of trowing exceptions? (or where else the problem may be?)
My config:
HikariConfig config = new HikariConfig();
config.setPoolName("Main DB Pool");
config.addDataSourceProperty("url", "jdbc:mysql://" + host);
config.setUsername(user);
config.setPassword(password);
config.setDataSourceClassName(dataSourceClass);
config.setMaximumPoolSize(maxPoolSize);
config.setIdleTimeout(30000);
config.setConnectionTimeout(60000);
config.setValidationTimeout(60000);
config.setConnectionInitSql("SELECT 1");
config.setConnectionTestQuery("SELECT 1");
config.setLeakDetectionThreshold(120000);
return new HikariDataSource(config);
Right from the docs. You will need to figure out what the external servers timeouts are and configure HikariCP's to be a little shorter.
⌚maxLifetime
This property controls the maximum lifetime of a connection in the pool. An in-use connection will never be retired, only when it is closed will it then be removed. We strongly recommend setting this value, and it should be at least 30 seconds less than any database or infrastructure imposed connection time limit. A value of 0 indicates no maximum lifetime (infinite lifetime), subject of course to the idleTimeout setting. Default: 1800000 (30 minutes)
I am Working on Multinode Cluster with four slaves node named as slave01,slave02,slave03 and slave04 and one master node as master
when i remove out the network cable during map task hadoop wait for status update for 100 seconds (due to property of whose value is 100000)
after that i can see that maptask get failed and hadoop start container cleanup which takes more than 10 minutes and it also doesn't schedule failed task anywhere.the i get error of no Route to host exception from application master to lost node.After which task get schedule on another node.
i want to reduce the time for trying container cleanup so that task can be schedule just after timeout of maptask on any node.
please help me that how can i do that by setting configuration.
I am attaching application master log in which i have remove slave01 during map task,in this case no of reduce task running is 1.
AttemptID:attempt_1463201584280_0004_m_000002_0 Timed out after 100 secs Container released on a lost node cleanup failed for container container_1463201584280_0004_01_000004 : java.net.NoRouteToHostException: No Route to Host from slave02/172.31.132.107 to slave01:58838 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost at sun.reflect.GeneratedConstructorAccessor51.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:757) at org.apache.hadoop.ipc.Client.call(Client.java:1473) at org.apache.hadoop.ipc.Client.call(Client.java:1400) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy37.stopContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.stopContainers(ContainerManagementProtocolPBClientImpl.java:110) at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy38.stopContainers(Unknown Source) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:206) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:608) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:706) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:369) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522) at org.apache.hadoop.ipc.Client.call(Client.java:1439) ... 15 more
This happened because of an bug in hadoop 2.6.3 in which connection retry in done at two levels ipc and yarn try to use 2.6.4 or download patch it will get resolved.
I recently got the following messages in our logs followed by a JVM crash (Due to OOME). I am not sure what to make of this and would really appreciate any guidance.
2015-03-19 21:15:02,457 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:26:29,543 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#15da1b6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#b35b08a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#51cfdd17 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#19397937
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#5c7d3838
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#7aea62dd
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#55622ff2
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#74004a8
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.net.SocketOutputStream.socketWrite0(Native Method)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
java.net.SocketOutputStream.write(SocketOutputStream.java:153)
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1751)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3425)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2196)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:718)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor306.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:302)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:282)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
2015-03-19 21:56:59,137 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:56:59,143 [eXistThread-18676] ERROR (XQueryServlet.java [process]:566) - Java heap space
java.lang.OutOfMemoryError: Java heap space
at org.exist.storage.btree.BTree$BTreeNode.read(BTree.java:1269)
at org.exist.storage.btree.BTree$BTreeNode.access$16(BTree.java:1239)
at org.exist.storage.btree.BTree.getBTreeNode(BTree.java:460)
at org.exist.storage.btree.BTree.scanSequential(BTree.java:413)
at org.exist.storage.btree.BTree$BTreeNode.scanNextPage(BTree.java:2039)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1835)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.access$12(BTree.java:1734)
at org.exist.storage.btree.BTree.query(BTree.java:379)
at org.exist.storage.structural.NativeStructuralIndexWorker.scanByType(NativeStructuralIndexWorker.java:259)
at org.exist.dom.VirtualNodeSet.getNodesFromIndex(VirtualNodeSet.java:457)
at org.exist.dom.VirtualNodeSet.realize(VirtualNodeSet.java:585)
at org.exist.dom.VirtualNodeSet.iterator(VirtualNodeSet.java:860)
at org.exist.dom.AbstractNodeSet.iterator(AbstractNodeSet.java:1)
at org.exist.storage.structural.NativeStructuralIndexWorker.findDescendantsByTagName(NativeStructuralIndexWorker.java:162)
at org.exist.xquery.LocationStep.getAttributes(LocationStep.java:645)
at org.exist.xquery.LocationStep.eval(LocationStep.java:434)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.Predicate.selectByNodeSet(Predicate.java:446)
at org.exist.xquery.Predicate.evalPredicate(Predicate.java:326)
at org.exist.xquery.LocationStep.processPredicate(LocationStep.java:251)
at org.exist.xquery.LocationStep.applyPredicate(LocationStep.java:238)
at org.exist.xquery.LocationStep.eval(LocationStep.java:462)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.LetExpr.eval(LetExpr.java:142)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.BindingExpression.eval(BindingExpression.java:164)
2015-03-19 21:56:59,147 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#79180a12 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#243c6d0c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#50191373 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#3a9d08ca
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#3ecdd11
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#44ff846d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#5ce5850a
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#eec1d04
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#6b8d4d9d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#53e9706d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#23d472cf
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#4dbe4f8c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#4c5e0203
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#54ac79fd
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#546e2bad
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#6b13cc83
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#57e185f8
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#60357d68
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#45231180
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#3021aa73
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#6bb437ca
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#2021c9e9
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#7d53637c
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#409c2c97
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#adc5929
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#241ca71a
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#42b26866
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#636b1c33
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#b160466
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#4af34669
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1b53e609
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#2062ebd4
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1b6cfe8a
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#4b7c2380
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#4f9be748
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#78108924
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#474b002
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#2ebee32f
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#3e0fe017
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#42aa175b
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#637f5bac
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#3a017b77
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#7b4f2b78
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.java:370)
com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.java:2532)
com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477)
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2596)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
com.newrelic.agent.TransactionService.addTransaction(TransactionService.java:142)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1104)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1087)
com.newrelic.agent.TracerService$TracerServiceImpl.getTracer(TracerService.java:136)
com.newrelic.agent.TracerService.getTracer(TracerService.java:41)
com.newrelic.agent.instrumentation.InvocationPoint.invoke(InvocationPoint.java:55)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
We are using hiberanate and c3p0 with the following c3p0 configuration:
We are using the following maven artifacts for hibernate and c3p0:
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.3.6.Final</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-c3p0</artifactId>
<version>4.3.6.Final</version>
</dependency>
with the following c3p0 configuration:
configuration = new Configuration().setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect")
.setProperty("hibernate.connection.provider_class", "org.hibernate.connection.C3P0ConnectionProvider")
.setProperty("hibernate.c3p0.idle_test_period", "1000")
.setProperty("hibernate.c3p0.min_size", "20")
.setProperty("hibernate.c3p0.max_size", "50")
.setProperty("hibernate.c3p0.timeout", "1800")
.setProperty("hibernate.c3p0.max_statements", "50")
The server is under very light load ~5 queries a second. (Java 8)
I stumbled into the same exception and the reason was a wrong password of the db user...
c3p0 you're a funny guy
So, directly the issue is that the Connection pool was trying to acquire new Connections, but the tasks attempting to acquire those tasks froze for a long period of time, so long that c3p0 decided the tasks must be deadlocked and then discarded and replaced the Thread pool. Later, the Thread pool was hung on idle connection test tasks.
Normally, "hung" tasks tend to look like the second Thread underneath the first "Pool thread stack traces:" label above: performing network IO that fails to complete. Your circumstances are odd in that two of the three threads are not stuck in IO. They have barely begun to do anything, yet they aren't live. Then you experience an OutOfMemoryError, and you get another APPARENT DEADLOCK on idle connection test tasks that also look like they mostly should be live.
Maybe your application is very close to some kind of resource limits that are causing things to run very sluggishly? Straightforwardly, you might increase the amount of memory available to this application (or modify it to have a lower memory footprint). You experience an OOME the second time the Thread pool tries to flush and recreate hung Threads, not directly provoked by that, but quite possibly caused by the growing Thread footprint. (In your logs, are there lots of these APPARENT DEADLOCKs previously? If you force a JVM Thread dump, do you see lots of com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread instances around, still hung on old, not completing tasks?
Some general comments: If your load is no more than 5-ish simultaneous queries, why such a heavy pool? why not min_size 5-ish, max 10-ish? Your max_statements setting is way too small for the number of Connections you permit in the pool. I'd omit this entirely until you get things working smoothly. Then, to get a bit better performance, you might set the more-straightforward-to-reason-about maxStatementsPerConnection parameter if you want.
Mostly you need to keep your application's footprint (memory? Threads?) well below the resources alotted to it, either by increasing resources, reducing its footprint, or fixing any issue that might exist that cause its resource footprint to increase into limits. I'd start by making the pool smaller, the available memory larger, and configuring the pool to be much smaller.
Authentication issue for me too. I just added the domain part to the database server name and it all worked. Very misleading error.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Connection Pool Exception: Cannot get a connection, pool error Timeout waiting for idle object
I am getting Cannot get a connection, pool error Timeout waiting for idle object error, When I try to create more than 250 threads in my web application. I am creating web application using JSF 2.0 and Hibernate.
I have tried with modified hibernate.xml,server.xml,context.xml and also mysql properties.
The followings are I am getting.
WARN (JDBCExceptionReporter.java:233) - SQL Error: 0, SQLState: null
ERROR (JDBCExceptionReporter.java:234) - Cannot get a connection, pool error Timeout waiting for idle object
ERROR (BaseServlet.java:301) - ******** java.lang.Thread.getStackTrace(Thread.java:1426)
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source)
Caused by: org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source)
Please if have any idea about this help me......
Maybe you have set a timeout for "dead" connections and some queries take longer than that. That means that your pool removed a busy connection as "dead" from the pool and requests another from the DB - until the DB pulls the plug.
To debug this, enable logging for your connection pool, so you can see when it requests new connections.
also check your mysql connection settings. and try to close connection when you are done with your db coz next time(beyond maxConnectionAge limit) that connection state will be dead .
I had such issue before, what you need to do is
close hibernate sessions when you are done with it.
e.g.
Session sess = factory.openSession();
Transaction tx;
try {
tx = sess.beginTransaction();
//do some work
...
tx.commit();
}
catch (Exception e) {
if (tx!=null) tx.rollback();
throw e;
}
finally {
sess.close(); // closing session
}