I'm using C3P0 as a connection pool in Tomcat, and I'm seeing very worrying errors:
2010-09-16 13:25:00,160 [Timer-0] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#43502400 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2010-09-16 13:25:01,407 [Timer-0] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#43502400 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 10
Active Threads: 0
Active Tasks:
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#6e4151a7
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6,5,main]
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5,5,main]
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4,5,main]
java.lang.Object.wait(Native Method)
... many more, exact same stack trace
Line 534 is:
while (true) {
Runnable myTask;
synchronized ( ThreadPoolAsynchronousRunner.this ) {
while ( !should_stop && pendingTasks.size() == 0 )
ThreadPoolAsynchronousRunner.this.wait( POLL_FOR_STOP_INTERVAL ); // <- here
if (should_stop) ...
It looks very much like all the threads are idle. They're waiting for work. 0 active threads, and only 1 task to complete. Any clues on what's going wrong?
Here's the configuration:
ds.setUser(userName);
ds.setPassword(password);
ds.setMaxPoolSize(16);
ds.setMaxConnectionAge(1800);
ds.setAcquireRetryAttempts(4);
ds.setMaxIdleTime(900);
ds.setNumHelperThreads(10);
ds.setCheckoutTimeout(1000);
I just experienced a similar issue against an Oracle database, but in my case Managed Thread and Active Thread counts were the same.
Managed Threads: 3
Active Threads: 3
For me it was actually an authentication error but appeared as the APPARENT DEADLOCK error because of the way I was doing login auditing.
2013-08-12 11:29:04,910 [Timer-4] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#34996454 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2013-08-12 11:29:04,914 [Timer-4] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#34996454 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#6730b844 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#2f91ad49 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#507ac05 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#3aae7ed7
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:150)
java.net.SocketInputStream.read(SocketInputStream.java:121)
oracle.net.ns.Packet.receive(Packet.java:300)
oracle.net.ns.DataPacket.receive(DataPacket.java:106)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:380)
oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:760)
oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:401)
oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:150)
java.net.SocketInputStream.read(SocketInputStream.java:121)
oracle.net.ns.Packet.receive(Packet.java:300)
oracle.net.ns.DataPacket.receive(DataPacket.java:106)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:380)
oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:760)
oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:401)
oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:150)
java.net.SocketInputStream.read(SocketInputStream.java:121)
oracle.net.ns.Packet.receive(Packet.java:300)
oracle.net.ns.DataPacket.receive(DataPacket.java:106)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:380)
oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:760)
oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:401)
oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
This sounds like you have already acquired a connection from the pool and do not return it in time.
C3P0 determines "apparent deadlocks" when a connection is acquired but not returned to the pool within the deadlock detection timeout.
If you move the connection acquisition closer to the "action" and immediately return it to the pool after the database work is done, this message will disappear.
This will sort out your problem
ds.setMaxStatements(1000);
ds.setMaxStatementsPerConnection(100); (the maximum number of prepared statments your system can execute on a single connection)
check out : https://forum.hibernate.org/viewtopic.php?t=947246&highlight=apparent+deadlock+c3p0
Remember to close your statements after you are done with them !!
My comment on answer https://stackoverflow.com/a/18192588/1019307 received enough up-votes to suggest it should be an answer.
I received this error because I couldn't get through the firewall to the database server. Check to see if that is your problem.
#eebbesen, I got the same error as you did. I am running Tomcat version 9.0.6. I have hibernate core ver 5.2.10, hibernate c3p0 ver 3.6.3 in my maven project. Mine was not an authentication error, but rather due to me having previously changed the name of my computer. This did not have an immediate effect on tomcat, but upon a restart of my machine, when I tried to bring up tomcat again via eclipse (Oxygen 2), I could no longer start eclipse due exactly to the issue you raised.
I googled this and I found this link, which tipped me off to the issue:
https://community.oracle.com/thread/339825
where it says:
First see if OracleServiceXE and OracleXETNSListener services are
running. Replace 127.0.0.1 in the url, with the IP or the name of your
machine. It must match the host declared in the tnsnames.ora file.
Later it mentions where to find this tnsnames.ora file, and for me it was here:
C:\oraclexe\app\oracle\product\11.2.0\server\network\ADMIN
Looking at this tnsnames.ora file, I saw this:
XE =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = MyMachineName-7)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = XE)
)
)
But I had recently renamed my machine to MyMachineName-5. I changed the 7 to a 5 and saved the file. I checked the "listener.ora" file in this directory, and it had the same issue:
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
(ADDRESS = (PROTOCOL = TCP)(HOST = MyMachineName-7)(PORT = 1521))
)
)
I changed the 7 to a 5 and saved the file.
Then I opened Task Manager, clicked the "Services" tab, and looked at the "Oracle" services. I did a restart on: OracleXETNSListener, OracleXEClrAgent, OracleServiceXE. I went to restart tomcat in eclipse again, and this time there was now problem.
Appendix:
I also googled this:
https://community.oracle.com/thread/2267906
This lead me to try:
1) Turned off the firewall in Windows Defender (already turned off mcaffee firewall)
2) Started sqlplus to make sure that I could log in with the credentials I was using in my hibernate file: hibernate.cfg.xml
C:\oraclexe\app\oracle\product\11.2.0\server\bin\sqlplus.exe
3) Started the desktop shortcut to Oracle Database 11g
This failed for me even after I fixed the machine name issue though, something I still have to look into.
4) Using dbVisualizer, I tried making a connection to Oracle. This only worked after I resolved the .ora file machine name issue: double click the connection and click the "ping server" button.
I've had the same problem but the cause was a bit hard to spot as it was caused by some simultaneous resources trying to acquire a connection at the same time.
As you can read if the pool had not been initialized the code provided to init it by calling a setup function.
public Connection getConnection() throws SQLException {
if (mCPDS == null) {
setupPool();
}
return mCPDS.getConnection();
}
The problem was that many resources were trying to acquire the connection at the beginning of the program so more than one were instantiating the pool causing your problem after a while.
The solution was just to declare the method synchronized to keep out other resources while one has called the method and it's still inside instantiating the pool for instance.
public synchronized Connection getConnection() throws SQLException {
if (mCPDS == null) {
setupPool();
}
return mCPDS.getConnection();
}
This may be a design error for not using a singleton but fixes the problem lacking some performance.
We ran into this problem and solved it by adding this to the C3P0 config:
<property name="statementCacheNumDeferredCloseThreads" value="1"/>
as per: this from the C3P0 doc
I had the same (was unable to detect) problem solved by correctly closing Statement and Resultset instances (somehow left unclosed):
String SQL = "SELECT 1";
try {
con = DriverManager.getConnection(host, userName, userPassword);
stmt = con.prepareStatement(SQL, ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE);
try {
rs = stmt.executeQuery(SQL);
try {
rs.next();
// ...
} finally {
rs.close();
}
} finally {
stmt.close();
}
} catch (SQLException ex) {
Logger.getLogger(MyClass.class.getName()).log(Level.SEVERE, null, ex);
}
Hi my friend just to comment, I had the same case. I just configured my spring-hibernate eclipse project and showed up the same exception, it should be noted that my project still does not have any query.
I resolved that issue with below steps:
1) Clean project : Project--> Clean...
2) Build project : Project--> Build Project
I hope it works for you.
I've just had the same problem suddenly: after noticing that the deadlock was present only when launching my application in debug mode (I'm using IntelliJ) and it was fine when running with normal run, I started to dig it.
I finally figured out that a breakpoint was blocking the connection: I don't know why Intellij didn't "listen" that the application was passing through that breakpoint, but was hanged somewhere cause of a breakpoint, and this was causing appartent deadlock
After removing all breakpoints in my project, everything started smooth again.
Hope this helps someone
Similar problem was encountered on glassfish4 server while deploying the application. Turned out it was a database configuration issue. Just make sure your database connectivity configurations are proper, verify that the hostname provided in the configuration allows the connection to database. Try connecting to the database manually with the configured username and the hostname/domain. If required, allow the db user to connect from the required domain. Rebuild the application with correct db configurations and then deploy it.
Related
Problem Description: MongoDB version is 3.4
In fact, did not do anything on the normal query, write,
because it is in the testing phase, QPS is small.
Question:
1: How is this anomaly produced.
2: what configuration or adjustment needs to be done? help me
02-01 15:11:47 WARN - Got socket exception on connection [connectionId{localValue:43}] to 172.16.199.96:22001. All connections to 172.16.199.96:22001 will be closed.
02-01 15:11:47 INFO - Closed connection [connectionId{localValue:43}] to 172.16.199.96:22001 because there was a socket exception raised by this connection.
org.springframework.data.mongodb.UncategorizedMongoDbException: Exception receiving message; nested exception is com.mongodb.MongoSocketReadException: Exception receiving message
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:107)
at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2135)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1978)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1784)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1767)
at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:641)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:606)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:598)
at com.xxx.xxx.xxx.xxx(xxxService.java:46)
at com.xxx.xxx.xxx.xxx(xxxService.java:157)
at com.xxx.xxx.xxx.xxx(xxxService.java:142)
at com.xxx.xxx.xxx.xxx(xxxService.java:87)
at com.alibaba.dubbo.common.bytecode.Wrapper2.invokeMethod(Wrapper2.java)
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:61)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:132)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:100)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:98)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:81)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mongodb.MongoSocketReadException: Exception receiving message
at com.mongodb.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:483)
at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:228)
at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:96)
at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:440)
at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:112)
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168)
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289)
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176)
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:216)
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:207)
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:113)
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:516)
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:510)
at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:431)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:404)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:510)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:81)
at com.mongodb.Mongo.execute(Mongo.java:836)
at com.mongodb.Mongo$2.execute(Mongo.java:823)
at com.mongodb.DBCursor.initializeCursor(DBCursor.java:870)
at com.mongodb.DBCursor.hasNext(DBCursor.java:142)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1964)
... 37 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.mongodb.connection.SocketStream.read(SocketStream.java:85)
at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:494)
at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:224)
... 57 common frames omitted
java version 1.8.
spring boot version 1.5.3.
deployed with docker.
mongo.hosts=ip:port,ip:port,ip:port
mongo.database.name=dbname
mongo.username=username
mongo.password=pwd
mongo.connections.per.host=32
mongo.max.wait.time=2000
mongo.connect.timeout=2000
You can try,
autoConnectRetry simply means the driver will automatically attempt to reconnect to the server(s) after unexpected disconnects. In production environments you usually want this set to true.
This is from another post, How to configure MongoDB Java driver MongoOptions for production use?
for everybody who is experiencing the same random MongoSocketReadException, you may need the socketTimeoutMS or maxIdleTimeMS parameters instead. The parameter autoConnectRetry is not exposed any more in the mongodb connection string.
Our situation: we switched to mongodb atlas serverless solution for our development and testing environments, ever since then we got this MongoSocketReadException like every 15 min. or randomly. We are also behind a enterprise firewall.
According to https://www.mongodb.com/docs/v6.0/tutorial/connection-pool-performance-tuning/:
a misconfigured firewall closes a socket connection incorrectly and the driver cannot detect that the connection closed improperly.
you need => Use socketTimeoutMS to ensure that sockets are always closed. Set socketTimeoutMS to two or three times the length of the slowest operation that the driver runs.
because the socketTimeoutMS is by default 0, which will never timeout.
And another parameter maxIdleTimeMS may also affect the connection because if the socket is closed and on the client side it's not detected, the connection will be still waiting in idle time and not cloesd. And by default it's 0 meaning it waits forever with no upper boundaries.
So configure this to a small amount may help the driver to close the the problematic connection with its closed socket, before it tries to connect to the db using the same connection and presumes the connection is still there.
So our solution:
...mongodbUri...?socketTimeoutMS=150000&maxIdleTimeMS=150000
we changed the socketTimeoutMS from 0 to 15s and same for the maxIdleTimeMS.
I recently got the following messages in our logs followed by a JVM crash (Due to OOME). I am not sure what to make of this and would really appreciate any guidance.
2015-03-19 21:15:02,457 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:26:29,543 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#15da1b6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#b35b08a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#51cfdd17 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#19397937
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#5c7d3838
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#7aea62dd
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#55622ff2
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#74004a8
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.net.SocketOutputStream.socketWrite0(Native Method)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
java.net.SocketOutputStream.write(SocketOutputStream.java:153)
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1751)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3425)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2196)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:718)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor306.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:302)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:282)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
2015-03-19 21:56:59,137 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:56:59,143 [eXistThread-18676] ERROR (XQueryServlet.java [process]:566) - Java heap space
java.lang.OutOfMemoryError: Java heap space
at org.exist.storage.btree.BTree$BTreeNode.read(BTree.java:1269)
at org.exist.storage.btree.BTree$BTreeNode.access$16(BTree.java:1239)
at org.exist.storage.btree.BTree.getBTreeNode(BTree.java:460)
at org.exist.storage.btree.BTree.scanSequential(BTree.java:413)
at org.exist.storage.btree.BTree$BTreeNode.scanNextPage(BTree.java:2039)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1835)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.access$12(BTree.java:1734)
at org.exist.storage.btree.BTree.query(BTree.java:379)
at org.exist.storage.structural.NativeStructuralIndexWorker.scanByType(NativeStructuralIndexWorker.java:259)
at org.exist.dom.VirtualNodeSet.getNodesFromIndex(VirtualNodeSet.java:457)
at org.exist.dom.VirtualNodeSet.realize(VirtualNodeSet.java:585)
at org.exist.dom.VirtualNodeSet.iterator(VirtualNodeSet.java:860)
at org.exist.dom.AbstractNodeSet.iterator(AbstractNodeSet.java:1)
at org.exist.storage.structural.NativeStructuralIndexWorker.findDescendantsByTagName(NativeStructuralIndexWorker.java:162)
at org.exist.xquery.LocationStep.getAttributes(LocationStep.java:645)
at org.exist.xquery.LocationStep.eval(LocationStep.java:434)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.Predicate.selectByNodeSet(Predicate.java:446)
at org.exist.xquery.Predicate.evalPredicate(Predicate.java:326)
at org.exist.xquery.LocationStep.processPredicate(LocationStep.java:251)
at org.exist.xquery.LocationStep.applyPredicate(LocationStep.java:238)
at org.exist.xquery.LocationStep.eval(LocationStep.java:462)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.LetExpr.eval(LetExpr.java:142)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.BindingExpression.eval(BindingExpression.java:164)
2015-03-19 21:56:59,147 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#79180a12 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#243c6d0c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#50191373 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#3a9d08ca
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#3ecdd11
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#44ff846d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#5ce5850a
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#eec1d04
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#6b8d4d9d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#53e9706d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#23d472cf
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#4dbe4f8c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#4c5e0203
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#54ac79fd
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#546e2bad
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#6b13cc83
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#57e185f8
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#60357d68
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#45231180
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#3021aa73
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#6bb437ca
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#2021c9e9
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#7d53637c
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#409c2c97
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#adc5929
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#241ca71a
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#42b26866
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#636b1c33
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#b160466
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#4af34669
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1b53e609
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#2062ebd4
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1b6cfe8a
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#4b7c2380
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#4f9be748
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#78108924
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#474b002
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#2ebee32f
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#3e0fe017
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#42aa175b
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#637f5bac
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#3a017b77
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#7b4f2b78
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.java:370)
com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.java:2532)
com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477)
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2596)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
com.newrelic.agent.TransactionService.addTransaction(TransactionService.java:142)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1104)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1087)
com.newrelic.agent.TracerService$TracerServiceImpl.getTracer(TracerService.java:136)
com.newrelic.agent.TracerService.getTracer(TracerService.java:41)
com.newrelic.agent.instrumentation.InvocationPoint.invoke(InvocationPoint.java:55)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
We are using hiberanate and c3p0 with the following c3p0 configuration:
We are using the following maven artifacts for hibernate and c3p0:
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.3.6.Final</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-c3p0</artifactId>
<version>4.3.6.Final</version>
</dependency>
with the following c3p0 configuration:
configuration = new Configuration().setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect")
.setProperty("hibernate.connection.provider_class", "org.hibernate.connection.C3P0ConnectionProvider")
.setProperty("hibernate.c3p0.idle_test_period", "1000")
.setProperty("hibernate.c3p0.min_size", "20")
.setProperty("hibernate.c3p0.max_size", "50")
.setProperty("hibernate.c3p0.timeout", "1800")
.setProperty("hibernate.c3p0.max_statements", "50")
The server is under very light load ~5 queries a second. (Java 8)
I stumbled into the same exception and the reason was a wrong password of the db user...
c3p0 you're a funny guy
So, directly the issue is that the Connection pool was trying to acquire new Connections, but the tasks attempting to acquire those tasks froze for a long period of time, so long that c3p0 decided the tasks must be deadlocked and then discarded and replaced the Thread pool. Later, the Thread pool was hung on idle connection test tasks.
Normally, "hung" tasks tend to look like the second Thread underneath the first "Pool thread stack traces:" label above: performing network IO that fails to complete. Your circumstances are odd in that two of the three threads are not stuck in IO. They have barely begun to do anything, yet they aren't live. Then you experience an OutOfMemoryError, and you get another APPARENT DEADLOCK on idle connection test tasks that also look like they mostly should be live.
Maybe your application is very close to some kind of resource limits that are causing things to run very sluggishly? Straightforwardly, you might increase the amount of memory available to this application (or modify it to have a lower memory footprint). You experience an OOME the second time the Thread pool tries to flush and recreate hung Threads, not directly provoked by that, but quite possibly caused by the growing Thread footprint. (In your logs, are there lots of these APPARENT DEADLOCKs previously? If you force a JVM Thread dump, do you see lots of com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread instances around, still hung on old, not completing tasks?
Some general comments: If your load is no more than 5-ish simultaneous queries, why such a heavy pool? why not min_size 5-ish, max 10-ish? Your max_statements setting is way too small for the number of Connections you permit in the pool. I'd omit this entirely until you get things working smoothly. Then, to get a bit better performance, you might set the more-straightforward-to-reason-about maxStatementsPerConnection parameter if you want.
Mostly you need to keep your application's footprint (memory? Threads?) well below the resources alotted to it, either by increasing resources, reducing its footprint, or fixing any issue that might exist that cause its resource footprint to increase into limits. I'd start by making the pool smaller, the available memory larger, and configuring the pool to be much smaller.
Authentication issue for me too. I just added the domain part to the database server name and it all worked. Very misleading error.
We have several servers that each run an Oracle database 11g Release 11.2.0.1.0 - 64bit. We are connecting via JDBC like this:
public Connection createConnection(String drvClass, String connURL, String user, String pass)
throws ClassNotFoundException, SQLException {
Class.forName(drvClass);
Connection conn = DriverManager.getConnection(connURL, user, pass);
for (SQLWarning warn = conn.getWarnings(); warn != null; warn = warn.getNextWarning()) {
System.out.println("SQL Warning:");
System.out.println("State : " + warn.getSQLState());
System.out.println("Message: " + warn.getMessage());
System.out.println("Error : " + warn.getErrorCode());
}
return conn;
}
drvClass would be oracle.jdbc.OracleDriver. The program which tries to connect runs on each server. The database is reachable from out of other programs with the exact same connection properties.
It is also possible to let this program run on another server and let it connect to the problematic database. It can establish the connection.
The program does not work if it's running on the server locally. We tried both IP and servername.
On one server the code hangs at DriverManager.getConnection() an I cannot find out why. Does anyone have any idea what could cause this?
There is no entry about this in the DB logs.
Stacktrace of blocking thread:
"pool-27-thread-1" - Thread t#1483
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at oracle.net.ns.Packet.receive(Packet.java:239)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:255)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:973)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:291)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:474)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at com.companyname.DBConnectionInternal.DBConnection.createConnection(DBConnection.java:19)
at com.companyname.exportadapter.ExportCollector.initDatabase(ExportCollector.java:259)
at com.companyname.exportadapter.ExportCollector.run(ExportCollector.java:120)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <50be77> (a java.util.concurrent.ThreadPoolExecutor$Worker)
If i set DriverManager.setLoginTimeout(10)
then i get
Io exception: Socket read timed out.
you may making some unneccessary connections.
make Connection class static
,whenever you are creating new connection check older is alive or close then and then you must create new connection other wise return old connection.
like
if(conn!=null & !conn.isClosed()){
// code for create connection
}
It also depends on how the database side is configured, so check it with DBA of your system.
I would like to suggest using Connection pooling.
hope this helps.
You might want to enable JDBC debug logging for the ojdbc driver: http://docs.oracle.com/cd/B28359_01/java.111/b31224/diagnose.htm
That might give you some information about what the driver is doing.
Have you tried telnet-ing to the database server from the client machine (to assert it's reachable)?
The server was misconfigured. For some reason it had a virtual adapter configured which returned an ip to which nothing could connect. From outside the resolving worked correctly. Don't know why it never timed out with the wrong IP though.
I have console application that hangs during execution. Here is my configuration:
cfg.setProperty("hibernate.connection.driver_class", "com.mysql.jdbc.Driver");
cfg.setProperty("hibernate.connection.url", "jdbc:mysql://localhost:3306/db?user=db&password=db");
cfg.setProperty("hibernate.connection.username", "db");
cfg.setProperty("hibernate.connection.password", "db");
cfg.setProperty("hibernate.connection.pool_size", "5");
cfg.setProperty("hibernate.connection.autocommit", "false");
cfg.setProperty("hibernate.c3p0.min_size", "5");
cfg.setProperty("hibernate.c3p0.max_size", "20");
cfg.setProperty("hibernate.c3p0.timeout", "300");
cfg.setProperty("hibernate.c3p0.max_statements", "50");
cfg.setProperty("hibernate.c3p0.idle_test_period", "3000");
Here is my stacktrace:
"main" prio=10 tid=0x000000000168f800 nid=0x1c37 in Object.wait() [0x00007fa60d0ad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007400f4c68> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x00000007400f4c68> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:84)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:281)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1392)
at org.kriyak.parser.IndexArchiveRapid.indexFile(IndexArchiveRapid.java:70)
at org.kriyak.parser.IndexArchiveRapid.main(IndexArchiveRapid.java:53)
I open only one conencton and it doesn't seem that I leak them. And also I use one thread. I haven't adjusted any mysql settings except memory usage. Mysql works fine from console. Why can this happen? Is this c3p0 error?
does this happen immediately, or after a while? that is, do checkouts initially succeed, but then hang like this? if so, it looks like a Connection leak. please try setting c3p0 params unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces to see if there is a leak. See
http://www.mchange.com/projects/c3p0/#configuring_to_debug_and_workaround_broken_clients , http://www.mchange.com/projects/c3p0/#unreturnedConnectionTimeout , http://www.mchange.com/projects/c3p0/#debugUnreturnedConnectionStackTraces .
if this happens immediately, if no Connections are successfully checked out, the question is whether the pool ever succeeds at acquiring Connections. by default, if it never does succeed, after about 30 seconds your thread should break with a failure. (it doesn't look like you've done this, but if for example you'd set acquireRetryAttempts to zero, c3p0 might hang indefinitely waiting for Connections.)
to debug c3p0 issues, it's helpful to capture the version and config information that c3p0 dumps to logs at INFO level on pool initialization.
good luck!
Also, you haven't seem to initialize the checkoutTime parameter for c3p0, which specifies the amount of time a client should wait for acquiring a connection from the connection pool.
see http://www.mchange.com/projects/c3p0/#checkoutTimeout
You may need to increase c3p0.numHelperThreads. The helper threads are responsible for creating new database connections and adding them to the pool. If there aren't enough threads to keep up with demand then application threads will be stuck waiting in awaitAvailable(). To confirm this is the case look at all the HelperThreads and see if they are all in use and in the process of connecting to the database.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Connection Pool Exception: Cannot get a connection, pool error Timeout waiting for idle object
I am getting Cannot get a connection, pool error Timeout waiting for idle object error, When I try to create more than 250 threads in my web application. I am creating web application using JSF 2.0 and Hibernate.
I have tried with modified hibernate.xml,server.xml,context.xml and also mysql properties.
The followings are I am getting.
WARN (JDBCExceptionReporter.java:233) - SQL Error: 0, SQLState: null
ERROR (JDBCExceptionReporter.java:234) - Cannot get a connection, pool error Timeout waiting for idle object
ERROR (BaseServlet.java:301) - ******** java.lang.Thread.getStackTrace(Thread.java:1426)
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source)
Caused by: org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
sun.reflect.GeneratedMethodAccessor378.invoke(Unknown Source)
Please if have any idea about this help me......
Maybe you have set a timeout for "dead" connections and some queries take longer than that. That means that your pool removed a busy connection as "dead" from the pool and requests another from the DB - until the DB pulls the plug.
To debug this, enable logging for your connection pool, so you can see when it requests new connections.
also check your mysql connection settings. and try to close connection when you are done with your db coz next time(beyond maxConnectionAge limit) that connection state will be dead .
I had such issue before, what you need to do is
close hibernate sessions when you are done with it.
e.g.
Session sess = factory.openSession();
Transaction tx;
try {
tx = sess.beginTransaction();
//do some work
...
tx.commit();
}
catch (Exception e) {
if (tx!=null) tx.rollback();
throw e;
}
finally {
sess.close(); // closing session
}