I have a weak connection with database server and time consuming query which is sometimes fails with exception:
Caused by: java.sql.SQLException: I/O Error: Socket closed
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2481)
at net.sourceforge.jtds.jdbc.TdsCore.getNextRow(TdsCore.java:805)
at net.sourceforge.jtds.jdbc.JtdsResultSet.next(JtdsResultSet.java:611)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:885)
at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:731)
at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:477)
at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:146)
at net.sourceforge.jtds.jdbc.TdsData.readData(TdsData.java:901)
at net.sourceforge.jtds.jdbc.TdsCore.tdsRowToken(TdsCore.java:3175)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2433)
How can connection interruption be handled? currently I have to manually re-run the operation until it executes successfully, maybe it could be done at jdbc driver level
ps socketTimeout property doesn't seem to affect this
Are you sure this is because of query/procedure taking long time ? As Socket closed error normally means something went wrong with the network connection itself, basically something that your driver could not control...
In general, I would suggest to move to a Pool based mechanism, which allows greater control on your persistence layer interaction.
Related
The program runs for a period of time and always reports an error, restarting and returning to normal, the error is as follows:
javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not prepare statementat org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1626)at org.hibernate.query.Query.getResultList(Query.java:165)at net.skycloud.platform.user.repository.KeycloakGroupDao.findAll(KeycloakGroupDao.java:111)at net.skycloud.platform.user.service.UserGroupService.getAllUserGroups(UserGroupService.java:247)at net.skycloud.platform.user.api.rest.UserController.searchUserAndGroup(UserController.java:362)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.base/java.lang.reflect.Method.invoke(Method.java:566)at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)at java.base/java.lang.Thread.run(Thread.java:829)Caused by: org.hibernate.exception.GenericJDBCException: could not prepare statementat org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:186)at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151)at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2122)at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2059)at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2037)at org.hibernate.loader.Loader.doQuery(Loader.java:956)at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:357)at org.hibernate.loader.Loader.doList(Loader.java:2868)at org.hibernate.loader.Loader.doList(Loader.java:2850)at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2682)at org.hibernate.loader.Loader.list(Loader.java:2677)at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:338)at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:2181)at org.hibernate.internal.AbstractSharedSessionContract.list(AbstractSharedSessionContract.java:1204)at org.hibernate.query.internal.NativeQueryImpl.doList(NativeQueryImpl.java:177)at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617)... 52 common frames omittedCaused by: java.sql.SQLException: Connection is closedat com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:515)at com.sun.proxy.$Proxy156.prepareStatement(Unknown Source)at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:337)at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176)... 67 common frames omitted
I hope to fix this bug
Its a database operations level exception using EntityManager or any other session manager (which is not your case) as exception suggest, how ever it can have many reasons such database has closed the connection or network terminated the connection socket, and to solve it you can configure to keep the connection alive but remember you can not hold the hibernate session alive for long time by default .
Also you may have look on OSIV anti pattern.
Like mentioned here we have almost the same issues with a java project (playframework) with neo4j-ogm and the ogm-bolt-driver (3.2.27). Connection resets sometimes, throws exception and reconnects with the next request (mostly).
The nodes will be queried from database via repositories and neo4j-ogm session (will be opened for small pieces of work).
Any advice will be appreciated.
play.api.UnexpectedException: Unexpected exception[ConnectionException: Connection to the database failed]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:358)
at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:264)
at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:430)
at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:422)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:454)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: org.neo4j.ogm.exception.ConnectionException: Connection to the database failed
at org.neo4j.ogm.drivers.bolt.driver.BoltDriverExceptionTranslator.translateExceptionIfPossible(BoltDriverExceptionTranslator.java:38)
at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:601)
at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:558)
at org.neo4j.ogm.session.delegates.LoadOneDelegate.load(LoadOneDelegate.java:87)
at org.neo4j.ogm.session.delegates.LoadOneDelegate.load(LoadOneDelegate.java:53)
at org.neo4j.ogm.session.Neo4jSession.load(Neo4jSession.java:178)
at modules.shared.repositories.CrudRepository.find(CrudRepository.java:50)
at config.secured.Secured.getUserFromSession(Secured.java:88)
at config.secured.Secured.getUser(Secured.java:56)
at config.secured.Secured.getUser(Secured.java:39)
at config.secured.DataWizardSecurity$AuthenticatedAction.call(DataWizardSecurity.java:63)
at config.filter.SafeFormFactoryToRequest.call(SafeFormFactoryToRequest.java:24)
at config.filter.Messaged.call(Messaged.java:29)
at play.core.j.JavaAction.$anonfun$apply$8(JavaAction.scala:175)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:672)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:431)
at play.core.j.HttpExecutionContext.$anonfun$execute$1(HttpExecutionContext.scala:64)
at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:70)
at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:59)
at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:393)
at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:302)
at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:276)
at scala.concurrent.impl.Promise$DefaultPromise.map(Promise.scala:146)
at scala.concurrent.Future$.apply(Future.scala:672)
at play.core.j.JavaAction.apply(JavaAction.scala:176)
at play.api.mvc.Action.$anonfun$apply$4(Action.scala:82)
at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:168)
at scala.util.Try$.apply(Try.scala:210)
at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:168)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:199)
at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:417)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:448)
... 12 common frames omitted
Caused by: org.neo4j.driver.exceptions.ServiceUnavailableException: Connection to the database failed
at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:143)
at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:98)
at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:92)
at org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction.newOrExistingNativeTransaction(BoltTransaction.java:62)
at org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction.<init>(BoltTransaction.java:50)
at org.neo4j.ogm.drivers.bolt.driver.BoltDriver.lambda$null$0(BoltDriver.java:128)
at org.neo4j.ogm.session.transaction.DefaultTransactionManager.openTransaction(DefaultTransactionManager.java:75)
at org.neo4j.ogm.session.Neo4jSession.beginTransaction(Neo4jSession.java:530)
at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:580)
... 49 common frames omitted
Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause: null
at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.transformError(ChannelErrorHandler.java:127)
at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.fail(ChannelErrorHandler.java:113)
at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.exceptionCaught(ChannelErrorHandler.java:99)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:345)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:376)
at org.neo4j.driver.internal.shaded.io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at org.neo4j.driver.internal.shaded.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at org.neo4j.driver.internal.shaded.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
As I mention in my answer here, this exception comes from Neo4j Java Driver and there may be several reasons for this to happen.
For instance, this might happen when driver acquires a connection from internal connection pool that has been idle for too long. Some cloud providers limit connection idle time for load balancers. Driver may fail as soon as it tries using such connection.
You can enable connection liveness check to assist with this. It makes sure that a short network exchange happens for every connection that has been idle over the configured timeout. If exchange fails, another connection is picked or established before it is provided for further driver usage. While this involves an extra network exchange, it is likely to prevent driver failures that might lead to higher-level retry management with potentially exponential delay. The timeout can be set to zero if you wish to test every connection.
Sample configuration:
var config = Config.builder()
.withConnectionLivenessCheckTimeout( 60000, TimeUnit.MILLISECONDS )
.build();
var driver = GraphDatabase.driver( URL, AuthTokens, config );
I'm trying to understand why this error occurs:
javax.net.ssl|WARNING|01|main|2021-07-04 12:08:30.668 CEST|SSLSocketImpl.java:497|SSLSocket duplex close failed (
"throwable" : {
java.net.SocketException: Socket is closed
at java.base/java.net.Socket.shutdownInput(Socket.java:1538)
at java.base/sun.security.ssl.BaseSSLSocketImpl.shutdownInput(BaseSSLSocketImpl.java:216)
at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:751)
at java.base/sun.security.ssl.SSLSocketImpl.bruteForceCloseInput(SSLSocketImpl.java:701)
at java.base/sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:562)
at java.base/sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:486)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.close(SSLSocketImpl.java:1034)
at com.ibm.db2.jcc.t4.a0.j(a0.java:343)
at com.ibm.db2.jcc.t4.b.freeTransport_(b.java:5523)
at com.ibm.db2.jcc.t4.a.close_(a.java:455)
at com.ibm.db2.jcc.am.Agent.close(Agent.java:345)
at com.ibm.db2.jcc.t4.b.b(b.java:965)
at com.ibm.db2.jcc.t4.b.a(b.java:804)
at com.ibm.db2.jcc.t4.b.a(b.java:441)
at com.ibm.db2.jcc.t4.b.a(b.java:414)
at com.ibm.db2.jcc.t4.b.<init>(b.java:352)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:233)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:200)
at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:182)
at com.example.MainApplication.main(MainApplication.java:36)}
)
javax.net.ssl|ALL|01|main|2021-07-04 12:08:30.668 CEST|SSLSocketImpl.java:1217|Closing output stream
Exception in sql: com.ibm.db2.jcc.am.SqlNonTransientConnectionException
DB2 SQL Error: SQLCODE=-20157, SQLSTATE=08004, SQLERRMC=WEBADMIN;QUIESCE DATABASE;;, DRIVER=4.25.13
com.ibm.db2.jcc.am.SqlNonTransientConnectionException: DB2 SQL Error: SQLCODE=-20157, SQLSTATE=08004
The exception is thrown when I'm invoking getConnection() method of the DriverManager:
connection = DriverManager.getConnection(DB_URL, properties);
I'm using Java 11.0.11 from Oracle (non-OpenJDK).
Talk with your DBA team or whoever manages the database - you get this exception because someone (or some job) has put the database into a specific state that is used for maintenance activity.
Normally this is a temporary situation, and the database (or Db2-instance) needs to be brought back to normal mode by an unquiesce action, when the maintenance activity is completed. After the unquiesce action, your connection should complete as normal.
The SQLCODE (-20157) and SQLERRMC ( SQLERRMC=WEBADMIN;QUIESCE DATABASE;) that are in the message tell you the cause of this exception.
Lookup SQL20157N in the docs to get the detailed explanation.
Problem Description: MongoDB version is 3.4
In fact, did not do anything on the normal query, write,
because it is in the testing phase, QPS is small.
Question:
1: How is this anomaly produced.
2: what configuration or adjustment needs to be done? help me
02-01 15:11:47 WARN - Got socket exception on connection [connectionId{localValue:43}] to 172.16.199.96:22001. All connections to 172.16.199.96:22001 will be closed.
02-01 15:11:47 INFO - Closed connection [connectionId{localValue:43}] to 172.16.199.96:22001 because there was a socket exception raised by this connection.
org.springframework.data.mongodb.UncategorizedMongoDbException: Exception receiving message; nested exception is com.mongodb.MongoSocketReadException: Exception receiving message
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:107)
at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2135)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1978)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1784)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1767)
at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:641)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:606)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:598)
at com.xxx.xxx.xxx.xxx(xxxService.java:46)
at com.xxx.xxx.xxx.xxx(xxxService.java:157)
at com.xxx.xxx.xxx.xxx(xxxService.java:142)
at com.xxx.xxx.xxx.xxx(xxxService.java:87)
at com.alibaba.dubbo.common.bytecode.Wrapper2.invokeMethod(Wrapper2.java)
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:61)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:132)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:100)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:98)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:81)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mongodb.MongoSocketReadException: Exception receiving message
at com.mongodb.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:483)
at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:228)
at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:96)
at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:440)
at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:112)
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168)
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289)
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176)
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:216)
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:207)
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:113)
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:516)
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:510)
at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:431)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:404)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:510)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:81)
at com.mongodb.Mongo.execute(Mongo.java:836)
at com.mongodb.Mongo$2.execute(Mongo.java:823)
at com.mongodb.DBCursor.initializeCursor(DBCursor.java:870)
at com.mongodb.DBCursor.hasNext(DBCursor.java:142)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1964)
... 37 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.mongodb.connection.SocketStream.read(SocketStream.java:85)
at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:494)
at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:224)
... 57 common frames omitted
java version 1.8.
spring boot version 1.5.3.
deployed with docker.
mongo.hosts=ip:port,ip:port,ip:port
mongo.database.name=dbname
mongo.username=username
mongo.password=pwd
mongo.connections.per.host=32
mongo.max.wait.time=2000
mongo.connect.timeout=2000
You can try,
autoConnectRetry simply means the driver will automatically attempt to reconnect to the server(s) after unexpected disconnects. In production environments you usually want this set to true.
This is from another post, How to configure MongoDB Java driver MongoOptions for production use?
for everybody who is experiencing the same random MongoSocketReadException, you may need the socketTimeoutMS or maxIdleTimeMS parameters instead. The parameter autoConnectRetry is not exposed any more in the mongodb connection string.
Our situation: we switched to mongodb atlas serverless solution for our development and testing environments, ever since then we got this MongoSocketReadException like every 15 min. or randomly. We are also behind a enterprise firewall.
According to https://www.mongodb.com/docs/v6.0/tutorial/connection-pool-performance-tuning/:
a misconfigured firewall closes a socket connection incorrectly and the driver cannot detect that the connection closed improperly.
you need => Use socketTimeoutMS to ensure that sockets are always closed. Set socketTimeoutMS to two or three times the length of the slowest operation that the driver runs.
because the socketTimeoutMS is by default 0, which will never timeout.
And another parameter maxIdleTimeMS may also affect the connection because if the socket is closed and on the client side it's not detected, the connection will be still waiting in idle time and not cloesd. And by default it's 0 meaning it waits forever with no upper boundaries.
So configure this to a small amount may help the driver to close the the problematic connection with its closed socket, before it tries to connect to the db using the same connection and presumes the connection is still there.
So our solution:
...mongodbUri...?socketTimeoutMS=150000&maxIdleTimeMS=150000
we changed the socketTimeoutMS from 0 to 15s and same for the maxIdleTimeMS.
I'm using HIkariCP, Hibernate and MySQL.
The problem is: When application starts, everything works fine. But when I leave it idle for some time and then get back and try to get connection from Hikari, it throws exception:
Caused by: java.sql.SQLTransientConnectionException: Main DB Pool - Connection is not available, request timed out after 60072ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:115)
... 59 common frames omitted
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 820 887 milliseconds ago. The last packet sent successfully to the server was 60 060 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3459)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3900)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2486)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2444)
at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:157)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:169)
... 64 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:101)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:144)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:174)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3469)
... 74 common frames omitted
MySQL database is located in external server and it has some timeouts/session expiration features and I cannot change it. I suppose the problem is caused exactly by some kind of session expiration due to too big inactivity time (because problem happens only after application remains idle for some time). It may by similar problem, I don't know what is the exact reason why connection is lost.
And the question is: how to force Hikari to refresh the connection in such situation instead of trowing exceptions? (or where else the problem may be?)
My config:
HikariConfig config = new HikariConfig();
config.setPoolName("Main DB Pool");
config.addDataSourceProperty("url", "jdbc:mysql://" + host);
config.setUsername(user);
config.setPassword(password);
config.setDataSourceClassName(dataSourceClass);
config.setMaximumPoolSize(maxPoolSize);
config.setIdleTimeout(30000);
config.setConnectionTimeout(60000);
config.setValidationTimeout(60000);
config.setConnectionInitSql("SELECT 1");
config.setConnectionTestQuery("SELECT 1");
config.setLeakDetectionThreshold(120000);
return new HikariDataSource(config);
Right from the docs. You will need to figure out what the external servers timeouts are and configure HikariCP's to be a little shorter.
⌚maxLifetime
This property controls the maximum lifetime of a connection in the pool. An in-use connection will never be retired, only when it is closed will it then be removed. We strongly recommend setting this value, and it should be at least 30 seconds less than any database or infrastructure imposed connection time limit. A value of 0 indicates no maximum lifetime (infinite lifetime), subject of course to the idleTimeout setting. Default: 1800000 (30 minutes)