I would like to start my Java/Play! Service even though the Database is not yet reachable. So far, if the database is not available the service stops. Is it possible in Play to try and reconnect every second or so without completely stopping the service and reconnecting to db as soon as connection is available.
Our application.conf currently looks like this:
play.evolutions.enabled=false
//play.db.pool=bonecp
db {
default.driver=org.postgresql.Driver
default.url="jdbc:postgresql://postgresdb.postgres:5432/postcode"
default.username="<username>"
default.password="<strong-password>"
}
play.db {
config = "db"
prototype.hikaricp = {
connectionTimeout = 250 milliseconds
initializationFailFast = false
readOnly = true
}
}
ebean.default = ["models.*"]
Currently I get the following logs:
2017-05-15 13:26:54.525 [error] postcode-play: Connection error:
org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
Ort: Datei: postinit.c, Routine: InitPostgres, Zeile: 779.
Server SQLState: 53300
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2538)
at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:122)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:227)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
at org.postgresql.Driver.makeConnection(Driver.java:431)
at org.postgresql.Driver.connect(Driver.java:247)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
2017-05-15 13:26:54.626 [error] postcode-play: Connection error:
org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
Ort: Datei: postinit.c, Routine: InitPostgres, Zeile: 779.
Server SQLState: 53300
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2538)
at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:122)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:227)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
at org.postgresql.Driver.makeConnection(Driver.java:431)
at org.postgresql.Driver.connect(Driver.java:247)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
.
.
.
Indicating that the server does not stop altogether but tries to connect to db multiple times which is as desired. But if the database becomes available the following logs are written:
2017-05-15 14:08:43.748 [error] postcode-play:
! #7430ed7gf - Internal server error, for (GET) [/?postcode=88682&country=DE] ->
play.api.Configuration$$anon$1: Configuration error[Cannot connect to database [default]]
at play.api.Configuration$.configError(Configuration.scala:155)
at play.api.Configuration.reportError(Configuration.scala:808)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:48)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
at scala.collection.immutable.List.foreach(List.scala:392)
at play.api.db.DefaultDBApi.connect(DefaultDBApi.scala:42)
at play.api.db.DBApiProvider.get$lzycompute(DBModule.scala:72)
at play.api.db.DBApiProvider.get(DBModule.scala:62)
at play.api.db.DBApiProvider.get(DBModule.scala:58)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-4 - Connection is not available, request timed out after 5001ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:138)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:44)
at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
at scala.collection.immutable.List.foreach(List.scala:392)
at play.api.db.DefaultDBApi.connect(DefaultDBApi.scala:42)
Caused by: org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:265)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
at org.postgresql.Driver.makeConnection(Driver.java:431)
at org.postgresql.Driver.connect(Driver.java:247)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:193)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:430)
Caused by: java.net.ConnectException: Verbindungsaufbau abgelehnt (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:62)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194
)
The server cannot reach out to the database even though it is available now. If I restart the server the database will be found and can be worked with, but the database lways has to be there on startup. How can I fix that, so that the app can be started independently of the database?
If the database becomes unavailable after the application has started it is no problem after the db becomes available again. it is automatically reconnected.
Any help is greatly apreciated!
It seems like you are leaving some connections open to Postgres. These connections must be closed or else Postgres will hit its max_connections limit.
You could increase max_connections but it's not recommended as performance of Postgres will degrade if it's set too high. Also, as you are using HikariCP it wouldn't be a bad idea to take a look at the HikariCP docs, in particular the idleTimeout and the minimumIdle config variables should help you.
If the above doesn't work then you must confirm exactly how many connections are being made to Postgres. Maybe even check the number of sessions you personally are making to Postgres via SSH or through other apps, not just the number of Play Framework connections.
Related
Like mentioned here we have almost the same issues with a java project (playframework) with neo4j-ogm and the ogm-bolt-driver (3.2.27). Connection resets sometimes, throws exception and reconnects with the next request (mostly).
The nodes will be queried from database via repositories and neo4j-ogm session (will be opened for small pieces of work).
Any advice will be appreciated.
play.api.UnexpectedException: Unexpected exception[ConnectionException: Connection to the database failed]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:358)
at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:264)
at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:430)
at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:422)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:454)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: org.neo4j.ogm.exception.ConnectionException: Connection to the database failed
at org.neo4j.ogm.drivers.bolt.driver.BoltDriverExceptionTranslator.translateExceptionIfPossible(BoltDriverExceptionTranslator.java:38)
at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:601)
at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:558)
at org.neo4j.ogm.session.delegates.LoadOneDelegate.load(LoadOneDelegate.java:87)
at org.neo4j.ogm.session.delegates.LoadOneDelegate.load(LoadOneDelegate.java:53)
at org.neo4j.ogm.session.Neo4jSession.load(Neo4jSession.java:178)
at modules.shared.repositories.CrudRepository.find(CrudRepository.java:50)
at config.secured.Secured.getUserFromSession(Secured.java:88)
at config.secured.Secured.getUser(Secured.java:56)
at config.secured.Secured.getUser(Secured.java:39)
at config.secured.DataWizardSecurity$AuthenticatedAction.call(DataWizardSecurity.java:63)
at config.filter.SafeFormFactoryToRequest.call(SafeFormFactoryToRequest.java:24)
at config.filter.Messaged.call(Messaged.java:29)
at play.core.j.JavaAction.$anonfun$apply$8(JavaAction.scala:175)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:672)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:431)
at play.core.j.HttpExecutionContext.$anonfun$execute$1(HttpExecutionContext.scala:64)
at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:70)
at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:59)
at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:393)
at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:302)
at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:276)
at scala.concurrent.impl.Promise$DefaultPromise.map(Promise.scala:146)
at scala.concurrent.Future$.apply(Future.scala:672)
at play.core.j.JavaAction.apply(JavaAction.scala:176)
at play.api.mvc.Action.$anonfun$apply$4(Action.scala:82)
at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:168)
at scala.util.Try$.apply(Try.scala:210)
at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:168)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:199)
at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:417)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:448)
... 12 common frames omitted
Caused by: org.neo4j.driver.exceptions.ServiceUnavailableException: Connection to the database failed
at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:143)
at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:98)
at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:92)
at org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction.newOrExistingNativeTransaction(BoltTransaction.java:62)
at org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction.<init>(BoltTransaction.java:50)
at org.neo4j.ogm.drivers.bolt.driver.BoltDriver.lambda$null$0(BoltDriver.java:128)
at org.neo4j.ogm.session.transaction.DefaultTransactionManager.openTransaction(DefaultTransactionManager.java:75)
at org.neo4j.ogm.session.Neo4jSession.beginTransaction(Neo4jSession.java:530)
at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:580)
... 49 common frames omitted
Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause: null
at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.transformError(ChannelErrorHandler.java:127)
at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.fail(ChannelErrorHandler.java:113)
at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.exceptionCaught(ChannelErrorHandler.java:99)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:345)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:376)
at org.neo4j.driver.internal.shaded.io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at org.neo4j.driver.internal.shaded.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at org.neo4j.driver.internal.shaded.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
As I mention in my answer here, this exception comes from Neo4j Java Driver and there may be several reasons for this to happen.
For instance, this might happen when driver acquires a connection from internal connection pool that has been idle for too long. Some cloud providers limit connection idle time for load balancers. Driver may fail as soon as it tries using such connection.
You can enable connection liveness check to assist with this. It makes sure that a short network exchange happens for every connection that has been idle over the configured timeout. If exchange fails, another connection is picked or established before it is provided for further driver usage. While this involves an extra network exchange, it is likely to prevent driver failures that might lead to higher-level retry management with potentially exponential delay. The timeout can be set to zero if you wish to test every connection.
Sample configuration:
var config = Config.builder()
.withConnectionLivenessCheckTimeout( 60000, TimeUnit.MILLISECONDS )
.build();
var driver = GraphDatabase.driver( URL, AuthTokens, config );
I'm trying to establish Oracle DB connection from the application which runs on an EC2 instance of AWS. Oracle DB is in on-prem server. Firewall has been opened and I'm able to telnet to the SCAN and VIP hosts of that DB from my EC2 instance. But, still I'm getting the below exception:
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:392) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:434) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:687) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:343) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
... 239 common frames omitted
Caused by: java.net.UnknownHostException: <<hostname>>: unknown error
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_40-internal]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907) ~[na:1.8.0_40-internal]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302) ~[na:1.8.0_40-internal]
at java.net.InetAddress.getAllByName0(InetAddress.java:1255) ~[na:1.8.0_40-internal]
at java.net.InetAddress.getAllByName(InetAddress.java:1171) ~[na:1.8.0_40-internal]
at java.net.InetAddress.getAllByName(InetAddress.java:1105) ~[na:1.8.0_40-internal]
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:117) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.net.nt.ConnOption.connect(ConnOption.java:133) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:370) ~[ojdbc6-11.2.0.3.jar:11.2.0.3.0]
JDBC URL:
jdbc:oracle:thin:#(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)
(HOST = <<IP address of the host>>)(PORT = 1590)))(LOAD_BALANCE = yes)
(CONNECT_DATA =(SERVICE_NAME = <<example.service.com>>)(FAILOVER_MODE =(TYPE = SELECT)
(METHOD = BASIC))))`
Are you using java 8 ? If so then probably it's related to this bug
Java 7 or 9 would probably give you a more useful error message instead of "unknown error" (eg possibly "Name or service not known")
Apart from that, did you try a tnsping from the host you're trying to connect from ?
Also as per the below in the Oracle driver documentation when using a tnsnames entry in the jdbc URL it should be as below, using OCI driver :
Note that you can also specify the database by a TNSNAMES entry. You can find the available TNSNAMES entries listed in the file tnsnames.ora on the client computer from which you are connecting. For example, if you want to connect to the database on host myhost as user scott with password tiger that has a TNSNAMES entry of MyHostString, enter:
Connection conn = DriverManager.getConnection
("jdbc:oracle:oci8:#MyHostString","scott","tiger");
Problem Description: MongoDB version is 3.4
In fact, did not do anything on the normal query, write,
because it is in the testing phase, QPS is small.
Question:
1: How is this anomaly produced.
2: what configuration or adjustment needs to be done? help me
02-01 15:11:47 WARN - Got socket exception on connection [connectionId{localValue:43}] to 172.16.199.96:22001. All connections to 172.16.199.96:22001 will be closed.
02-01 15:11:47 INFO - Closed connection [connectionId{localValue:43}] to 172.16.199.96:22001 because there was a socket exception raised by this connection.
org.springframework.data.mongodb.UncategorizedMongoDbException: Exception receiving message; nested exception is com.mongodb.MongoSocketReadException: Exception receiving message
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:107)
at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2135)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1978)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1784)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1767)
at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:641)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:606)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:598)
at com.xxx.xxx.xxx.xxx(xxxService.java:46)
at com.xxx.xxx.xxx.xxx(xxxService.java:157)
at com.xxx.xxx.xxx.xxx(xxxService.java:142)
at com.xxx.xxx.xxx.xxx(xxxService.java:87)
at com.alibaba.dubbo.common.bytecode.Wrapper2.invokeMethod(Wrapper2.java)
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:61)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:132)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:100)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:98)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:81)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mongodb.MongoSocketReadException: Exception receiving message
at com.mongodb.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:483)
at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:228)
at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:96)
at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:440)
at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:112)
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168)
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289)
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176)
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:216)
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:207)
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:113)
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:516)
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:510)
at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:431)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:404)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:510)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:81)
at com.mongodb.Mongo.execute(Mongo.java:836)
at com.mongodb.Mongo$2.execute(Mongo.java:823)
at com.mongodb.DBCursor.initializeCursor(DBCursor.java:870)
at com.mongodb.DBCursor.hasNext(DBCursor.java:142)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1964)
... 37 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.mongodb.connection.SocketStream.read(SocketStream.java:85)
at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:494)
at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:224)
... 57 common frames omitted
java version 1.8.
spring boot version 1.5.3.
deployed with docker.
mongo.hosts=ip:port,ip:port,ip:port
mongo.database.name=dbname
mongo.username=username
mongo.password=pwd
mongo.connections.per.host=32
mongo.max.wait.time=2000
mongo.connect.timeout=2000
You can try,
autoConnectRetry simply means the driver will automatically attempt to reconnect to the server(s) after unexpected disconnects. In production environments you usually want this set to true.
This is from another post, How to configure MongoDB Java driver MongoOptions for production use?
for everybody who is experiencing the same random MongoSocketReadException, you may need the socketTimeoutMS or maxIdleTimeMS parameters instead. The parameter autoConnectRetry is not exposed any more in the mongodb connection string.
Our situation: we switched to mongodb atlas serverless solution for our development and testing environments, ever since then we got this MongoSocketReadException like every 15 min. or randomly. We are also behind a enterprise firewall.
According to https://www.mongodb.com/docs/v6.0/tutorial/connection-pool-performance-tuning/:
a misconfigured firewall closes a socket connection incorrectly and the driver cannot detect that the connection closed improperly.
you need => Use socketTimeoutMS to ensure that sockets are always closed. Set socketTimeoutMS to two or three times the length of the slowest operation that the driver runs.
because the socketTimeoutMS is by default 0, which will never timeout.
And another parameter maxIdleTimeMS may also affect the connection because if the socket is closed and on the client side it's not detected, the connection will be still waiting in idle time and not cloesd. And by default it's 0 meaning it waits forever with no upper boundaries.
So configure this to a small amount may help the driver to close the the problematic connection with its closed socket, before it tries to connect to the db using the same connection and presumes the connection is still there.
So our solution:
...mongodbUri...?socketTimeoutMS=150000&maxIdleTimeMS=150000
we changed the socketTimeoutMS from 0 to 15s and same for the maxIdleTimeMS.
I'm using HIkariCP, Hibernate and MySQL.
The problem is: When application starts, everything works fine. But when I leave it idle for some time and then get back and try to get connection from Hikari, it throws exception:
Caused by: java.sql.SQLTransientConnectionException: Main DB Pool - Connection is not available, request timed out after 60072ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:115)
... 59 common frames omitted
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 820 887 milliseconds ago. The last packet sent successfully to the server was 60 060 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3459)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3900)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2486)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2444)
at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:157)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:169)
... 64 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:101)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:144)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:174)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3469)
... 74 common frames omitted
MySQL database is located in external server and it has some timeouts/session expiration features and I cannot change it. I suppose the problem is caused exactly by some kind of session expiration due to too big inactivity time (because problem happens only after application remains idle for some time). It may by similar problem, I don't know what is the exact reason why connection is lost.
And the question is: how to force Hikari to refresh the connection in such situation instead of trowing exceptions? (or where else the problem may be?)
My config:
HikariConfig config = new HikariConfig();
config.setPoolName("Main DB Pool");
config.addDataSourceProperty("url", "jdbc:mysql://" + host);
config.setUsername(user);
config.setPassword(password);
config.setDataSourceClassName(dataSourceClass);
config.setMaximumPoolSize(maxPoolSize);
config.setIdleTimeout(30000);
config.setConnectionTimeout(60000);
config.setValidationTimeout(60000);
config.setConnectionInitSql("SELECT 1");
config.setConnectionTestQuery("SELECT 1");
config.setLeakDetectionThreshold(120000);
return new HikariDataSource(config);
Right from the docs. You will need to figure out what the external servers timeouts are and configure HikariCP's to be a little shorter.
⌚maxLifetime
This property controls the maximum lifetime of a connection in the pool. An in-use connection will never be retired, only when it is closed will it then be removed. We strongly recommend setting this value, and it should be at least 30 seconds less than any database or infrastructure imposed connection time limit. A value of 0 indicates no maximum lifetime (infinite lifetime), subject of course to the idleTimeout setting. Default: 1800000 (30 minutes)
I have a java code that generate and fill tables to MySQL database.
The method is to create a connection to generate a table and then close that connection afterward.
Then create new connection to fill the table and so on.
I have about 14000 table to generate. The problem is that after generating about 800 tables I got this message and I have to restart mysql again.
Is there a cap to the number of connection I could create? is there anyway to increase it? Any Ideas?
Thanks
java.sql.SQLException: Communication link failure: java.net.SocketException, underlying cause: Connection reset
** BEGIN NESTED EXCEPTION **
java.net.SocketException
MESSAGE: Connection reset
STACKTRACE:
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1449)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1826)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1803)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:2353)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:750)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:1627)
at com.mysql.jdbc.Connection.<init>(Connection.java:427)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:395)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at db.DbConnection.establishConnection(DbConnection.java:21)
at db.DbActivities.addToDocSent(DbActivities.java:125)
at db.DbGeneration.main(DbGeneration.java:58)
** END NESTED EXCEPTION **