I am trying to connect to a remote server and have set the connect timeout to 120 seconds. Sometimes it connects, while sometimes it throws us Connect timeout even before trying for 2 mins.
setConnectTimeout(120000);
setReadTimeout(120000);
I am using AsyncHttpClientConfig for setting these properties.
Below is the error :
java.util.concurrent.ExecutionException: java.net.ConnectException:
connection timed out: at
com.ning.http.client.providers.netty.future.NettyResponseFuture.abort(NettyResponseFuture.java:231)
at
com.ning.http.client.providers.netty.request.NettyConnectListener.onFutureFailure(NettyConnectListener.java:137)
at
com.ning.http.client.providers.netty.request.NettyConnectListener.operationComplete(NettyConnectListener.java:145)
at
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:409)
at
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:400)
at
org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:362)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:142)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
java.net.ConnectException: connection timed out: at
com.ning.http.client.providers.netty.request.NettyConnectListener.onFutureFailure(NettyConnectListener.java:133)
... 13 more Caused by:
org.jboss.netty.channel.ConnectTimeoutException: connection timed out:
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:139)
... 8 more
Related
i am developing flink application as Kafka source. I have completed all required changes to code, but its falling into following error when run on one emr cluster. Could you guys let me know the workaround if anyone already experienced.
Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:302)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:342)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-xxx-xxx-xxx.dqa.capitalone.com/xx.xxx.xxx.xxx:38257
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
... 19 more
Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-xxx-xxx-xxx.dqa.capitalone.com/xx.xxx.xxx.xxx:38257
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:336)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:750)
EMR Cluster-id:
It's failing with a connection failure, trying to talk to ip-xxx-xxx-xxx.dqa.capitalone.com/xx.xxx.xxx.xxx:38257. It seems likely that internal corporate address is not reachable from inside of AWS.
Context
Spring Batch Application & Postgres are hosted on the same server.
Spring Batch Step is running in a multi-threaded manner executing SELECT queries.
The Spring Batch Step executes correctly for awhile, then randomly hits the below error while executing a query against Postgres.
Error
2021-04-29 20:57:31.974 [SimpleAsyncTaskExecutor-1] ERROR org.springframework.batch.core.step.AbstractStep - Encountered an error executing step ECIP_TN in job ECIP_tom_JOB
com.lit.tom.core.exception.ApplicationException: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection to 127.0.0.1:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at com.lit.tom.core.driver.retriever.sql.SqlRetriever.getData(SqlRetriever.java:98) ~[tom-engine-1.2.11-SNAPSHOT.jar!/:1.2.11-SNAPSHOT]
at com.lit.tom.core.driver.retriever.LegacyDataRetriever.retrieveLegacyRecords(LegacyDataRetriever.java:48) ~[tom-engine-1.2.11-SNAPSHOT.jar!/:1.2.11-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection to 127.0.0.1:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:371) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:452) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:462) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:490) ~[spring-jdbc-5.2.13.RELEASE.jar!/:5.2.13.RELEASE]
at com.lit.tom.core.driver.retriever.sql.SqlRetriever.getData(SqlRetriever.java:59) ~[tom-engine-1.2.11-SNAPSHOT.jar!/:1.2.11-SNAPSHOT]
... 78 more
Caused by: org.postgresql.util.PSQLException: Connection to 127.0.0.1:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:303) ~[postgresql-42.2.19.jar!/:42.2.19]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.2.19.jar!/:42.2.19]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223) ~[postgresql-42.2.19.jar!/:42.2.19]
at org.postgresql.Driver.makeConnection(Driver.java:465) ~[postgresql-42.2.19.jar!/:42.2.19]
at org.postgresql.Driver.connect(Driver.java:264) ~[postgresql-42.2.19.jar!/:42.2.19]
... 78 more
Caused by: java.net.ConnectException: Cannot assign requested address (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_292]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_292]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_292]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_292]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_292]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_292]
Investigative Work
Postgres
Postgres is running, and accepting data on that port.
#Relevant postgres configuration
#listen_addresses = 'localhost'
max_connections = 100
port = 5432
Java
The error is reached faster with more threads.
8, 6, 4, and 2 threads have failed.
Single threaded runs fine.
I'm lost on what to look at, any help would be appreciated.
I discovered the problem, and it was in line with the comments on this post.
The max number of connections was being reached.
The error was coming from a child project, and they had changed the DataSource from HikariCP to the DriverManagerDataSource.
Relevant documentation snippet:
NOTE: This class is not an actual connection pool; it does not actually pool Connections. It just serves as simple replacement for a full-blown connection pool, implementing the same standard interface, but creating new Connections on every call.
The solution was to change back to HikariCP.
I have created a micro Service application and added the APNS library of type "com.notnoop.apns" and version "1.0.0.Beta6".I have deployed this application on GCP on a kubernetes Cluster. The push Notifications were working fine. But sometimes I am getting the below Exception
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.OutputStream.write(OutputStream.java:75)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:328)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:312)
at com.notnoop.apns.internal.ApnsServiceImpl.push(ApnsServiceImpl.java:46)
at com.notnoop.apns.internal.AbstractApnsService.push(AbstractApnsService.java:89)
at com.notnoop.apns.internal.ApnsServiceImpl.push(ApnsServiceImpl.java:36)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
... 22 common frames omitted
This issue is being Reproduced many times now a days and If I am Restarting the Service in google Cloud, Then the Push Notifications are Working Fine but again after sometime I am facing the same issue. I have even checked with the validity of the Certificate and its not Expired yet.
I have a java servlet application running on Tomcat8 over Windows Server 2012 r2.
Oracle 11g r2 is serving on different machine with same subnet, so no firewall exist between java application and oracle.
The java application is multi-threaded and has a haavy load on peak times.
my question is; when everything is alright and working fine below exception occurs.
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor189.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.mulberry.request.MulberryRequestProcessor.run(MulberryRequestProcessor.java:168)
at com.mulberry.request.MulberryRequestProcessor.lambda$run$0(MulberryRequestProcessor.java:78)
at com.mulberry.request.MulberryRequestProcessor$$Lambda$32/461612128.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:743)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:277)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:181)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:699)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:633)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:186)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127)
at com.mulberry.data.connection.ConnectionManager.getConnection(ConnectionManager.java:86)
at com.mulberry.data.MulberryOrm.getConnection(MulberryOrm.java:103)
at com.mulberry.data.MulberryOrm.insertMulti(MulberryOrm.java:201)
at com.mulberry.data.MulberryOrm.upsertMulti(MulberryOrm.java:192)
at com.aril.profile.service.impl.LprBulkdataService.saveLoadProfileData(LprBulkdataService.java:86)
... 11 more
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:506)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:595)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:230)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1452)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:496)
... 25 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:161)
at oracle.net.nt.ConnOption.connect(ConnOption.java:159)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:428)
... 30 more
When this exception occured on oracle site;
dedicated user count : ~5200
(Also more than 1 tomcat instace is running with same purpose)
oracle parameter sessions is setted 12000
oracle running machine is also windows server 2012 with 64 cpu and 96GB ram
I am using tomcat jdbc connection pool and applications getting connections under tomcat pool.
Connection string is
jdbc:oracle:thin:#[IPADDRESS]:1521/[SERVICENAME]
Ethernet between servers are 10gbit so dont think about network delay.
Can anyone help me to detect my problem ?
you can set timeout ;
Properties props = new Properties();
props.setProperty("user", "dbuser");
props.setProperty("password", "dbpassword");
props.setProperty(OracleConnection.CONNECTION_PROPERTY_THIN_NET_CONNECT_TIMEOUT, "2000");
Connection con = DriverManager.getConnection("<JDBC connection string>", props);
I'm seeing below error in WSO2 API Manager 1.6 after running the wso2server.sh for sometime with average load. I got below error repeatedly on the error log.
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 13 more
[2015-03-03 22:50:48,527] INFO - ReceiverGroup Resending the failed events....
[2015-03-03 22:50:48,527] ERROR - EventPublisher Cannot send events to TCP,localhost:7612,TCP,localhost:7712
org.wso2.carbon.databridge.agent.thrift.exception.EventPublisherException: Cannot send Events
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.ThriftEventPublisher.publish(ThriftEventPublisher.java:93)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.EventPublisher.publishEvent(EventPublisher.java:130)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.EventPublisher.run(EventPublisher.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.wso2.carbon.databridge.commons.thrift.service.general.ThriftEventTransmissionService$Client.recv_publish(ThriftEventTransmissionService.java:146)
at org.wso2.carbon.databridge.commons.thrift.service.general.ThriftEventTransmissionService$Client.publish(ThriftEventTransmissionService.java:133)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.ThriftEventPublisher.publish(ThriftEventPublisher.java:86)
... 5 more
What could be the reason and how can I fix this?
It seems like that you have enabled publishing runtime statistics to BAM. From the log i can see that the server is trying to publish to "TCP,localhost:7612,TCP,localhost:7712" which would be thrift ports. So if you don't have a BAM server running with thrift port 7612 this error would come. You can find more information on publishing runtime statistics to BAM here [1].
[1] https://docs.wso2.com/display/AM160/Publishing+API+Runtime+Statistics