I'm seeing below error in WSO2 API Manager 1.6 after running the wso2server.sh for sometime with average load. I got below error repeatedly on the error log.
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 13 more
[2015-03-03 22:50:48,527] INFO - ReceiverGroup Resending the failed events....
[2015-03-03 22:50:48,527] ERROR - EventPublisher Cannot send events to TCP,localhost:7612,TCP,localhost:7712
org.wso2.carbon.databridge.agent.thrift.exception.EventPublisherException: Cannot send Events
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.ThriftEventPublisher.publish(ThriftEventPublisher.java:93)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.EventPublisher.publishEvent(EventPublisher.java:130)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.EventPublisher.run(EventPublisher.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.wso2.carbon.databridge.commons.thrift.service.general.ThriftEventTransmissionService$Client.recv_publish(ThriftEventTransmissionService.java:146)
at org.wso2.carbon.databridge.commons.thrift.service.general.ThriftEventTransmissionService$Client.publish(ThriftEventTransmissionService.java:133)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.ThriftEventPublisher.publish(ThriftEventPublisher.java:86)
... 5 more
What could be the reason and how can I fix this?
It seems like that you have enabled publishing runtime statistics to BAM. From the log i can see that the server is trying to publish to "TCP,localhost:7612,TCP,localhost:7712" which would be thrift ports. So if you don't have a BAM server running with thrift port 7612 this error would come. You can find more information on publishing runtime statistics to BAM here [1].
[1] https://docs.wso2.com/display/AM160/Publishing+API+Runtime+Statistics
Related
i am developing flink application as Kafka source. I have completed all required changes to code, but its falling into following error when run on one emr cluster. Could you guys let me know the workaround if anyone already experienced.
Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:302)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:342)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-xxx-xxx-xxx.dqa.capitalone.com/xx.xxx.xxx.xxx:38257
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
... 19 more
Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-xxx-xxx-xxx.dqa.capitalone.com/xx.xxx.xxx.xxx:38257
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:336)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:750)
EMR Cluster-id:
It's failing with a connection failure, trying to talk to ip-xxx-xxx-xxx.dqa.capitalone.com/xx.xxx.xxx.xxx:38257. It seems likely that internal corporate address is not reachable from inside of AWS.
We have Sprintboot application on Azure Linux VM which is connecting to IBM MQ 8 and connection goes via KPN.
Recently we have upgraded IBM MQ from 8 to 9. No settings are changed on application, but with IBM MQ 9 - we can see that post 30 mins - connections are getting timed out and new connections are getting created post every 30 mins.
Checked with IBM MQ 9 team - no timeout setting at their end.
Error -
[com.ibm.mq.allclient-9.2.3.0.jar!/:9.2.3.0 - p923-L210708]
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9208: Error on receive from host 'Host Details'. [1=javax.net.ssl.SSLException[Connection timed out (Read failed)],3=,4=TCP,5=InputStream.read]
Caused by: javax.net.ssl.SSLException: Connection timed out (Read failed)
Suppressed: java.net.SocketException: Broken pipe (Write failed)
enter code here`Caused by: java.net.SocketException: Connection timed out (Read failed)
I have created a micro Service application and added the APNS library of type "com.notnoop.apns" and version "1.0.0.Beta6".I have deployed this application on GCP on a kubernetes Cluster. The push Notifications were working fine. But sometimes I am getting the below Exception
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.OutputStream.write(OutputStream.java:75)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:328)
at com.notnoop.apns.internal.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:312)
at com.notnoop.apns.internal.ApnsServiceImpl.push(ApnsServiceImpl.java:46)
at com.notnoop.apns.internal.AbstractApnsService.push(AbstractApnsService.java:89)
at com.notnoop.apns.internal.ApnsServiceImpl.push(ApnsServiceImpl.java:36)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
... 22 common frames omitted
This issue is being Reproduced many times now a days and If I am Restarting the Service in google Cloud, Then the Push Notifications are Working Fine but again after sometime I am facing the same issue. I have even checked with the validity of the Certificate and its not Expired yet.
I am trying to connect to a remote server and have set the connect timeout to 120 seconds. Sometimes it connects, while sometimes it throws us Connect timeout even before trying for 2 mins.
setConnectTimeout(120000);
setReadTimeout(120000);
I am using AsyncHttpClientConfig for setting these properties.
Below is the error :
java.util.concurrent.ExecutionException: java.net.ConnectException:
connection timed out: at
com.ning.http.client.providers.netty.future.NettyResponseFuture.abort(NettyResponseFuture.java:231)
at
com.ning.http.client.providers.netty.request.NettyConnectListener.onFutureFailure(NettyConnectListener.java:137)
at
com.ning.http.client.providers.netty.request.NettyConnectListener.operationComplete(NettyConnectListener.java:145)
at
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:409)
at
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:400)
at
org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:362)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:142)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
java.net.ConnectException: connection timed out: at
com.ning.http.client.providers.netty.request.NettyConnectListener.onFutureFailure(NettyConnectListener.java:133)
... 13 more Caused by:
org.jboss.netty.channel.ConnectTimeoutException: connection timed out:
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:139)
... 8 more
WSO2 ESB sometimes lost JMS connection to Websphere MQ. Establishing new connection last at least 1-2 seconds. This cause timeouts in proxy service using this connection.
WSO2 ESB 4.8.1, IBM Webpshere MQ 7.0.1.12
Strange stacktrace:
2014-12-18 06:47:51,183 [-] [JMSCCThreadPoolWorker-7] ERROR ServiceTaskManager JMS Connection failed : JMSWMQ1107: A problem with this connection has occurred. - shutting down worker tasks
2014-12-18 06:47:52,212 [-] [JMSCCThreadPoolWorker-7] ERROR ServiceTaskManager Error closing shared Connection
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0019: Failed to disconnect from queue manager 'name' using connection mode '1' and host name 'host(1441)'. Please see the linked exception for more information.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:608)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)
at com.ibm.msg.client.wmq.internal.WMQConnection.close(WMQConnection.java:742)
at com.ibm.msg.client.jms.internal.JmsConnectionImpl.close(JmsConnectionImpl.java:352)
at com.ibm.mq.jms.MQConnection.close(MQConnection.java:93)
at org.apache.axis2.transport.jms.ServiceTaskManager.stop(ServiceTaskManager.java:237)
at org.apache.axis2.transport.jms.ServiceTaskManager.start(ServiceTaskManager.java:166)
at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.onException(ServiceTaskManager.java:631)
at com.ibm.msg.client.jms.internal.JmsProviderExceptionListener.run(JmsProviderExceptionListener.java:429)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:100)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:298)
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1220)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)
... 12 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009
at com.ibm.mq.jmqi.remote.internal.RemoteHconn.enterCall(RemoteHconn.java:428)
at com.ibm.mq.jmqi.remote.internal.RemoteHconn.enterCall(RemoteHconn.java:350)
at com.ibm.mq.jmqi.remote.internal.RemoteHconn.enterCall(RemoteHconn.java:325)
at com.ibm.mq.jmqi.remote.internal.RemoteFAP.MQDISC(RemoteFAP.java:2541)
at com.ibm.msg.client.wmq.internal.WMQConnection.close(WMQConnection.java:724)
... 11 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9213: A communications error for 'TCP' occurred. [1=java.net.SocketException[Unrecognized Windows Sockets error: 0: recv failed],4=TCP,5=sockInStream.read]
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.receive(RemoteTCPConnection.java:1418)
at com.ibm.mq.jmqi.remote.internal.RemoteRcvThread.receiveBuffer(RemoteRcvThread.java:737)
at com.ibm.mq.jmqi.remote.internal.RemoteRcvThread.receiveOneTSH(RemoteRcvThread.java:701)
at com.ibm.mq.jmqi.remote.internal.RemoteRcvThread.run(RemoteRcvThread.java:146)
... 5 more
Caused by: java.net.SocketException: Unrecognized Windows Sockets error: 0: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.receive(RemoteTCPConnection.java:1409)
... 8 more
2014-12-18 06:49:53,302 [-] [DefaultQuartzScheduler_Worker-1] ERROR MessageInjector Proxy Service: monitoring_Service not found
2014-12-18 06:49:53,302 [-] [DefaultQuartzScheduler_Worker-1] ERROR JobRunShell Job synapse.simple.quartz.monitoring_Task threw an unhandled Exception:
org.apache.synapse.SynapseException: Proxy Service: monitoring_Service not found
at org.apache.synapse.startup.tasks.MessageInjector.handleError(MessageInjector.java:333)
at org.apache.synapse.startup.tasks.MessageInjector.execute(MessageInjector.java:217)
at org.apache.synapse.startup.quartz.SimpleQuartzJob.execute(SimpleQuartzJob.java:88)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
2014-12-18 06:49:53,302 [-] [DefaultQuartzScheduler_Worker-1] ERROR ErrorLogger Job (synapse.simple.quartz.monitoring_Task threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.apache.synapse.SynapseException: Proxy Service: monitoring_Service not found]
at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: org.apache.synapse.SynapseException: Proxy Service: monitoring_Service not found
at org.apache.synapse.startup.tasks.MessageInjector.handleError(MessageInjector.java:333)
at org.apache.synapse.startup.tasks.MessageInjector.execute(MessageInjector.java:217)
at org.apache.synapse.startup.quartz.SimpleQuartzJob.execute(SimpleQuartzJob.java:88)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
... 1 more
The key error you're seeing within the stacktrace is:
Caused by: java.net.SocketException: Unrecognized Windows Sockets error: 0: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
Which indicates that the JVM received something it wasn't expecting over a Windows socket. It looks like you're using an Oracle JVM rather than the IBM one shipped with MQ.
I'd suggest you try upgrading to a newer JVM if one is available. Note that the MQ 7.0.1 client you're using supports Java 5 and Java 6. If you want to use Java 7 you'll need to use an MQ client of 7.1 or later. All IBM MQ clients are available free of charge via the MQC support packs:
MQ 7.1 clients: http://www-01.ibm.com/support/docview.wss?uid=swg24031412
MQ 7.5 clients: http://www-01.ibm.com/support/docview.wss?uid=swg24032744
MQ 8.0 clients: http://www-01.ibm.com/support/docview.wss?uid=swg24037500
If a newer JVM isn't available from Oracle, I'd suggest opening a support ticket with them so they can investigate.