For a customer we build a HTML5 websocket application with Tomcat 8.0.29.
If we start the application from our network or from our home network all works fine. But if the customer starts the application from his network, after a certain time the websocket stops with a error. This can happen after 5 or 20 Minutes.
We have tested it with and without SSL and on two different servers. Only in the customer network the connection breaks with a error.
We also make a test with an websocket echo example which is included in Tomcat. the same as with our websocket. After a certain time the websocket stops with a error. But only if we start the application from the customer network.
When the echo example stops with an error the following message will written in server.log
08-Dec-2015 10:20:37.757 SEVERE [http-apr-8081-exec-2] org.apache.tomcat.websocket.pojo.PojoEndpointBase.onError No error handling configured for [websocket.echo.EchoAnnotation] and the following error occurred
java.io.IOException: Unexpected error [730,054] reading data from the APR/native socket [1,639,490,672] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper#231e01e4:1639490672].<br/>
at org.apache.coyote.http11.upgrade.AprServletInputStream.doRead(AprServletInputStream.java:133)<br/>
at org.apache.coyote.http11.upgrade.AbstractServletInputStream.read(AbstractServletInputStream.java:124)<br/>
at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:51)<br/>
at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsReadListener.onDataAvailable(WsHttpUpgradeHandler.java:183)<br/>
at org.apache.coyote.http11.upgrade.AbstractServletInputStream.onDataAvailable(AbstractServletInputStream.java:198)<br/>
at org.apache.coyote.http11.upgrade.AbstractProcessor.upgradeDispatch(AbstractProcessor.java:96)<br/>
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:669)<br/>
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2500)<br/>
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2489)<br/>
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<br/>
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<br/>
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)<br/>
at java.lang.Thread.run(Unknown Source)
If we starts the test on http://www.websocket.org/echo.html the connection will not closed with an error.
For me it looks like a problem with Tomcat. But what can I do to make it run properly?
Tomcat: 8.0.29 (also with earlier versions)
Windows 7: 64 Bit
Protocol: HTTP/1.1
I apologize to the readers who came here specifically because of websockets, but we had a similar error in our application while serving files over REST API from Spring. While searching for answers, one of the places I found first was this question. Here's what I was able to find out about the error:
The numbers in Unexpected error [730,054] are somehow significant and are some sort of error codes. In our case, the error looked like this:
java.io.IOException: Unexpected error [120,001] writing data to the APR/native socket [140,041,540,128,928] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper#4f1861c:140041540128928].
I found that this supposedly means that the connection was interrupted by the client.
We were able to solve the issue by reimplementing the controller method to always return the generic ResponseEntity<Resource> object with ByteArrayResource body and content type application/octet-stream:
responseEntity = ResponseEntity.ok()
.headers(createHttpHeaders())
.contentLength(file.length())
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.body(new FileSystemResource(file));
It appears that the errors in our case were caused by non-standard headers we were using, such as Content-Type: application/force-download. It's possible that it caused browsers to terminate the connection in a non-standard way.
Related
My company’s web application is using GWT both for front-end and back-end and we’d like to remove GWT from the project. Our current objective would be to make a new login page in Angular2 using the existing GWT back-end.
The main problem we encounter at the moment is that all the requests coming from outside the current GWT front-end seem to be intercepted/blocked and don’t return anything. I’m currently using Postman to make GET requests from the server.
Here is the URL I’m calling : http://localhost:9997/RestServer/api/users/1.
Postman’s response:
« Could not get any response
There was an error connecting to http://localhost:9997/RestServer/api/users/1. »
GWT Development Mode’s console returns an error:
[TRACE] Connection received from 127.0.0.1:60296
[ERROR] Unrecognized command for client; closing connection
com.google.gwt.dev.shell.BrowserChannelException: Invalid message type 71
at com.google.gwt.dev.shell.BrowserChannel$Message.readMessageType(BrowserChannel.java:1135)
at com.google.gwt.dev.shell.BrowserChannelServer.processConnection(BrowserChannelServer.java:248)
at com.google.gwt.dev.shell.BrowserChannelServer.run(BrowserChannelServer.java:222)
at java.lang.Thread.run(Thread.java:745)
Have you got any idea why my requests are intercepted? Why I can't even receive a http response?
If you need more informations to answer just ask for it.
Port 9997 in the old dev mode is not an HTTP server. This is the port on which the legacy browser plugin (unsupported in modern FF and Chrome due to breaking changes in those browsers) would connect to run Java code and enable remote debugging, hotswapping, etc.
Instead, you need to connect to the HTTP port, which is 8888 or 8080 or something like that.
First of all, my error is almost identical to what is reported in this question: WSDLException : An error occurred trying to resolve schema referenced at
Here is a snippet of my stack dump:
javax.wsdl.WSDLException: WSDLException (at /definitions/types/xs:schema/xs:schema): faultCode=OTHER_ERROR: An error occurred trying to resolve schema referenced at 'http://www.w3.org/2005/05/xmlmime', relative to 'http://server.subdom.domain.com:13080/SM/7/Common.xsd'.: java.net.ConnectException: Connection timed out: connect] MDC{}
2015-05-24 14:36:33,751 ERROR (c.d.g.w.c.ContexteApplicatif.contextInitialized) [main] catching MDC{}
javax.xml.rpc.ServiceException: Error processing WSDL document:
javax.wsdl.WSDLException: WSDLException (at /definitions/types/xs:schema/xs:schema): faultCode=OTHER_ERROR: An error occurred trying to resolve schema referenced at 'http://www.w3.org/2005/05/xmlmime', relative to 'http://server.subdom.domain.com:13080/SM/7/Common.xsd'.: java.net.ConnectException: Connection timed out: connect
at org.apache.axis.client.Service.initService(Service.java:250) ~[axis-1.4.jar:?]
This occurs in my embedded Tomcat server running from within Eclipse. It is running on a Windows machine and there is an httpProxy at the system level. However, the URL is an internal address for which no proxy is needed. Anyhow, I implemented programmatically a proxy with the following code just before the reference to the WSDL file:
System.setProperty("http.proxySet", "true");
System.setProperty("http.proxyHost", "proxyhost.subdom.domain.com");
System.setProperty("http.proxyPort", "8080");
System.setProperty("https.proxyHost", "proxyhost.subdom.domain.com");
System.setProperty("https.proxyPort", "8080");
And now I am getting an HTTP 502 error which indicate a bad gateway. So, I suppose this solution is the wrong one since I shouldn't need a proxy in first place. I can access the page from within a browser, indistinctly if I enable or disable the proxy settings. In addition, there is a script to configure the proxy and if I use the proxy host shown above and hardcode it in my browser instead of "system proxy" or "automatic setting" I cannot access the page.
To summarize, it behaves like there is like it needs a proxy setup, however it doesn't. The problem is elsewhere and I have no idea how I can make significant progress to debug this problem.
Any hints? Something with Tomcat? Something with Eclipse?
I haven't tried yet on a standalone Tomcat server since my code is not yet ready for deployement.
NOTE: BTW, I tried the command from the quoted post and I am getting the same error as well. Connect timed out without system properties defined for the proxy and 502 code otherwise. At the same time, if I am launching the Web Service Explorer from Eclipse I am perfectly able to access the webservice and invoke operations.
Further investigation: I decided to use WireShark to see what is going on with the request and it appears both HTTP requests (the wsdl and the common types definitions) were fulfilled without a glitch and no connection timed out at all, not a single error. I can see the XML in WireShark and the HTTP status is 200 OK and everything is perfectly fine at this level.
So, what is going on here? I have the same problem on a Linux server while the message is a bit different. What wsdl2java is doing to believe there was a problem and abort?
After further investigation and testing with Axis2 and CXF, I finally found the problem, thanks to the CXF's version of the wsdl2java script which is giving a bit more details.
First of all, the original solution proposed was almost correct. I actually need to add all the proxy information, however I also needed to specify the non-proxy hosts otherwise I am getting the 502 error. The messages from the Axis script were not very detailed about the offending request, while CXF's version was very clear and enabled me to finally solve my problem.
So, in addition, if you modify the wsdl2java script, add -Dhttp.nonProxyHosts=... in addition to other options. The same thing if you need to specify a proxy programmatically.
A client application has been built using Jdeveloper 10.1.3.2 and it is running on OC4J server. This application is sending data to external server application. It is working for quite long time without any issue. Lately a connection issue occurred and the following stack trace is generated:
com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.SocketException: Connection reset
at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:133)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:153)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:93)
at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:105)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:629)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:588)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:573)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:470)
at com.sun.xml.ws.client.Stub.process(Stub.java:319)
at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:157)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:140)
at $Proxy44.sendRem(Unknown Source)
After goggling I found out a good discussion about the error sockets - What's causing my java.net.SocketException: Connection reset? .One answer in this link says that the issue mostly from the client side because if it is from the server side the exception will be (SocketException reset by peer).
What I did:
I tried out to increase the socket time out for the OC4J with the help of this form How to change OC4J HTTP Timeout. What I did is I changed the propriety oracle.j2ee.http.socket.timeout to be 5000 instead of 500 (10 times longer)
But the error still there. So, any suggestion to over come this issue?
Note: I able to use telnet command for external server IP and Port and it is working fine.
-------------------------------------------------------- Update 1 --------------------------------------------------------
I increase the server clock skew where the client application is running using the following command on server start up:
-Dweblogic.wsee.security.clock.skew=72000000
-Dweblogic.wsee.security.delay.max=72000000
But no luck, problem is not resolved.
-------------------------------------------------------- Update 2 --------------------------------------------------------
I realized that the problem is not from application at all; I test the external URL using SoapUI and I got the same error Connection rest. I think this new update clreay shows that there is nothing wrong with program code. But I need to know where to go or check now. Where is the starting point now to overcome the issue. Any clue will be helpful.
As you can see from Update 2 in the question, the problem was not from the client application because same error occurred from SoapUI.
The problem was that the machine where the client application was running have low bandwidth which was not enough for APIs communication. Using simple speed test , I found out that the upload bandwidth was low comparing to minimum requirements given by server application team.
I concluded this fact by monitoring the network resource using Resource Monitor in Windows while the client application was running and by using online speed check
To solve the issue, the machine bandwidth has to be increased where the client application is running.
I am using some third party code which is actually a web service client. It works fine when I run it simply on JDK or even on Tomcat. But as soon as I integrated it in my web project for production phase with weblogic server, it started to act abnormally with an error FileNotFoundException which I haven't been able to solve.
As far as I understand Tomcat is for development phases and therefore sort of lenient, that's why I didn't encounter any problem earlier. But weblogicadds restrictions in its own way.
If I am unable to run that web service client on weblogic, I need to keep this piece of code outside the production server environment and run it on simple JDK. Now to access this external code, should I need to use Remote Method Invocation or is there any simpler solution? I have added full stack trace just by replacing actual URL with WEBSERVICEURL. Hope this wouldn't affect the answers.
Exception in thread "Timer-5" javax.xml.ws.WebServiceException: Failed to access the WSDL at: https://cert.hub.care360.com/orders/submission/service?wsdl. It failed with:
Response: '401: Unauthorized' for url: 'WEBSERVICEURL'.
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.tryWithMex(RuntimeWSDLParser.java:172)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:153)
at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:273)
at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:236)
at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:184)
at weblogic.wsee.jaxws.spi.WLSServiceDelegate.<init>(WLSServiceDelegate.java:42)
at weblogic.wsee.jaxws.spi.WLSProvider$ServiceDelegate.<init>(WLSProvider.java:206)
at weblogic.wsee.jaxws.spi.WLSProvider.createServiceDelegate(WLSProvider.java:57)
at weblogic.wsee.jaxws.spi.WLSProvider.createServiceDelegate(WLSProvider.java:41)
at javax.xml.ws.Service.<init>(Service.java:56)
at com.medplus.orders.OrderSubmissionService.<init>(OrderSubmissionService.java:42)
at com.adamsoft.edoc.util.QuestOrderService.sendOrder(QuestOrderService.java:137)
at com.adamsoft.edoc.util.QuestOrderService.run(QuestOrderService.java:308)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.FileNotFoundException: Response: '401: Unauthorized' for url: 'WEBSERVICEURL'
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:474)
at weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
at java.net.URL.openStream(URL.java:1010)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.createReader(RuntimeWSDLParser.java:843)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(RuntimeWSDLParser.java:289)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:138)
at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:273)
at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:237)
at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:187)
at weblogic.wsee.jaxws.spi.WLSServiceDelegate.<init>(WLSServiceDelegate.java:36)
at weblogic.wsee.jaxws.spi.WLSProvider$ServiceDelegate.<init>(WLSProvider.java:207)
at weblogic.wsee.jaxws.spi.WLSProvider.createServiceDelegate(WLSProvider.java:57)
at weblogic.wsee.jaxws.spi.WLSProvider.createServiceDelegate(WLSProvider.java:41)
at javax.xml.ws.Service.<init>(Service.java:56)
at com.medplus.orders.OrderSubmissionService.<init>(OrderSubmissionService.java:43)
at com.adamsoft.edoc.util.QuestOrderService.sendOrder(QuestOrderService.java:137)
at com.adamsoft.edoc.util.QuestOrderService.run(QuestOrderService.java:310)
... 2 more
I have a few web services on a Weblogic 10 server. Each of these is part of a larger system. Running locally and on our qa environment the system works flawless, replies fast, and as expected. Everything looks to be okay.
Before going into production we're going to stress test the system, thus see how much load we can have before reply time becomes to large. When testing the web services (e.g. using front end or SOAPUI) we hit a certain load (e.g. to many replies per sec or something like that, I'm not sure what exactly triggers the system to fail) we get the error listed below. I haven't got the slightest clue as to why. Seconds later the system replies flawless again, so I'm guessing that it has something to do with the number of requests...
Any ideas or hints is much appreciated! I'm lost here, so please - anything will help.
We're running: Weblogic 10.3.2, Spring 2.5.6 (for architectural reasons we cannot upgrade), Spring-WS 1.5.9 (for architectural reasons we cannot upgrade) and Stripes 1.5.4
<11-11-2011 08:43:58 CET> <Error> <HTTP> <BEA-101017> <[ServletContext#11242741[app:salesoverview-ws-web module:salesoverview-ws-web path:/salesoverview-ws-web spec-version:2.5], request: weblogic.servlet.internal.ServletRequestImpl#1fbbfc5[POST /salesoverview-ws-web/services HTTP/1.1 Accept-Encoding: gzip,deflate Content-Type: text/xml;charset=UTF-8 SOAPAction: "" User-Agent: Jakarta Commons-HttpClient/3.1 Content-Length: 425]] Root cause of ServletException.
org.springframework.ws.soap.saaj.SaajSoapMessageException: Could not write message to OutputStream: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error; nested exception is javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:169)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:97)
at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:57)
at org.springframework.ws.transport.http.MessageDispatcherServlet.doService(MessageDispatcherServlet.java:230)
Truncated. see log file for complete stacktrace
Caused By: javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
at weblogic.xml.saaj.SOAPMessageImpl.SOAPPart_writeTo(SOAPMessageImpl.java:1011)
at weblogic.xml.saaj.SOAPMessageImpl.writeTo(SOAPMessageImpl.java:816)
at org.springframework.ws.soap.saaj.Saaj13Implementation.writeTo(Saaj13Implementation.java:292)
at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:165)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
Truncated. see log file for complete stacktrace
>
By digging BEA-101017 I found a little info about the from the Weblogic error dok - although this doesn't help me:
Error: [context] Root cause of ServletException.
Description: [context] Root cause of ServletException, which the Web
application container caught while servicing the request.
Cause: The Web application container caught an unexpected exception.
Action: Check the exception for the exact error message.
Assuming that the web service from your example doesn't access other web services (and therefore the above trace corresponds to your web service sending the response):
It seems that your web service, via SAAJ, is trying to write to a disconnected (or otherwise unavailable) socket. An usual cause for this is that the client has disconnected while waiting for the server reply.
I'd suggest to:
Check if your client was waiting for too long before receiving the response, that could have caused it to disconnect.
Check if the operating system might be having issues allocating sockets. Use 'netstat' or other monitoring tool (like TCPView on Windows) to check how many sockets are open (most operating system impose limits on the number of sockets allowed per user or globally).
Ensure there are absolutely no network errors during your tests (shouldn't be the case if you are testing on localhost, but otherwise you need to ensure your network devices (routers, switches, other computers) are not dropping connections or packets. Perhaps this is happening when traffic load is high.
Make sure you have no threading conflicts that could cause your web service to use or close other requests' sockets (this would be a rare situation especially if you are using Spring).
Check this thread Official reasons for "Software caused connection abort: socket write error" and other possible causes of "Software caused connection abort" (note that the issue could be specific to your application server and operating system).
Hope that helps.
After debugging a lot I found out that the problem happened due to DB2 issues - we hit a corner of our database, which triggered an internal stack overflow, which then probagated to the Dao and onwards to the SOAP-part (only making it harder to detect due to Spring JDBC templates in the Dao).
A long story short and the issue was an uncaught exception, which by Spring-WS resulted in a "SaajSoapMessageException". The hint came from "Software caused connection abort: socket write error", but happened on the WS side (not client nor the communication between client/server).
Hint: Surround your database with try/catch and catch Exception thus being able to find the exact exception thrown. In my case it threw a DB2 exception ("SQLCode -1218") and this is normally used when you run out of resources (e.g. data source connections). I my case it was the SQL which DB2 didn't like - and really didn't like under load. I can't explain it, but it has to do with DB2s own internal resources - gah, go figure! :)
Thank you jjmontes, for hints and pointers, but it was not the problem in this case.