I have a few web services on a Weblogic 10 server. Each of these is part of a larger system. Running locally and on our qa environment the system works flawless, replies fast, and as expected. Everything looks to be okay.
Before going into production we're going to stress test the system, thus see how much load we can have before reply time becomes to large. When testing the web services (e.g. using front end or SOAPUI) we hit a certain load (e.g. to many replies per sec or something like that, I'm not sure what exactly triggers the system to fail) we get the error listed below. I haven't got the slightest clue as to why. Seconds later the system replies flawless again, so I'm guessing that it has something to do with the number of requests...
Any ideas or hints is much appreciated! I'm lost here, so please - anything will help.
We're running: Weblogic 10.3.2, Spring 2.5.6 (for architectural reasons we cannot upgrade), Spring-WS 1.5.9 (for architectural reasons we cannot upgrade) and Stripes 1.5.4
<11-11-2011 08:43:58 CET> <Error> <HTTP> <BEA-101017> <[ServletContext#11242741[app:salesoverview-ws-web module:salesoverview-ws-web path:/salesoverview-ws-web spec-version:2.5], request: weblogic.servlet.internal.ServletRequestImpl#1fbbfc5[POST /salesoverview-ws-web/services HTTP/1.1 Accept-Encoding: gzip,deflate Content-Type: text/xml;charset=UTF-8 SOAPAction: "" User-Agent: Jakarta Commons-HttpClient/3.1 Content-Length: 425]] Root cause of ServletException.
org.springframework.ws.soap.saaj.SaajSoapMessageException: Could not write message to OutputStream: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error; nested exception is javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:169)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:97)
at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:57)
at org.springframework.ws.transport.http.MessageDispatcherServlet.doService(MessageDispatcherServlet.java:230)
Truncated. see log file for complete stacktrace
Caused By: javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
at weblogic.xml.saaj.SOAPMessageImpl.SOAPPart_writeTo(SOAPMessageImpl.java:1011)
at weblogic.xml.saaj.SOAPMessageImpl.writeTo(SOAPMessageImpl.java:816)
at org.springframework.ws.soap.saaj.Saaj13Implementation.writeTo(Saaj13Implementation.java:292)
at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:165)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
Truncated. see log file for complete stacktrace
>
By digging BEA-101017 I found a little info about the from the Weblogic error dok - although this doesn't help me:
Error: [context] Root cause of ServletException.
Description: [context] Root cause of ServletException, which the Web
application container caught while servicing the request.
Cause: The Web application container caught an unexpected exception.
Action: Check the exception for the exact error message.
Assuming that the web service from your example doesn't access other web services (and therefore the above trace corresponds to your web service sending the response):
It seems that your web service, via SAAJ, is trying to write to a disconnected (or otherwise unavailable) socket. An usual cause for this is that the client has disconnected while waiting for the server reply.
I'd suggest to:
Check if your client was waiting for too long before receiving the response, that could have caused it to disconnect.
Check if the operating system might be having issues allocating sockets. Use 'netstat' or other monitoring tool (like TCPView on Windows) to check how many sockets are open (most operating system impose limits on the number of sockets allowed per user or globally).
Ensure there are absolutely no network errors during your tests (shouldn't be the case if you are testing on localhost, but otherwise you need to ensure your network devices (routers, switches, other computers) are not dropping connections or packets. Perhaps this is happening when traffic load is high.
Make sure you have no threading conflicts that could cause your web service to use or close other requests' sockets (this would be a rare situation especially if you are using Spring).
Check this thread Official reasons for "Software caused connection abort: socket write error" and other possible causes of "Software caused connection abort" (note that the issue could be specific to your application server and operating system).
Hope that helps.
After debugging a lot I found out that the problem happened due to DB2 issues - we hit a corner of our database, which triggered an internal stack overflow, which then probagated to the Dao and onwards to the SOAP-part (only making it harder to detect due to Spring JDBC templates in the Dao).
A long story short and the issue was an uncaught exception, which by Spring-WS resulted in a "SaajSoapMessageException". The hint came from "Software caused connection abort: socket write error", but happened on the WS side (not client nor the communication between client/server).
Hint: Surround your database with try/catch and catch Exception thus being able to find the exact exception thrown. In my case it threw a DB2 exception ("SQLCode -1218") and this is normally used when you run out of resources (e.g. data source connections). I my case it was the SQL which DB2 didn't like - and really didn't like under load. I can't explain it, but it has to do with DB2s own internal resources - gah, go figure! :)
Thank you jjmontes, for hints and pointers, but it was not the problem in this case.
Related
For a customer we build a HTML5 websocket application with Tomcat 8.0.29.
If we start the application from our network or from our home network all works fine. But if the customer starts the application from his network, after a certain time the websocket stops with a error. This can happen after 5 or 20 Minutes.
We have tested it with and without SSL and on two different servers. Only in the customer network the connection breaks with a error.
We also make a test with an websocket echo example which is included in Tomcat. the same as with our websocket. After a certain time the websocket stops with a error. But only if we start the application from the customer network.
When the echo example stops with an error the following message will written in server.log
08-Dec-2015 10:20:37.757 SEVERE [http-apr-8081-exec-2] org.apache.tomcat.websocket.pojo.PojoEndpointBase.onError No error handling configured for [websocket.echo.EchoAnnotation] and the following error occurred
java.io.IOException: Unexpected error [730,054] reading data from the APR/native socket [1,639,490,672] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper#231e01e4:1639490672].<br/>
at org.apache.coyote.http11.upgrade.AprServletInputStream.doRead(AprServletInputStream.java:133)<br/>
at org.apache.coyote.http11.upgrade.AbstractServletInputStream.read(AbstractServletInputStream.java:124)<br/>
at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:51)<br/>
at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsReadListener.onDataAvailable(WsHttpUpgradeHandler.java:183)<br/>
at org.apache.coyote.http11.upgrade.AbstractServletInputStream.onDataAvailable(AbstractServletInputStream.java:198)<br/>
at org.apache.coyote.http11.upgrade.AbstractProcessor.upgradeDispatch(AbstractProcessor.java:96)<br/>
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:669)<br/>
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2500)<br/>
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2489)<br/>
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<br/>
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<br/>
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)<br/>
at java.lang.Thread.run(Unknown Source)
If we starts the test on http://www.websocket.org/echo.html the connection will not closed with an error.
For me it looks like a problem with Tomcat. But what can I do to make it run properly?
Tomcat: 8.0.29 (also with earlier versions)
Windows 7: 64 Bit
Protocol: HTTP/1.1
I apologize to the readers who came here specifically because of websockets, but we had a similar error in our application while serving files over REST API from Spring. While searching for answers, one of the places I found first was this question. Here's what I was able to find out about the error:
The numbers in Unexpected error [730,054] are somehow significant and are some sort of error codes. In our case, the error looked like this:
java.io.IOException: Unexpected error [120,001] writing data to the APR/native socket [140,041,540,128,928] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper#4f1861c:140041540128928].
I found that this supposedly means that the connection was interrupted by the client.
We were able to solve the issue by reimplementing the controller method to always return the generic ResponseEntity<Resource> object with ByteArrayResource body and content type application/octet-stream:
responseEntity = ResponseEntity.ok()
.headers(createHttpHeaders())
.contentLength(file.length())
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.body(new FileSystemResource(file));
It appears that the errors in our case were caused by non-standard headers we were using, such as Content-Type: application/force-download. It's possible that it caused browsers to terminate the connection in a non-standard way.
I have a java1.6 gateway application runs on aix. During the day millions of messages passing through successfully but when I look at logs I see a few logs like stacktrace below.
log1:
java.net.SocketException: Invalid argument
at java.io.DataOutputStream.write(DataOutputStream.java:119)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:921)
at org.mule.providers.http.HttpServerConnection.writeResponse(HttpServerConnection.java:223)
at com.ibtech.smg.esb.providers.esb.ESBHttpMessageReceiver$NewHttpWorker.run(ESBHttpMessageReceiver.java:162)
at org.mule.impl.work.WorkerContext.run(WorkerContext.java:290)
log2:
java.net.SocketException: Invalid argument
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:125)
at java.net.SocketOutputStream.write(SocketOutputStream.java:171)
at java.io.DataOutputStream.write(DataOutputStream.java:119)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:304)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:308)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:154)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:288)
at java.io.BufferedWriter.flush(BufferedWriter.java:266)
at java.io.FilterWriter.flush(FilterWriter.java:112)
at org.mule.providers.http.ResponseWriter.flush(ResponseWriter.java:75)
I have searched internet but none of the trace or case seem to be fit with mine.
I have tried to generate situation in my development environment (on windows.) Tried big messages, closed socket situations, 0 size messages, but no luck. Can not see the same error. Do anybody has any clue why this error can occur ? This is a bug in application or os bug ?
The problem (java.net.SocketException: Invalid argument) occurs when you try to write a buffer larger than 64K using SocketOutputStream.write().
When you use SocketOutputStream.write(), remenber use SocketOutputStream.flush() to clear SocketOutputStream.
Another situation, connection has closed by server.
(use command 'netstat' to check status of your connection, will be 'CLOSE_WAIT')
When you using SocketOutputStream.write(), log also dispaly "java.net.SocketException: Invalid argument".
REF_DOC
A client application has been built using Jdeveloper 10.1.3.2 and it is running on OC4J server. This application is sending data to external server application. It is working for quite long time without any issue. Lately a connection issue occurred and the following stack trace is generated:
com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.SocketException: Connection reset
at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:133)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:153)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:93)
at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:105)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:629)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:588)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:573)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:470)
at com.sun.xml.ws.client.Stub.process(Stub.java:319)
at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:157)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:140)
at $Proxy44.sendRem(Unknown Source)
After goggling I found out a good discussion about the error sockets - What's causing my java.net.SocketException: Connection reset? .One answer in this link says that the issue mostly from the client side because if it is from the server side the exception will be (SocketException reset by peer).
What I did:
I tried out to increase the socket time out for the OC4J with the help of this form How to change OC4J HTTP Timeout. What I did is I changed the propriety oracle.j2ee.http.socket.timeout to be 5000 instead of 500 (10 times longer)
But the error still there. So, any suggestion to over come this issue?
Note: I able to use telnet command for external server IP and Port and it is working fine.
-------------------------------------------------------- Update 1 --------------------------------------------------------
I increase the server clock skew where the client application is running using the following command on server start up:
-Dweblogic.wsee.security.clock.skew=72000000
-Dweblogic.wsee.security.delay.max=72000000
But no luck, problem is not resolved.
-------------------------------------------------------- Update 2 --------------------------------------------------------
I realized that the problem is not from application at all; I test the external URL using SoapUI and I got the same error Connection rest. I think this new update clreay shows that there is nothing wrong with program code. But I need to know where to go or check now. Where is the starting point now to overcome the issue. Any clue will be helpful.
As you can see from Update 2 in the question, the problem was not from the client application because same error occurred from SoapUI.
The problem was that the machine where the client application was running have low bandwidth which was not enough for APIs communication. Using simple speed test , I found out that the upload bandwidth was low comparing to minimum requirements given by server application team.
I concluded this fact by monitoring the network resource using Resource Monitor in Windows while the client application was running and by using online speed check
To solve the issue, the machine bandwidth has to be increased where the client application is running.
I'm using Soap UI 4.6.0 to hit a WCF web service, and when I have really large message payloads, I'm seeing the following error:
Error getting response; java.net.SocketException: Connection reset
The WCF service has around 10 methods, each with progressively larger inputs (eg, 10 int properties, 50 int properties, 100 int properties, etc). This works with the smaller messages, but as they get around 2000-3000 int properties, the error occurs.
The call appears to succeed on the server side, and with this coming from java, I'm assuming I'm butting up against some size limitation/configuration in the client. Is this something I can tweak within Soap UI, the java runtime, or elsewhere?
For me the trick that worked was adding below entry in SoapUI-5.2.0.vmoptions file (it can be found in the bin directory of installa
-Dsoapui.https.protocols=SSLv3,TLSv1.2
Normally a connection reset means that one of the underlying servers timed out waiting for data from another server/application and it reset the connection.
You should try out the suggestions #kroonwijk gave it'll tell you which server is causing the reset and what is causing the server to reset the connection.
Also see What's causing my java.net.SocketException: Connection reset?
If above solutions won't work for you then try this:
Close SoapUI
Go to SoapUi directory for example: C:\Program Files\SmartBear\SoapUI-5.3.0\
Rename directory "jre" to "jre.ignore"
Done. Open SoapUi and it should work now.
Today I managed to recreate the farms with Scalr.net and apparently after a few times restarting tomcat and fixing issues, I get this error once again. The thing is I was using MySQL with a clean install on the entire server, that includes Java 6.1_24, Tomcat 5.5.33, Sakai 2.7.1. The issue I keep running into is user denied when the fact that I have this user in the MySQL Instance, as well giving it complete remote access with sakai#% and even this is not working when it was working about an hour ago since this post was made.
... Continued from above log, everything before logs just fine
2011-03-31 18:31:14,120 WARN main org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy - Could not retrieve default auto-commit and transaction isolation settings
org.apache.commons.dbcp.SQLNestedException: Error preloading the connection pool
... continued over 400+ lines...
Here is another error in regards to the access denied error...
2011-03-31 18:31:16,854 WARN main org.hibernate.cfg.SettingsFactory - Could not obtain connection metadata
java.sql.SQLException: Access denied for user 'sakai'#'ec2-50-17-184-70.compute-1.amazonaws.com' (using password: YES)
.... continued....
I now get this error whenever I startup, this is with a fresh install of tomcat/sakai
SEVERE: Unable to set localhost. This prevents creation of a GUID. Cause was: ec2-72-44-56-167.compute-1.amazonaws.com: ec2-72-44-56-167.compute-1.amazonaws.com
java.net.UnknownHostException: ec2-72-44-56-167.compute-1.amazonaws.com: ec2-72-44-56-167.compute-1.amazonaws.com
(This most recent error (Localhost) was simply fixed by restarting the amazon aws instance. Thankfully) Although I keep getting the same errors even with a fresh install... Almost as if the information is being refreshed from a cache... Or something
As with the last question you posted on this topic, the error message seems very clear: the user 'sakai'#... does not have access to login to the database you have set it up to. I recommend taking a look at the Mysql documentation to understand how to administer the user accounts to find out if you've missed a setting somewhere to allow this account to have access.
I believe I may have figured out how to fix this problem. It has nothing to do with mysql, or the apache server itself. It has to do with the failure of Scalr.net not Initializing the IP or something of that sort. After doing some research I found some issues with the HostInit issues such as....
Cannot deliver message 'HostInit' (message_id: af9dcfdb-a09e-4971-bdb7-7871b3f7e21c) via REST to server '50.17.135.98' (server_id: e49cfec9-5bcb-44d1-bbc5-fde32450fc89). Error: 0 Timeout was reached; connect() timed out! (http://50.17.135.98:8013/control)
Cannot deliver message 'BlockDeviceAttached' (message_id: a153d83f-3d96-4d53-920a-ccb80701675a) via REST to server '50.17.135.98' (server_id: e49cfec9-5bcb-44d1-bbc5-fde32450fc89). Error: 0 Timeout was reached; connect() timed out! (http://50.17.135.98:8013/control)
Cannot deliver message 'HostUp' (message_id: 1adde27c-9982-4551-b266-c3c432d1dd44) via REST to server '50.17.135.98' (server_id: e49cfec9-5bcb-44d1-bbc5-fde32450fc89). Error: 0 Timeout was reached; connect() timed out! (http://50.17.135.98:8013/control)
Cannot deliver message 'HostInit' (message_id: f1aa4b14-ef57-4361-ae56-87702d674b11) via REST to server '50.17.135.98' (server_id: e49cfec9-5bcb-44d1-bbc5-fde32450fc89). Error: 0 Timeout was reached; connect() timed out! (http://50.17.135.98:8013/control)
So what I did was I made a snapshot image of the apache server/mysql etc. and terminated them allowing the recreation of the instance and this managed to solve the problem in one manner.