Amazon sqs throwing exception connection reset - java

I am using AmazonSQSAsyncClient to connect with Amazon SQS, but sometimes I see the following execution in the logs:
INFO [AmazonHttpClient:444] Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:712)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:380)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2169)
at com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:468)
at com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:1476)
I am using the AmazonSQSAsyncClient through out the application as singleton.
Code snippet is below.
static{
if(sqsObj == null){
sqsObj = new AmazonSQSAsyncClient(new ClasspathPropertiesFileCredentialsProvider("app.properties"));
sqsObj.setRegion(Region.getRegion(Regions.valueOf("sample region"));
}
}
By using sqsobj, I am doing operations like create queue, send message and receive messages. It works fine but some time throws the exception as above. After restarting the application it works fine for sometime.
I am using aws-java-sdk-1.7.1.
Please suggest on this.

The "connection reset" means that the other party (i.e. the server) closed the connection.
Internally the SDK builds and reuses a http connection to not have to open a connection every time you make a request.
The log message tells you that the connection was reset, but normally this is not something to worry about. The SDK will retry automatically in cases like this. So if you are only seeing this sporadically in the logs but everything works as expected you should probably not worry about it

Related

java.net.SocketException: Connection Reset 2 [duplicate]

This question already has answers here:
java.net.SocketException: Connection reset
(14 answers)
Closed 3 years ago.
We are seeing frequent but intermittent java.net.SocketException: Connection reset errors in our logs. We are unsure as to where the Connection reset error is actually coming from, and how to go about debugging.
The issue appears to be unrelated to the messages we are attempting to send.
Note that the message is not connection reset by peer.
Any suggestions on what the typical causes of this exception might be, and how we might proceed?
Here is a representative stack trace (com.companyname.mtix.sms is our component):
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at com.companyname.mtix.sms.services.impl.message.SendTextMessage.sendTextMessage(SendTextMessage.java:127)
at com.companyname.mtix.sms.services.MessageServiceImpl.sendTextMessage(MessageServiceImpl.java:125)
at com.companyname.mtix.sms.services.remote.MessageServiceRemoteImpl.sendTextMessage(MessageServiceRemoteImpl.java:43)
at sun.reflect.GeneratedMethodAccessor203.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:453)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.companyname.mtix.sms.http.filters.NoCacheFilter.doFilter(NoCacheFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.companyname.mtix.sms.http.filters.MessageFilter.doFilter(MessageFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:61)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.ajaxanywhere.AAFilter.doFilter(AAFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Our component is a web application, running under Tomcat, that calls a third party Web service that sends SMS messages, it so happens. The line of our code on which the exception gets thrown from is the last line in the code snippet below.
String aggregatorResponse = null;
HttpClient httpClient = prepareHttpClient( username, password );
PostMethod postMethod = preparePostMethod( textUrl );
try {
SybaseTextMessageBuilder builder = new SybaseTextMessageBuilder();
URL notifyUrl = buildNotificationUrl( textMessage, codeSetManager );
String smsRequestDocument = builder.buildTextMessage( textMessage, notifyUrl );
LOG.debug( "Sybase MT document created as: \n" + smsRequestDocument );
postMethod.setRequestEntity( new StringRequestEntity( smsRequestDocument ) );
LOG.debug( "commiting SMS to aggregator: " + textMessage.toString() );
int httpStatus = httpClient.executeMethod( postMethod );
The javadoc for SocketException states that it is
Thrown to indicate that there is an error in the underlying protocol such as a TCP error
In your case it seems that the connection has been closed by the server end of the connection. This could be an issue with the request you are sending or an issue at their end.
To aid debugging you could look at using a tool such as Wireshark to view the actual network packets. Also, is there an alternative client to your Java code that you could use to test the web service? If this was successful it could indicate a bug in the Java code.
As you are using Commons HTTP Client have a look at the Common HTTP Client Logging Guide. This will tell you how to log the request at the HTTP level.
This error happens on your side and NOT the other side. If the other side reset the connection, then the exception message should say:
java.net.SocketException reset by peer
The cause is the connection inside HttpClient is stale. Check stale connection for SSL does not fix this error. Solution: dump your client and recreate.
If you experience this trying to access Web services deployed on a Glassfish3 server, you might want to tune your http-thread-pool settings. That fixed SocketExceptions we had when many concurrent threads was calling the web service.
Go to admin console
Navigate to "Configurations"->"Server config"->"Thread pools"->"http-thread-pool".
Change setting "Max Thread Pool Size" from 5 to 32
Change setting "Min Thread Pool Size" from 2 to 16
Restart Glassfish.
I did also stumble upon this error. In my case the problem was I was using JRE6, with support for TLS1.0. The server only supported TLS1.2, so this error was thrown.
In my case, this was because my Tomcat was set with an insufficient maxHttpHeaderSize for a particularly complicated SOLR query.
Hope this helps someone out there!
This error occurs on the server side when the client closed the socket connection before the response could be returned over the socket. In a web app scenario not all of these are dangerous, since they can be created manually. For example, by quitting the browser before the reponse was retrieved.
I get this error all the time and consider it normal.
It happens when one side tries to read when the other side has already hung up. Thus depending on the protocol this may or may not designate a problem.
If my client code specifically indicates to the server that it is going to hang up, then both client and server can hang up at the same time and this message would not happen.
The way I implement my code is for the client to just hang up without saying goodbye.
The server can then catch the error and ignore it. In the context of HTTP, I believe one level of the protocol allows more then one request per connection while the other doesn't.
Thus you can see how potentially one side could keep hanging up on the other. I doubt the error you are receiving is of any piratical concern and you could simply catch it to keep it from filling up your log files.
This is an old thread, but I ran into java.net.SocketException: Connection reset yesterday.
The server-side application had its throttling settings changed to allow only 1 connection at a time! Thus, sometimes calls went through and sometimes not. I solved the problem by changing the throttling settings.
The Exception means that the socket was closed unexpectedly from the other side. Since you are calling a web service, this should not happen - most likely you're sending a request that triggers a bug in the web service.
Try logging the entire request in those cases, and see if you notice anything unusual. Otherwise, get in contact with the web service provider and send them your logged problematical request.
I know this thread is little old, but would like to add my 2 cents.
We had the same "connection reset" error right after our one of the releases.
The root cause was, our apache server was brought down for deployment. All our third party traffic goes thru apache and we were getting connection reset error because of it being down.
I got this error when the text file I was trying to read contained a string that matched an antivirus signature on our firewall.
FWIW, I was getting this error when I was accidentally making a GET request to an endpoint that was expecting a POST request. Presumably that was just that particular servers way of handling the problem.
I was getting exactly that error too: Connection reset by peer. The exception was being raised by Spring's REST template upon running the postForObject() method. For me the problem was too long HTTP URL request. So first check whether the URL produced is what it should be and, if your server really should be able to handle requests of that length, simply go to server's configuration and raise the default allowed length of URL requests.
That solved the problem for me, but be aware: the application might not run on some internet browsers, especially old ones, as they have fixed max length of URL requests.
Hope it helps...
I was getting this error because the port I tried to connect to was closed.

How to Debug HttpRequestExecutor.execute(request, connection, context);.?

The below lines of code
HttpRequestExecutor httpexecutor = new HttpRequestExecutor();
HttpResponse response2 = httpexecutor.execute(request2, conn, context);
log.info("Status Line "+response2.getStatusLine());
while executing the second statement my request/response is blocked by proxy. How to debug this.?
While the same statement executing from my local environment where no proxy is configured, am able to see
the log returns the below code
"Status Line HTTP/1.1 200 OK"
I think what you are looking for is to set a timeout on the request - basically telling the executor to give up if no connection can be established within a certain period of time.
This SO article covers this: Java HTTP Client Request with defined timeout
the accepted answer has important information about the fact that Apache's implementation has two timeouts - be sure to read that.
Note that it won't be possible for you to actually distinguish between a connection failure that was caused by some other problem - all you will know is that your attempt timed out. In your response to the user, you'll probably wind up saying "Unable to connect to host XXXXX. Possible causes are that your Internet connection is down, or that you have a proxy server that prevents outbound connections to that host."

Glassfish throws com.sun.xml.ws.client.ClientTransportException: The server sent HTTP status code 500: Internal Server Error

I have deployed jax-ws web service into glassfish 3.1.My client request for the service method which returns 5000 to 10000 list of objects.In between processing server throws ClientTransportException with following stack-trace.
com.sun.xml.ws.client.ClientTransportException: The server sent HTTP status code 500: Internal Server Error
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.createResponsePacket(HttpTransportPipe.java:314)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:265)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:184)
at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:109)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:641)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:600)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:585)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:482)
at com.sun.xml.ws.client.Stub.process(Stub.java:323)
at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:161)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:113)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:93)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:144)
at $Proxy190.webservicemethodcall(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
I try to monitor the glassfish request but it show errorcount 1 in request statistics but it don't provide me any proper reason of errorcount.
It has been observed in multiple test,i got the client Transport at client but at server the method thread separately working properly up to last line.It don't aware of broken connection.
I think that the connection is broken so thread can not return the response at last.
Note : If return response is small like up to 3000 objects it works fine.But i don't thing it is matter of size.It is matter of timeout.My request connection is broken before creating responce
Please help me
A HTTP 500 means Internal Server Error, which is no fault of your client. Something about your request is failing on the server. You should look there for more info. Your client side stack trace isn't going to help.
You can try any combination of the following:
While your request is running, run a continuous ping from the client. You should see a break in the pinging (or an increase in TTL at least to confirm the theory)
Set the following JVM property for a dump of the HTTP message exchange between server and client
-Dcom.sun.xml.ws.transport.http.HttpAdapter.dump=true
Try TCPMon
Implement a JAX-WS SOAP Handler to capture the exact moment when the pipe runs dry. This might have the extra benefit of throwing a meaningful exception when the handler attempts to log a message and gets burnt by the absent message
If your logging policy is not clearly defined, your exception might be silently swallowed on server side.
I would try to add a try/catch block to detect where this happens (and eventually remove it later if you improve your logging strategy)
public returnType yourMethod(){
try {
.... all your code
}catch (final Throwable t) {
log.error("Failed to wait for device update: " + t.getMessage());
//eventually re-throw the error
}
}

Difference between ConnectionTimeoutException and SSLException with message "Connection timed out"

On my Android code (using Apache HTTP Client) I have already set the following parameters :
HttpConnectionParams.setConnectionTimeout(params, 30 * SECOND_IN_MILLIS);
HttpConnectionParams.setSoTimeout(params, 30 * SECOND_IN_MILLIS);
I am connecting to an HTTPS web-service. On slow networks, I get a ConnectionTimeoutException after expected 30 seconds (and I retry then) ; but there are "few" cases (with no pattern I could observe), I get either of these :
javax.net.ssl.SSLException: Read error: ssl=0xe71160: I/O error during
system call, Connection timed out javax.net.ssl.SSLException: Write
error: ssl=0xe71160: I/O error during system call, Broken pipe
I got this code from SO, which is still under test :
registry.register(new Scheme("https",
SSLCertificateSocketFactory.getHttpSocketFactory(30 * SECOND_IN_MILLIS, null), 443));
I can understand Connection establishment timeout and socket timeout - my query is do we really need this 3rd timeout for SSL ? How is the purpose different from SO_TIMEOUT ?
PS : These excepions seem to come in picture while reading/writing data - which should have resulted in data loss - but I cannot observe that either.
The difference is that the ConnectionTimeoutException occurs when no connection can be established at all. That means the server is not reachable at all.
The SSLException occurs when the connection was established already but during the SSL handshake the connection times out. Because the timeout occurs after the SSL handshake has started the SSLException is thrown as higher level exception.
You can try to check if a ConnectionTimeoutException is entered as the cause of the SSLException.

How to debug SocketTimeoutException?

We are getting a java.net.SocketTimeoutException on server B when client A connects to server B. No idea why. The client is sending data to the server and the server then throws this exception. How would one troubleshoot this issue?
Note currently this has happened only once. Not sure if this is reproduceable. Attempting to setup the test again..
I had same problems, when my users used 3G or 2G network. It means, that you send request to server, and can't estabilish connection, because of weak internet signal. You can increase timeouts on your connection
URLConnection connection;
int timeout = 30 * 1000;
connection.setConnectTimeout(timeout);
connection.setReadTimeout(timeout);
But if you have weaaak weeeaaaak internet connection, timeouts does not help you.
I'm just created 1 testFunction in WebService (or you can use one of yours) for testing connection with server before calling another required functions, and if I get SockectTimeoutException calling this function - just report to user notification "Weak internet connection!".
No data arrived at the receiver within the timeout period. That's all it means. Debugging it means finding out why the data you think was sent wasn't sent. A missing flush() for example.

Categories

Resources