I have a server that is experiencing some wierd exceptions.
The server is calling a backend service and during "normal" operation this works without fault, however if that service goes down for a short timespan my server completely dies.
After the backend service has restarted I am getting unknown host exceptions for all my http calls.
If I restart my server it works again, but it seems odd that I even get the error in the first place.
An example of a call I make:
Client client = ClientBuilder.newClient();
response = client.target(uri)
.request(MediaType.APPLICATION_JSON_TYPE)
.header("XApiKey", catalogApiKey)
.get();
and the stacktrace is rather standard:
java.net.UnknownHostException.
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:259)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:159)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:436)
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.get(ClientInvocationBuilder.java:159)
if it is relevant I am running on a wildfly server.
What could cause this kind of behaviour?
If you look at the source of InetAddress (e.g. http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/net/InetAddress.java) everywhere that UnknownHostException is instantiated it is instantiated with either a host string or a clearly non-empty string as the constructor parameter.
However, the exception in the stacktrace appears to have an empty message. So I infer that the code is doing a lookup on a hostname that is an empty string.
I would add some code to print the value of uri when this problem occurs. I suspect that is (sporadically) bogus in some interesting way.
Java caches DNS entries. The default is "forever" as defined in $JAVA_HOME/jre/lib/security/java.security under the entry networkaddress.cache.ttl.
I would encourage you to consider reducing the "positive" cache to a small amount, perhaps 10 seconds or so. You can modify the java.security file or have code like:
java.security.Security.setProperty("networkaddress.cache.ttl", "10");
In your code. There is also a "negative" cache that caches entries that have failed. It may take some experimentation in your environment with the positive and negative caches to find the right setting.
Related
I am working with a commercial application which is throwing a SocketException with the message,
An existing connection was forcibly closed by the remote host
This happens with a socket connection between client and server. The connection is alive and well, and heaps of data is being transferred, but it then becomes disconnected out of nowhere.
Has anybody seen this before? What could the causes be? I can kind of guess a few causes, but also is there any way to add more into this code to work out what the cause could be?
Any comments / ideas are welcome.
... The latest ...
I have some logging from some .NET tracing,
System.Net.Sockets Verbose: 0 : [8188] Socket#30180123::Send() DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Error: 0 : [8188] Exception in the Socket#30180123::Send - An existing connection was forcibly closed by the remote host DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Verbose: 0 : [8188] Exiting Socket#30180123::Send() -> 0#0
Based on other parts of the logging I have seen the fact that it says 0#0 means a packet of 0 bytes length is being sent. But what does that really mean?
One of two possibilities is occurring, and I am not sure which,
The connection is being closed, but data is then being written to the socket, thus creating the exception above. The 0#0 simply means that nothing was sent because the socket was already closed.
The connection is still open, and a packet of zero bytes is being sent (i.e. the code has a bug) and the 0#0 means that a packet of zero bytes is trying to be sent.
What do you reckon? It might be inconclusive I guess, but perhaps someone else has seen this kind of thing?
This generally means that the remote side closed the connection (usually by sending a TCP/IP RST packet). If you're working with a third-party application, the likely causes are:
You are sending malformed data to the application (which could include sending an HTTPS request to an HTTP server)
The network link between the client and server is going down for some reason
You have triggered a bug in the third-party application that caused it to crash
The third-party application has exhausted system resources
It's likely that the first case is what's happening.
You can fire up Wireshark to see exactly what is happening on the wire to narrow down the problem.
Without more specific information, it's unlikely that anyone here can really help you much.
Using TLS 1.2 solved this error.
You can force your application using TLS 1.2 with this (make sure to execute it before calling your service):
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Another solution :
Enable strong cryptography in your local machine or server in order to use TLS1.2 because by default it is disabled so only TLS1.0 is used.
To enable strong cryptography , execute these commande in PowerShell with admin privileges :
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
You need to reboot your computer for these changes to take effect.
This is not a bug in your code. It is coming from .Net's Socket implementation. If you use the overloaded implementation of EndReceive as below you will not get this exception.
SocketError errorCode;
int nBytesRec = socket.EndReceive(ar, out errorCode);
if (errorCode != SocketError.Success)
{
nBytesRec = 0;
}
Had the same bug. Actually worked in case the traffic was sent using some proxy (fiddler in my case). Updated .NET framework from 4.5.2 to >=4.6 and now everything works fine. The actual request was:
new WebClient().DownloadData("URL");
The exception was:
SocketException: An existing connection was forcibly closed by the
remote host
Simple solution for this common annoying issue:
Just go to your ".context.cs" file (located under ".context.tt" which located under your "*.edmx" file).
Then, add this line to your constructor:
public DBEntities()
: base("name=DBEntities")
{
this.Configuration.ProxyCreationEnabled = false; // ADD THIS LINE!
}
I've got this exception because of circular reference in entity.In entity that look like
public class Catalog
{
public int Id { get; set; }
public int ParentId { get; set; }
public Catalog Parent { get; set; }
public ICollection<Catalog> ChildCatalogs { get; set; }
}
I added [IgnoreDataMemberAttribute] to the Parent property. And that solved the problem.
If Running In A .Net 4.5.2 Service
For me the issue was compounded because the call was running in a .Net 4.5.2 service. I followed #willmaz suggestion but got a new error.
In running the service with logging turned on, I viewed the handshaking with the target site would initiate ok (and send the bearer token) but on the following step to process the Post call, it would seem to drop the auth token and the site would reply with Unauthorized.
Solution
It turned out that the service pool credentials did not have rights to change TLS (?) and when I put in my local admin account into the pool, it all worked.
I had the same issue and managed to resolve it eventually. In my case, the port that the client sends the request to did not have a SSL cert binding to it. So I fixed the issue by binding a SSL cert to the port on the server side. Once that was done, this exception went away.
For anyone getting this exception while reading data from the stream, this may help. I was getting this exception when reading the HttpResponseMessage in a loop like this:
using (var remoteStream = await response.Content.ReadAsStreamAsync())
using (var content = File.Create(DownloadPath))
{
var buffer = new byte[1024];
int read;
while ((read = await remoteStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
await content.WriteAsync(buffer, 0, read);
await content.FlushAsync();
}
}
After some time I found out the culprit was the buffer size, which was too small and didn't play well with my weak Azure instance. What helped was to change the code to:
using (Stream remoteStream = await response.Content.ReadAsStreamAsync())
using (FileStream content = File.Create(DownloadPath))
{
await remoteStream.CopyToAsync(content);
}
CopyTo() method has a default buffer size of 81920. The bigger buffer sped up the process and the errors stopped immediately, most likely because the overall download speeds increased. But why would download speed matter in preventing this error?
It is possible that you get disconnected from the server because the download speeds drop below minimum threshold the server is configured to allow. For example, in case the application you are downloading the file from is hosted on IIS, it can be a problem with http.sys configuration:
"Http.sys is the http protocol stack that IIS uses to perform http communication with clients. It has a timer called MinBytesPerSecond that is responsible for killing a connection if its transfer rate drops below some kb/sec threshold. By default, that threshold is set to 240 kb/sec."
The issue is described in this old blogpost from TFS development team and concerns IIS specifically, but may point you in a right direction. It also mentions an old bug related to this http.sys attribute: link
In case you are using Azure app services and increasing the buffer size does not eliminate the problem, try to scale up your machine as well. You will be allocated more resources including connection bandwidth.
I got the same issue while using .NET Framework 4.5. However, when I update the .NET version to 4.7.2 connection issue was resolved. Maybe this is due to SecurityProtocol support issue.
For me, it was because the app server I was trying to send email from was not added to our company's SMTP server's allowed list.
I just had to put in SMTP access request for that app server.
This is how it was added by the infrastructure team (I don't know how to do these steps myself but this is what they said they did):
1. Log into active L.B.
2. Select: Local Traffic > iRules > Data Group List
3. Select the appropriate Data Group
4. Enter the app server's IP address
5. Select: Add
6. Select: Update
7. Sync config changes
Yet another possibility for this error to occur is if you tried to connect to a third-party server with invalid credentials too many times and a system like Fail2ban is blocking your IP address.
I was trying to connect to the MQTT broker using the GO client,
broker address was given as address + port, or tcp://address:port
Example: ❌
mqtt://test.mosquitto.org
which indicates that you wish to establish an unencrypted connection.
To request MQTT over TLS use one of ssl, tls, mqtts, mqtt+ssl or tcps.
Example: ✅
mqtts://test.mosquitto.org
In my case, enable the IIS server & then restart and check again.
We are using a SpringBoot service. Our restTemplate code looks like below:
#Bean
public RestTemplate restTemplate(final RestTemplateBuilder builder) {
return builder.requestFactory(() -> {
final ConnectionPool okHttpConnectionPool =
new ConnectionPool(50, 30, TimeUnit.SECONDS);
final OkHttpClient okHttpClient =
new OkHttpClient.Builder().connectionPool(okHttpConnectionPool)
// .connectTimeout(30, TimeUnit.SECONDS)
.retryOnConnectionFailure(false).build();
return new OkHttp3ClientHttpRequestFactory(okHttpClient);
}).build();
}
All our call were failing after the ReadTimeout set for the restTemplate. We increased the time, and our issue was resolved.
This error occurred in my application with the CIP-protocol whenever I didn't Send or received data in less than 10s.
This was caused by the use of the forward open method. You can avoid this by working with an other method, or to install an update rate of less the 10s that maintain your forward-open-connection.
I have a problem while trying my hands on the Hello World example explained here.
Kindly note that I have just modified the HelloEntity.java file to be able to return something other than "Hello, World!". Most certain my changes are taking time and hence I am getting the below Timeout error.
I am currently trying (doing a PoC) on a single node to understand the Lagom framework and do not have liberty to deploy multiple nodes.
I have also tried modifying the default lagom.circuit-breaker in application.conf "call-timeout = 100s" however, this does not seem to have helped.
Following is the exact error message for your reference:
{"name":"akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://hello-impl-application/system/sharding/HelloEntity#1074448247]] after [5000 ms]. Sender[null] sent message of type \"com.lightbend.lagom.javadsl.persistence.CommandEnvelope\".","detail":"akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://hello-impl-application/system/sharding/HelloEntity#1074448247]] after [5000 ms]. Sender[null] sent message of type \"com.lightbend.lagom.javadsl.persistence.CommandEnvelope\".\n\tat akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:595)\n\tat akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:605)\n\tat akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)\n\tat scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)\n\tat scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)\n\tat scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)\n\tat scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)\n\tat akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)\n\tat akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)\n\tat akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)\n\tat akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)\n\tat java.lang.Thread.run(Thread.java:748)\n"}
Question: Is there a way to increase the akka Timeout by modifying the application.conf or any of the java source files in the Hello World project? Can you please help me with the exact details.
Thanks in advance for you time and help.
The call timeout is the timeout for circuit breakers, which is configured using lagom.circuit-breaker.default.call-timeout. But that's not what is timing out above, the thing that is timing out above is the request to your HelloEntity, that timeout is configured using lagom.persistence.ask-timeout. The reason why there's a timeout on requests to entities is because in a multi-node environment, your entities are sharded across nodes, so an ask on them may go to another node, which is why a timeout is needed in case that node is not responding.
All that said, I don't think changing the ask-timeout will solve your problem. If you have a single node, then your entities should respond instantly if everything is working ok.
Is that the only error you're seeing in the logs?
Are you seeing this in devmode (ie, using the runAll command), or are you running the Lagom service some other way?
Is your database responding?
Thanks James for the help/pointer.
Adding following lines to resources/application.conf did the trick for me:
lagom.persistence.ask-timeout=30s
hello {
..
..
call-timeout = 30s
call-timeout = ${?CIRCUIT_BREAKER_CALL_TIMEOUT}
..
}
A Call is a Service-to-Service communication. That’s a SeviceClient communicating to a remote server. It uses a circuit breaker. It is a extra-service call.
An ask (in the context of lagom.persistence) is sending a command to a persistent entity. That happens across the nodes insied your Lagom service. It is not using circuit breaking. It is an intra-service call.
This question already has answers here:
java.net.SocketException: Connection reset
(14 answers)
Closed 3 years ago.
We are seeing frequent but intermittent java.net.SocketException: Connection reset errors in our logs. We are unsure as to where the Connection reset error is actually coming from, and how to go about debugging.
The issue appears to be unrelated to the messages we are attempting to send.
Note that the message is not connection reset by peer.
Any suggestions on what the typical causes of this exception might be, and how we might proceed?
Here is a representative stack trace (com.companyname.mtix.sms is our component):
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at com.companyname.mtix.sms.services.impl.message.SendTextMessage.sendTextMessage(SendTextMessage.java:127)
at com.companyname.mtix.sms.services.MessageServiceImpl.sendTextMessage(MessageServiceImpl.java:125)
at com.companyname.mtix.sms.services.remote.MessageServiceRemoteImpl.sendTextMessage(MessageServiceRemoteImpl.java:43)
at sun.reflect.GeneratedMethodAccessor203.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:453)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.companyname.mtix.sms.http.filters.NoCacheFilter.doFilter(NoCacheFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.companyname.mtix.sms.http.filters.MessageFilter.doFilter(MessageFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:61)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.ajaxanywhere.AAFilter.doFilter(AAFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Our component is a web application, running under Tomcat, that calls a third party Web service that sends SMS messages, it so happens. The line of our code on which the exception gets thrown from is the last line in the code snippet below.
String aggregatorResponse = null;
HttpClient httpClient = prepareHttpClient( username, password );
PostMethod postMethod = preparePostMethod( textUrl );
try {
SybaseTextMessageBuilder builder = new SybaseTextMessageBuilder();
URL notifyUrl = buildNotificationUrl( textMessage, codeSetManager );
String smsRequestDocument = builder.buildTextMessage( textMessage, notifyUrl );
LOG.debug( "Sybase MT document created as: \n" + smsRequestDocument );
postMethod.setRequestEntity( new StringRequestEntity( smsRequestDocument ) );
LOG.debug( "commiting SMS to aggregator: " + textMessage.toString() );
int httpStatus = httpClient.executeMethod( postMethod );
The javadoc for SocketException states that it is
Thrown to indicate that there is an error in the underlying protocol such as a TCP error
In your case it seems that the connection has been closed by the server end of the connection. This could be an issue with the request you are sending or an issue at their end.
To aid debugging you could look at using a tool such as Wireshark to view the actual network packets. Also, is there an alternative client to your Java code that you could use to test the web service? If this was successful it could indicate a bug in the Java code.
As you are using Commons HTTP Client have a look at the Common HTTP Client Logging Guide. This will tell you how to log the request at the HTTP level.
This error happens on your side and NOT the other side. If the other side reset the connection, then the exception message should say:
java.net.SocketException reset by peer
The cause is the connection inside HttpClient is stale. Check stale connection for SSL does not fix this error. Solution: dump your client and recreate.
If you experience this trying to access Web services deployed on a Glassfish3 server, you might want to tune your http-thread-pool settings. That fixed SocketExceptions we had when many concurrent threads was calling the web service.
Go to admin console
Navigate to "Configurations"->"Server config"->"Thread pools"->"http-thread-pool".
Change setting "Max Thread Pool Size" from 5 to 32
Change setting "Min Thread Pool Size" from 2 to 16
Restart Glassfish.
I did also stumble upon this error. In my case the problem was I was using JRE6, with support for TLS1.0. The server only supported TLS1.2, so this error was thrown.
In my case, this was because my Tomcat was set with an insufficient maxHttpHeaderSize for a particularly complicated SOLR query.
Hope this helps someone out there!
This error occurs on the server side when the client closed the socket connection before the response could be returned over the socket. In a web app scenario not all of these are dangerous, since they can be created manually. For example, by quitting the browser before the reponse was retrieved.
I get this error all the time and consider it normal.
It happens when one side tries to read when the other side has already hung up. Thus depending on the protocol this may or may not designate a problem.
If my client code specifically indicates to the server that it is going to hang up, then both client and server can hang up at the same time and this message would not happen.
The way I implement my code is for the client to just hang up without saying goodbye.
The server can then catch the error and ignore it. In the context of HTTP, I believe one level of the protocol allows more then one request per connection while the other doesn't.
Thus you can see how potentially one side could keep hanging up on the other. I doubt the error you are receiving is of any piratical concern and you could simply catch it to keep it from filling up your log files.
This is an old thread, but I ran into java.net.SocketException: Connection reset yesterday.
The server-side application had its throttling settings changed to allow only 1 connection at a time! Thus, sometimes calls went through and sometimes not. I solved the problem by changing the throttling settings.
The Exception means that the socket was closed unexpectedly from the other side. Since you are calling a web service, this should not happen - most likely you're sending a request that triggers a bug in the web service.
Try logging the entire request in those cases, and see if you notice anything unusual. Otherwise, get in contact with the web service provider and send them your logged problematical request.
I know this thread is little old, but would like to add my 2 cents.
We had the same "connection reset" error right after our one of the releases.
The root cause was, our apache server was brought down for deployment. All our third party traffic goes thru apache and we were getting connection reset error because of it being down.
I got this error when the text file I was trying to read contained a string that matched an antivirus signature on our firewall.
FWIW, I was getting this error when I was accidentally making a GET request to an endpoint that was expecting a POST request. Presumably that was just that particular servers way of handling the problem.
I was getting exactly that error too: Connection reset by peer. The exception was being raised by Spring's REST template upon running the postForObject() method. For me the problem was too long HTTP URL request. So first check whether the URL produced is what it should be and, if your server really should be able to handle requests of that length, simply go to server's configuration and raise the default allowed length of URL requests.
That solved the problem for me, but be aware: the application might not run on some internet browsers, especially old ones, as they have fixed max length of URL requests.
Hope it helps...
I was getting this error because the port I tried to connect to was closed.
I'm developing a project using Grizzly 2.3.22 with its Websocket support. Everything was OK until OOM happened. Looking through the dump I found that all the memory was eaten up by a single org.glassfish.grizzly.nio.transport.TCPNIOConnection holding a huge (1,5GB) write queue. I guess one of the client developers was debugging their connected application and stopped on a breakpoint for a long time. Anyway, this can easily happen if a client has a very slow connection - my server should be ready for that.
In the Grizzly documentation I found the maxPendingBytes property, which seem like a solution, at least for now. But I cannot get it to work at all. I set log level to ALL for AbstractNIOAsyncQueueWriter, connect with the client, put it on hold and observe how the server's queue grows like this:
TRACE 2016-07-05 21:02:26.330 [nioEventLoopGroup-2-1] o.g.g.n.AbstractNIOAsyncQueueWriter - AsyncQueueWriter.write connection=TCPNIOConnection{localSocketAddress={/127.0.0.1:8445}, peerSocketAddress={/127.0.0.1:56185}}, record=org.glassfish.grizzly.asyncqueue.AsyncWriteQueueRecord#1e35bafb, directWrite=false, size=165, isUncountable=false, bytesToReserve=165, pendingBytes=16170
TRACE 2016-07-05 21:02:26.368 [nioEventLoopGroup-2-1] o.g.g.n.AbstractNIOAsyncQueueWriter - AsyncQueueWriter.write connection=TCPNIOConnection{localSocketAddress={/127.0.0.1:8445}, peerSocketAddress={/127.0.0.1:56185}}, record=org.glassfish.grizzly.asyncqueue.AsyncWriteQueueRecord#3d6e05dd, directWrite=false, size=165, isUncountable=false, bytesToReserve=165, pendingBytes=16335
...
When I set maxPendingBytes=10000 I expect an exception thrown when the pendingBytes from the log above becomes larger than 10000, but it doesn't happen.
Moreover, I tried debugging the server with the Grizzly's source code, and found that while the property's value does get assigned to the NIOConnection.maxAsyncWriteQueueSize field, the AbstractNIOAsyncQueueWriter.canWrite(...) method - the only place where the field seems to be used - is never called.
I'm at a loss. Am I missing something here?
I am working with a webapp that runs in a Tomcat 6 server.
With some request (that came from specific types of clients) it happens that the method getParameter of ServletRequestWrapper handles internally all CharConversionException logging to what i thing is the standard output of the server activity information about that exception. The thing is that sometimes it can be logging sensitive data (as password)... for example, it can log things like this :
INFO: Character decoding failed. Parameter [pw] with value [holaãã%20%222522%2] has been ignored. Note that the name and value quoted here may be corrupted due to the failed decoding. Use debug level logging to see the original, non-corrupted values.
java.io.CharConversionException: EOF
at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:80)
at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:46)
at org.apache.tomcat.util.http.Parameters.urlDecode(Parameters.java:410)
at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:370)
at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:217)
at org.apache.catalina.connector.Request.parseParameters(Request.java:2647)
at org.apache.catalina.connector.Request.getParameter(Request.java:1106)
at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:355)
at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:158)
at myClasss (myClass.java:666)
I am not looking to resolve the problem in server, as i see is a problem from the client and the client must solve. I am looking forward to "hide" the value associated with the bad parameter that is outputted in the log.
I am not an expert of tomcat logging system and how to configure it, i visited and read some material (this and this too..) but couldn't find a clue that pointed me into the right direction (if there is any..).
I've already took at look this ServletRequestWrapper or ServletResponseWrapper in production?, but there is no clue about how to modify this internal message.
Well thanks for everything!.
Greetings
Victor
First two remarks:
The wrong encoding is not strictly a client problem; there are just different settings. So allow me to point to some server settings. Furthermore searching for "servlet filter character encoding" will yield some ServletFilters that set the request encoding right for getRequestParameter. (GET functions differently than POST!)
"%2" at the end is a bit suspicious, isn't it.
The output looks like log output, and indeed in Parameters.java I found org.apache.juli.logging.Log.This yet another logging library of Tomcat, seems based on java.util.logging, and you may set the level to FATAL/ERROR in the WEB-INF/classes/logging.properties for org.apache.tomcat.util.http.Parameters=SEVERE.