I am seeing a lot of Connection Resets in Production.There could be multiple causes to it but I wanted to ensure that there are no Connection leakages coming from in code.I am using Jersey Client in code
Client this.client = ApacheHttpClient.create();
client.resource("/stores/"+storeId).type(MediaType.APPLICATION_JSON_TYPE).put(ClientResponse.class,indexableStore);
Originally I was instantiating client in the following fashion
Client this.client = Client.create() and we changed it to ApacheHttpClient.create(). I am not calling close() on the response but I am assuming ApacheHttpClient would do that internally as HttpClient executeMethod gets invoked which handles all the boiler plate stuff for us. Could there be a potential connection leakage in the way the code is written ?
Like you said Connection Reset could be caused by many possible reasons. One such possibility could be that server timed out while processing the request, thats why the client receives connection reset. The comments section of the answered question here discusses possible causes of connection reset in detail. One possible solution I can think of is to configure HttpClient to retry the request in case of a failure. You could set the HttpMethodRetryHandler like below to do so (Reference). You may perhaps need to modify the code based on the exception you receive.
HttpMethodRetryHandler retryHandler = new HttpMethodRetryHandler()
{
public boolean retryMethod(
final HttpMethod method,
final IOException exception,
int executionCount)
{
if (executionCount >= 5)
{
// Do not retry if over max retry count
return false;
}
if (exception instanceof NoHttpResponseException)
{
// Retry if the server dropped connection on us
return true;
}
if (!method.isRequestSent())
{
// Retry if the request has not been sent fully or
// if it's OK to retry methods that have been sent
return true;
}
// otherwise do not retry
return false;
}
};
ApacheHttpClient client = ApacheHttpClient.create();
HttpClient hc = client.getClientHandler().getHttpClient();
hc.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, retryHandler);
client.resource("/stores/"+storeId).type(MediaType.APPLICATION_JSON_TYPE).put(ClientResponse.class,indexableStore);
Related
It's not clear must I close JAX RS Client/Response instances or not. And if I must, always or not?
According to documentation about the Client class:
Calling this method effectively invalidates all resource targets
produced by the client instance.
The WebTarget class does not have any invalidate()/close() method, but the Response class does.
According to documentation:
Close the underlying message entity input stream (if available and
open) as well as releases any other resources associated with the
response (e.g. buffered message entity data).
... The close() method
should be invoked on all instances that contain an un-consumed entity
input stream to ensure the resources associated with the instance are
properly cleaned-up and prevent potential memory leaks. This is
typical for client-side scenarios where application layer code
processes only the response headers and ignores the response entity.
The last paragraph is not clear to me. What does "un-consumed entity input stream" mean? If I get an InputSteam or a String from response, should I close the response explicitly?
We can get a response result without getting access to Response instance:
Client client = ...;
WebTarget webTarget = ...;
Invocation.Builder builder = webTarget.request(MediaType.APPLICATION_JSON_TYPE);
Invocation invocation = builder.buildGet();
InputStream reso = invocation.invoke(InputStream.class);
I'm working with RESTeasy implementation, and I expected that response will be closed inside of resteasy implementation, but I could not find it. Could anyone tell me why?
I know that the Response class will implement Closeable interface
But even know, the Response is used, without closing it.
According to the documentation close() is idempotent.
This operation is idempotent, i.e. it can be invoked multiple times with the same effect which also means that calling the close() method on an already closed message instance is legal and has no further effect.
So you can safely close the InputStream yourself and should.
That being said I style wise would not do invocation.invoke(InputStream.class) as the invoker(Class) is made for doing entity transformation. Instead if you want InputStream you should probably just call invocation.invoke() and deal with the Response object directly as you may want some header info before reading the stream.
The reason you want headers when dealing with a response InputStream is typical because you either don't care about the body or the body requires special processing and size considerations which is what the documentation is alluding to (e.g. HEAD request to ping server).
See also link
A message instance returned from this method will be cached for subsequent retrievals via getEntity(). Unless the supplied entity type is an input stream, this method automatically closes the an unconsumed original response entity data stream if open. In case the entity data has been buffered, the buffer will be reset prior consuming the buffered data to enable subsequent invocations of readEntity(...) methods on this response.
So if you choose anything other than InputStream you will not have to close the Response (but regardless its safe to do it anyways as its idempotent).
In short: do call close() or use closeable with try-with-resources-statements.
If you use the JAX-RS Client reference, calling close() on the client closes open sockets.
Calling close on Response releases the connection but not any open socket
It is not necessary required to call close() since Resteasy will release the connection under the covers. But it should be done if result is an InputStream or if you're dealing with Response results.
Resources/Reference:
According to the Resteasy documentation you should call close() on Response references.
In section 47.3 at the end it states that
Resteasy will release the connection under the covers. The only
counterexample is the case in which the response is an instance of
InputStream, which must be closed explicitly.
On the other hand, if the result of an invocation is an instance of
Response, then Response.close() method must be used to released the
connection.
You should probably execute this in a try/finally block. Again,
releasing a connection only makes it available for another use. It
does not normally close the socket.
Note that if ApacheHttpClient4Engine has created its own instance of
HttpClient, it is not necessary to wait for finalize() to close open
sockets. The ClientHttpEngine interface has a close() method for this
purpose.
Finally, if your javax.ws.rs.client.Client class has created the
engine automatically for you, you should call Client.close() and
this will clean up any socket connections.
Looking into the resteasy-client source code, Invocation#invoke(Class<T>) is simply calling Invocation#invoke() and calling Invocation#extractResult(GenericType<T> responseType, Response response, Annotation[] annotations) to extract the result from the Response:
#Override
public <T> T invoke(Class<T> responseType)
{
Response response = invoke();
if (Response.class.equals(responseType)) return (T)response;
return extractResult(new GenericType<T>(responseType), response, null);
}
Invocation#extractResult(GenericType<T> responseType, Response response, Annotation[] annotations) closes the Response in the finally block:
/**
* Extracts result from response throwing an appropriate exception if not a successful response.
*
* #param responseType
* #param response
* #param annotations
* #param <T>
* #return
*/
public static <T> T extractResult(GenericType<T> responseType, Response response, Annotation[] annotations)
{
int status = response.getStatus();
if (status >= 200 && status < 300)
{
try
{
if (response.getMediaType() == null)
{
return null;
}
else
{
T rtn = response.readEntity(responseType, annotations);
if (InputStream.class.isInstance(rtn)
|| Reader.class.isInstance(rtn))
{
if (response instanceof ClientResponse)
{
ClientResponse clientResponse = (ClientResponse)response;
clientResponse.noReleaseConnection();
}
}
return rtn;
}
}
catch (WebApplicationException wae)
{
try
{
response.close();
}
catch (Exception e)
{
}
throw wae;
}
catch (Throwable throwable)
{
try
{
response.close();
}
catch (Exception e)
{
}
throw new ResponseProcessingException(response, throwable);
}
finally
{
if (response.getMediaType() == null) response.close();
}
}
try
{
// Buffer the entity for any exception thrown as the response may have any entity the user wants
// We don't want to leave the connection open though.
String s = String.class.cast(response.getHeaders().getFirst("resteasy.buffer.exception.entity"));
if (s == null || Boolean.parseBoolean(s))
{
response.bufferEntity();
}
else
{
// close connection
if (response instanceof ClientResponse)
{
try
{
ClientResponse.class.cast(response).releaseConnection();
}
catch (IOException e)
{
// Ignore
}
}
}
if (status >= 300 && status < 400) throw new RedirectionException(response);
return handleErrorStatus(response);
}
finally
{
// close if no content
if (response.getMediaType() == null) response.close();
}
}
Priority & Dependency:
Here I made I simple test. But the result seems not so good.
I tried to make 100 request in a for loop in the same connection(the request url is the same, I am wondering whether this part influence the results).
If the index is i, then my request stream_id is i while the dependent stream_id is 100+i. If our assumption is right, the request can never get response because there is no stream_id from 101 to 200.
But the results shows there is no difference for setting the dependency and not. I got the response data frame one by one without timeout or waiting.
And also some other related test, the start point is to let the stream which depends on other stream to be sent first and the stream dependent later. But the result is same.
I am still thinking the reason of the results. Can anyone help me? Many thanks.
Code here:
public void run() throws Exception
{
host = "google.com";
port = 443;
//client init
HTTP2Client client = new HTTP2Client();
SslContextFactory sslContextFactory = new SslContextFactory(true);
client.addBean(sslContextFactory);
client.start();
//connect init
FuturePromise<Session> sessionPromise = new FuturePromise<>();
client.connect(sslContextFactory, new InetSocketAddress(host, port), new ServerSessionListener.Adapter(), sessionPromise);
Session session = sessionPromise.get(10, TimeUnit.SECONDS);
//headers init
HttpFields requestFields = new HttpFields();
requestFields.put("User-Agent", client.getClass().getName() + "/" + Jetty.VERSION);
final Phaser phaser = new Phaser(2);
//multiple request in one connection
for(int i=0;i<100;i++)
{
MetaData.Request metaData = new MetaData.Request("GET", new HttpURI("https://" + host + ":" + port + "/"), HttpVersion.HTTP_2, requestFields);
PriorityFrame testPriorityFrame = new PriorityFrame(i, 100+i, 4, true);
HeadersFrame headersFrame = new HeadersFrame(0, metaData, testPriorityFrame, true);
//listen header/data/push frame
session.newStream(headersFrame, new Promise.Adapter<Stream>(), new Stream.Listener.Adapter()
{
#Override
public void onHeaders(Stream stream, HeadersFrame frame)
{
System.err.println(frame+"headId:"+frame.getStreamId());
if (frame.isEndStream())
phaser.arrive();
}
#Override
public void onData(Stream stream, DataFrame frame, Callback callback)
{
System.err.println(frame +"streamid:"+ frame.getStreamId());
callback.succeeded();
if (frame.isEndStream())
phaser.arrive();
}
#Override
public Stream.Listener onPush(Stream stream, PushPromiseFrame frame)
{
System.err.println(frame+"pushid:"+frame.getStreamId());
phaser.register();
return this;
}
});
}
phaser.awaitAdvanceInterruptibly(phaser.arrive(), 5, TimeUnit.SECONDS);
client.stop();
}
The Jetty project did not implement (yet) HTTP/2 request prioritization.
We are discussing whether this is any useful for a server, whose concern is to write back the responses as quick as it can.
Having one client changing its mind on the priority of the requests, or making a request knowing that in reality it first wanted another request served, it's a lot of work for the server that in the meantime has to serve the other 10,000 clients connected to it.
By the time we the server has recomputed the priority tree for the dependent requests, it could have probably have served the requests already.
By the time the client realizes that it has to change the priority of a request, the whole response for it could already be in flight.
Having said that, we are certainly interested in real world use cases where request prioritization performed by the server yields a real performance improvement. We just have not seen it yet.
I would love to hear why you are interested in request prioritization and how you are leveraging it. Your answer could be a drive for the Jetty project to implement HTTP/2 priorities.
I am seeing a lot of Connection Resets in Production.There could be multiple causes to it but I wanted to ensure that there are no Connection leakages coming from in code.I am using Jersey Client in code
Client this.client = ApacheHttpClient.create();
client.resource("/stores/"+storeId).type(MediaType.APPLICATION_JSON_TYPE).put(ClientResponse.class,indexableStore);
Originally I was instantiating client in the following fashion
Client this.client = Client.create() and we changed it to ApacheHttpClient.create(). I am not calling close() on the response but I am assuming ApacheHttpClient would do that internally as HttpClient executeMethod gets invoked which handles all the boiler plate stuff for us. Could there be a potential connection leakage in the way the code is written ?
Like you said Connection Reset could be caused by many possible reasons. One such possibility could be that server timed out while processing the request, thats why the client receives connection reset. The comments section of the answered question here discusses possible causes of connection reset in detail. One possible solution I can think of is to configure HttpClient to retry the request in case of a failure. You could set the HttpMethodRetryHandler like below to do so (Reference). You may perhaps need to modify the code based on the exception you receive.
HttpMethodRetryHandler retryHandler = new HttpMethodRetryHandler()
{
public boolean retryMethod(
final HttpMethod method,
final IOException exception,
int executionCount)
{
if (executionCount >= 5)
{
// Do not retry if over max retry count
return false;
}
if (exception instanceof NoHttpResponseException)
{
// Retry if the server dropped connection on us
return true;
}
if (!method.isRequestSent())
{
// Retry if the request has not been sent fully or
// if it's OK to retry methods that have been sent
return true;
}
// otherwise do not retry
return false;
}
};
ApacheHttpClient client = ApacheHttpClient.create();
HttpClient hc = client.getClientHandler().getHttpClient();
hc.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, retryHandler);
client.resource("/stores/"+storeId).type(MediaType.APPLICATION_JSON_TYPE).put(ClientResponse.class,indexableStore);
So I have a netty-based websockets client that I am using for performance tests. My idea is that I can use it to simulate 100, 1000, etc simultaneous connections.
I've determined that my current approach to this is not working--the test harness is simply not creating enough websocket connections, althogh it bumps along happily, thinks it's still connected, etc. But my server simply does not show the correct number of connections when I use this test harness. I think most likely this is occurring because I am using various objects in the netty library across multiple threads at once and they don't handle that very well. ClientBootstrap, for example.
This is what I am doing per-thread. Can you tell me where I am going wrong, so that I can fix my test harness?
public void run(){
try{
// client bootstrap. There is one of these per thread. is that part of the problem?
ClientBootstrap bootstrap = new ClientBootstrap(new NIOClientSocketChannelFactory(Executors.newCachedThreadPool(), Executors.newCachedThreadPool())));
Channel ch = null;
try{
// set up ssl engine
final SSLEngine engine = createServerContext().createSSLEngine();
engine.setUseClientMode(true);
// there is a new handhsaker per thread, too. They all go to the same uri
final WebSocketClientHandshaker handshaker = new WebSocketClientHandhsakerFactory().newHandshaker(uri, WebSocketVersion.V08, null, false, null);
// set up the pipeline factory and pipeline
bootstrap.setPipelineFactory(new ChannelPipelieFactory(){
#Override
public Channelpipeline getPipeline() throws Exception(){
ChannelPipeline pipeline = Channels.pipeline();
pipeline.addLast("ssl", new SslHandler(engine));
pipeline.addLast("encoder", new HttpRequestEncoder();
pipeline.addLast("decoder", new HttpResponseDecoder();
// WebSocketClientHandler code not included, it's just a custom handler that sends requests via websockets
pipeline.addLast("ws-handler", new WebSocketClientHandler(handshaker);
return pipleline;
}
});
// connect websockets preflight over http
ChannelFuture future = bootstrap.connect(new InetSocketAddress(uri.getHost(), uri.getPort());
future.sync();
// do websockets handshake
ch = future.getChannel();
ChannelFuture handshakeFuture = handshaker.handshake(ch);
handshakeFuture.syncUninterruptably();
Thread.sleep(1000); // i had to add this. Sync should have meant that the above method didn't return until it was complete... but that was a lie. So I sleep for 1 second to solve that problem.
if(!handshakeDuture.isSuccess())
System.out.println("WHOAH errror");
// send message to server
ch.write(new TextWebSocketFrame("Foo"));
// wait for notifications to close
while(!getShutdownNow().get()) // shutdownNow is an atomicBoolean which is set to true when all my threads have been started up and a certain amount of time has passed
Thread.sleep(2000);
// send close; wait for response
ch.write(new CloseWebSocketFrame());
ch.getCloseFuture().awaitUninterruptibly();
}
}
}
}
I'm having problems with Htmlunit, I disabled JavaScript and set timeout to 10000 before calling getpage, I expected an exception after timeout but htmlunit waits forever.
After some search I realized someone in 2009 had the same problem (Connection timeout not working), he was complaining about "Connection timeout not working" and about some values in timeout not working but until now in 2011 didn't get any answer.
Someone here was asking about what exception is thrown but I think it doesn't throw it always. I can't get an answer from Apache HttpClient setTimeout, either. You can see another person asking about stop in timeout in Terminate or Stop HtmlUnit.
You can see how crazy it is if you try:
milisecReqTimeout = 10;
while(true)
{
_webclient.setTimeout(milisecReqTimeout);
milisecReqTimeout = milisecReqTimeout + 10;
_htmlpage = _webclient.getPage(url);
}
_thewebclient.setWebConnection(new HttpWebConnection(_thewebclient) {
#Override
protected synchronized AbstractHttpClient getHttpClient() {
AbstractHttpClient client = super.getHttpClient();
if (_TimeoutCliSocket > 0) {
//Sets the socket timeout (SO_TIMEOUT) in milliseconds to
//be used when executing the method.
//A timeout value of zero is interpreted as an infinite timeout.
//Time that a read operation will block for, before generating
//an java.io.InterruptedIOException
client.getParams().setParameter("http.socket.timeout",
_TimeoutCliSocket);
}
if (_TimeoutCliConnection > 0) {
//The timeout in milliseconds used when retrieving an
// HTTP connection from the HTTP connection manager.
// Zero means to wait indefinitely.
client.getParams().setParameter("http.connection-manager.timeout",
_TimeoutCliConnection);
}
client.getParams().setParameter("http.tcp.nodelay", true);
return client;
}
});
Bye
I found, with HttpUnit 1.6.2 setting these
final HttpClient client = new HttpClient();
final GetMethod method = new GetMethod(pUrl);
client.setConnectionTimeout((int) timeout);
client.setTimeout((int) timeout);
final int statusCode = client.executeMethod(method);
Seemed to do the trick. Both are deprecated methods. :(