Develop Reference

How to get persistent HttpConnection with Apache HttpClient?

In my test application I execute consecutive HttpGet requests to the same host with Apache HttpClient but upon each next request it turns out that the previous HttpConnection is closed and the new HttpConnection is created.
I use the same instance of HttpClient and don't close responses. From each entity I get InputStream, read from it with Scanner and then close the Scanner. I have tested KeepAliveStrategy, it returns true. The time between requests doesn't exceed keepAlive or connectionTimeToLive durations.
Can anyone tell me what could be the reason for such behavior?
Updated
I have found the solution. In order to keep the HttpConnecton alive it is necessary to set HttpClientConnectionManager when building HttpClient. I have used BasicHttpClientConnectionManager.
ConnectionKeepAliveStrategy keepAliveStrat = new DefaultConnectionKeepAliveStrategy() {
#Override
public long getKeepAliveDuration(HttpResponse response, HttpContext context)
{
long keepAlive = super.getKeepAliveDuration(response, context);
if (keepAlive == -1)
keepAlive = 120000;
return keepAlive;
}
};
HttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager();
try (CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager) // without this setting connection is not kept alive
.setDefaultCookieStore(store)
.setKeepAliveStrategy(keepAliveStrat)
.setConnectionTimeToLive(120, TimeUnit.SECONDS)
.setUserAgent(USER_AGENT)
.build())
{
HttpClientContext context = new HttpClientContext();
RequestConfig config = RequestConfig.custom()
.setCookieSpec(CookieSpecs.DEFAULT)
.setSocketTimeout(10000)
.setConnectTimeout(10000)
.build();
context.setRequestConfig(config);
HttpGet httpGet = new HttpGet(uri);
CloseableHttpResponse response = httpClient.execute(httpGet, context);
HttpConnection conn = context.getConnection();
HttpEntity entity = response.getEntity();
try (Scanner in = new Scanner(entity.getContent(), ENC))
{
// do something
}
System.out.println("open=" + conn.isOpen()); // now open=true
HttpGet httpGet2 = new HttpGet(uri2); // on the same host with other path
// and so on
}
Updated 2
In general checking connections with conn.isOpen() is not proper way to check the connections state because: "Internally HTTP connection managers work with instances of ManagedHttpClientConnection acting as a proxy for a real connection that manages connection state and controls execution of I/O operations. If a managed connection is released or get explicitly closed by its consumer the underlying connection gets detached from its proxy and is returned back to the manager. Even though the service consumer still holds a reference to the proxy instance, it is no longer able to execute any I/O operations or change the state of the real connection either intentionally or unintentionally." (HttpClent Tutorial)
As have pointed #oleg the proper way to trace connections is using the logger.

First of all you need to make sure remote server you're working with does support keep-alive connections. Just simply check whether remote server does return header Connection: Keep-Alive or Connection: Closed in each and every response. For Close case there is nothing you can do with that. You can use this online tool to perform such check.
Next, you need to implement the ConnectionKeepAliveStrategy as defined in paragraph #2.6 of this manual. Note that you can use existent DefaultConnectionKeepAliveStrategy since HttpClient version 4.0, so that your HttpClient will be constructed as following:
HttpClient client = HttpClients.custom()
.setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE)
.build();
That will ensure you HttpClient instance will reuse the same connection via keep-alive mechanism if it is being supported by server.

Your application must be closing response objects in order to ensure proper resource de-allocation of the underlying connections. Upon response closure HttpClient keeps valid connections alive and returns them back to the connection manager (connection pool).
I suspect your code simply leaks connections and every request ens up with a newly created connection while all previous connections keep on piling up in memory.

From the example at HttpClient website:
// In order to ensure correct deallocation of system resources
// the user MUST call CloseableHttpResponse#close() from a finally clause.
// Please note that if response content is not fully consumed the underlying
// connection cannot be safely re-used and will be shut down and discarded
// by the connection manager.
So as #oleg said you need to close the HttpResponse before checking the connection status.

Apache HttpComponents BasicHttpClientConnectionManager

I recently switched from java.net to org.apache.http.client, I have setup a ClosableHttpClient with the HttpClientBuilder. As connection manager I am using the BasicHttpClientConnectionManager.
Now I have the problem that very often when I create some HTTP request I get a timeout exception. It seems that the connection manager is keeping connections open to reuse them but if the system is idle for a few minutes then this connection will timeout and when I make the next request the first thing I get is a timeout. Repeating the same request one more time then usually works without any problem.
Is there a way to configure the BasicHttpClientConnectionManager in order to not reuse its connections and create a new connection each time?

There several ways of dealing with the problem
Evict idle connections once no longer needed. The code below effectively disables connection persistence by closing out persistent connections after each HTTP exchange.
BasicHttpClientConnectionManager cm = new BasicHttpClientConnectionManager();
CloseableHttpClient httpclient = HttpClients.custom().setConnectionManager(cm).build();
...
try (CloseableHttpResponse response = httpclient.execute(new HttpGet("/"))) {
System.out.println(response.getStatusLine());
EntityUtils.consume(response.getEntity());
}
cm.closeIdleConnections(0, TimeUnit.MILLISECONDS);
Limit connection keep-alive time to something relatively small-ish
BasicHttpClientConnectionManager cm = new BasicHttpClientConnectionManager();
CloseableHttpClient httpclient = HttpClients.custom()
.setConnectionManager(cm)
.setKeepAliveStrategy((response, context) -> 1000)
.build();
try (CloseableHttpResponse response = httpclient.execute(new HttpGet("/"))) {
System.out.println(response.getStatusLine());
EntityUtils.consume(response.getEntity());
}
(Recommended) Use pooling connection manager and set connection total time to live to a finite value. There are no benefits to using the basic connection manager compared to the pooling one unless your code is expected to run in an EJB container.
CloseableHttpClient httpclient = HttpClients.custom()
.setConnectionTimeToLive(5, TimeUnit.SECONDS)
.build();
try (CloseableHttpResponse response = httpclient.execute(new HttpGet("/"))) {
System.out.println(response.getStatusLine());
EntityUtils.consume(response.getEntity());
}

When should the HttpClient be closed? [duplicate]

This question already has answers here:
What is the difference between CloseableHttpClient and HttpClient in Apache HttpClient API?
(8 answers)
Closed 2 years ago.
I am building an Android app that will fire multiple HTTP requests (say a request every second) to a server to fetch data. What are the best practices I must follow?
Should I create and close the client after each request, like the following?
CloseableHttpClient httpClient = HttpClientBuilder.create().build();
try {
HttpPost request = new HttpPost("http://yoururl");
StringEntity params = new StringEntity(json.toString());
request.addHeader("content-type", "application/json");
request.setEntity(params);
httpClient.execute(request);
// handle response here...
} catch (Exception ex) {
// handle exception here
} finally {
httpClient.close();
}
Or should I create a client initially, use it for all requests and then finally close it when I'm done with it?

The idea of closing your HttpClient is about releasing the allocated ressources. Therefore, It depends on how often you plan on firing those HTTP requests.
Keep in mind that firing a request every 10 seconds is considered an eternity ;)

class to read http pages with reliable timout that return `java.io.InputStream`

Is there any class for reading http pages that return a java.io.InputStream and its timeout be reliable?
I tried java.net.URLConnection and it doesn't have a reliable timeout (it takes more time that it set to timeout reach)? My Code is here:
URLConnection con = url.openConnection();
con.setConnectTimeout(2000);
con.setReadTimeout(2000);
InputStream in = con.getInputStream();

I expect that the reason that the timeout is not working for you is that you are setting the timeout after the connection has been established, or you are using the wrong setter. It is also possible that you are using "non-standard" version of URLConnection ...
"Some non-standard implementation of this method ignores the specified timeout. To see the read timeout set, please call getReadTimeout()." (or getConnectTimeout())
If you posted the relevant part of your actual code we could give you a better answer ...
Alternatively, use the Apache HttpClient library.

You can use Apache HttpClient to read http pages, it also has an http parser.check this for further reference about httpclient. you can get an InputStream object using their API like this.
HttpClient httpclient = new DefaultHttpClient();
// Prepare a request object
HttpGet httpget = new HttpGet("http://www.apache.org/");
// Execute the request
HttpResponse response = httpclient.execute(httpget);
// Examine the response status
System.out.println(response.getStatusLine());
// Get hold of the response entity
HttpEntity entity = response.getEntity();
// If the response does not enclose an entity, there is no need
// to worry about connection release
if (entity != null) {
InputStream instream = entity.getContent();
and coming to timeout part, it totally depends on the network and you cant do much about it from your java code.

Does Apache HttpClient add the Cookies set by the java.net.CookieHandler to Request?

My simple Apache HttpClient (4.0.1) client application makes an HttpGet request to a server URL in the main() method and prints the response. On startup, the application registers an implementation of java.net.CookieHandler in a static block.
On checking the cookies received on the server side, I found that the cookies are not being received by the Server when the HttpClient makes the GET request.
On the other hand, when I replaced the Apache HttpClient with a plain java.net.URL(HTTP_URL).openStream(), the cookies were set by the CookieHandler on the Request and were received by the Server.
Is it that CookieHandler does not work with Apache HttpClient?
Code:
Client.java
static {
CookieHandler.setDefault(new CookieHandler() {
public Map get(URI u, List r) {
return Collections.singletonMap("Cookie",
Collections.singletonList(COOKIE_STRING));
}
});
}
Using HttpClient (does not put cookies on request)
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet(HTTP_URL);
client.execute(get);
Using java.net.URL (sets the cookies on request)
URL url = new URL(HTTP_URL);
InputStream is = url.openStream();

Is it that CookieHandler does not work with Apache HttpClient?
That is correct.
The Apache HttpClient codebase uses its own cookie and cookie store representations / mechanisms. Here is a link to the relevant section of the HttpClient tutorial. (It is pretty sketchy, but if you look at the javadocs for the relevant classes, you should be able to figure out how to use it.)
(If you are using an older version of Apache HttpClient, beware that the APIs have changed significantly.)