Develop Reference

How to set socket timeout in Java HTTP Client

We want to migrate all our apache-httpclient-4.x code to java-http-client code to reduce dependencies. While migrating them, i ran into the following issue under java 11:
How to set the socket timeout in Java HTTP Client?
With apache-httpclient-4.x we can set the connection timeout and the socket timeout like this:
DefaultHttpClient httpClient = new DefaultHttpClient();
int timeout = 5; // seconds
HttpParams httpParams = httpClient.getParams();
httpParams.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, timeout * 1000);
httpParams.setParameter(CoreConnectionPNames.SO_TIMEOUT, timeout * 1000);
With java-http-client i can only set the connection timeout like this:
HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.build()
But i found no way to set the socket timeout. Is there any way or an open issue to support that in the future?

You can specify it at the HttpRequest.Builder level via the timeout method:
HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.build();
HttpRequest httpRequest = HttpRequest.newBuilder()
.uri(URI.create("..."))
.timeout(Duration.ofSeconds(5)) //this
.build();
httpClient.send(httpRequest, HttpResponse.BodyHandlers.ofString());
If you've got connected successfully but not able to receive a response at the desired amount of time, java.net.http.HttpTimeoutException: request timed out will be thrown (in contrast with java.net.http.HttpConnectTimeoutException: HTTP connect timed out which will be thrown if you don't get a successful connection).

There doesn't seem to be a way to specify a timeout on the flow of packets (socket timeout) on the Java Http Client.
I found an enhancement request on OpenJDK which seems to cover this possibility - https://bugs.openjdk.org/browse/JDK-8258397
Content from the link
The HttpClient lets you set a connection timeout (HttpClient.Builder) and a request timeout (HttpRequest.Builder). However the request timeout will be cancelled as soon as the response headers have been read. There is currently no timeout covering the reception of the body.
A possibility for the caller is to make use of the CompletableFuture API (get/join will accept a timeout, or CF::orTimeout can be called).
IIRC - in that case, it will still be the responsibility of the caller to cancel the request. We might want to reexamine and possibility change that.
The disadvantage here is that some of our BodyHandlers (ofPublisher, ofInputStream) will return immediately - so the CF API won't help in this case.
This might be a good thing (or not).
Another possibility could be to add a body timeout on HttpRequest.Builder. This would then cover all cases - but do we really want to timeout in the case of ofInputStream or ofPublisher if the caller doesn't read the body fast enough?

How to get persistent HttpConnection with Apache HttpClient?

In my test application I execute consecutive HttpGet requests to the same host with Apache HttpClient but upon each next request it turns out that the previous HttpConnection is closed and the new HttpConnection is created.
I use the same instance of HttpClient and don't close responses. From each entity I get InputStream, read from it with Scanner and then close the Scanner. I have tested KeepAliveStrategy, it returns true. The time between requests doesn't exceed keepAlive or connectionTimeToLive durations.
Can anyone tell me what could be the reason for such behavior?
Updated
I have found the solution. In order to keep the HttpConnecton alive it is necessary to set HttpClientConnectionManager when building HttpClient. I have used BasicHttpClientConnectionManager.
ConnectionKeepAliveStrategy keepAliveStrat = new DefaultConnectionKeepAliveStrategy() {
#Override
public long getKeepAliveDuration(HttpResponse response, HttpContext context)
{
long keepAlive = super.getKeepAliveDuration(response, context);
if (keepAlive == -1)
keepAlive = 120000;
return keepAlive;
}
};
HttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager();
try (CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager) // without this setting connection is not kept alive
.setDefaultCookieStore(store)
.setKeepAliveStrategy(keepAliveStrat)
.setConnectionTimeToLive(120, TimeUnit.SECONDS)
.setUserAgent(USER_AGENT)
.build())
{
HttpClientContext context = new HttpClientContext();
RequestConfig config = RequestConfig.custom()
.setCookieSpec(CookieSpecs.DEFAULT)
.setSocketTimeout(10000)
.setConnectTimeout(10000)
.build();
context.setRequestConfig(config);
HttpGet httpGet = new HttpGet(uri);
CloseableHttpResponse response = httpClient.execute(httpGet, context);
HttpConnection conn = context.getConnection();
HttpEntity entity = response.getEntity();
try (Scanner in = new Scanner(entity.getContent(), ENC))
{
// do something
}
System.out.println("open=" + conn.isOpen()); // now open=true
HttpGet httpGet2 = new HttpGet(uri2); // on the same host with other path
// and so on
}
Updated 2
In general checking connections with conn.isOpen() is not proper way to check the connections state because: "Internally HTTP connection managers work with instances of ManagedHttpClientConnection acting as a proxy for a real connection that manages connection state and controls execution of I/O operations. If a managed connection is released or get explicitly closed by its consumer the underlying connection gets detached from its proxy and is returned back to the manager. Even though the service consumer still holds a reference to the proxy instance, it is no longer able to execute any I/O operations or change the state of the real connection either intentionally or unintentionally." (HttpClent Tutorial)
As have pointed #oleg the proper way to trace connections is using the logger.

First of all you need to make sure remote server you're working with does support keep-alive connections. Just simply check whether remote server does return header Connection: Keep-Alive or Connection: Closed in each and every response. For Close case there is nothing you can do with that. You can use this online tool to perform such check.
Next, you need to implement the ConnectionKeepAliveStrategy as defined in paragraph #2.6 of this manual. Note that you can use existent DefaultConnectionKeepAliveStrategy since HttpClient version 4.0, so that your HttpClient will be constructed as following:
HttpClient client = HttpClients.custom()
.setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE)
.build();
That will ensure you HttpClient instance will reuse the same connection via keep-alive mechanism if it is being supported by server.

Your application must be closing response objects in order to ensure proper resource de-allocation of the underlying connections. Upon response closure HttpClient keeps valid connections alive and returns them back to the connection manager (connection pool).
I suspect your code simply leaks connections and every request ens up with a newly created connection while all previous connections keep on piling up in memory.

From the example at HttpClient website:
// In order to ensure correct deallocation of system resources
// the user MUST call CloseableHttpResponse#close() from a finally clause.
// Please note that if response content is not fully consumed the underlying
// connection cannot be safely re-used and will be shut down and discarded
// by the connection manager.
So as #oleg said you need to close the HttpResponse before checking the connection status.

HttpClientError: The target server failed to respond

I am trying to hit a server using HTTP client using PoolingClientConnectionManager setting max connections for individual hosts
//Code that initializes my connection manager and HTTP client
HttpParams httpParam = httpclient.getParams();
HttpConnectionParams.setSoTimeout(httpParam, SOCKET_TIMEOUT);
HttpConnectionParams.setConnectionTimeout(httpParam, CONN_TIMEOUT);
httpclient.setParams(httpParam);
//Run a thread which closes Expired connections
new ConnectionManager(connManager).start();
//Code that executes my request
HttpPost httpPost = new HttpPost(url);
HttpEntity httpEntity = new StringEntity(request, "UTF-8");
httpPost.setEntity(httpEntity);
Header acceptEncoding = new BasicHeader("Accept-Encoding", "gzip,deflate");
httpPost.setHeader(acceptEncoding);
if(contenttype != null && !contenttype.equals("")){
Header contentType = new BasicHeader("Content-Type", contenttype);
httpPost.setHeader(contentType);
}
InputStream inputStream = null;
LOG.info(dataSource + URL + url + REQUEST + request);
HttpResponse response = httpclient.execute(httpPost);
That is we are using Connection pooling for http persistence .
We are getting this error sporadically :
The target server failed to respond
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
Does any one know how to resolve this?
We are shutting down idle connections as well.
Can some Please help.
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html

Probably, it is a bug in the HttpClient.
If you are using the HttpClient 4.4, please try to upgrade to 4.4.1.
If you want for more information, please look at this link.
If you can't upgrade, the following links might be helpful.
http://www.nuxeo.com/blog/using-httpclient-properly-avoid-closewait-tcp-connections/
Good luck!

Recently faced similar while using HttpClient 5.
On enabling the HttpClient logs and found that the issue was due to stale connections.
Adding the below helped to solve the issue, it detects and validates the connections that have become stale while kept inactive in the pool before reuse.
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setValidateAfterInactivity(timeinmilliseconds);

HttpClient 4 redirect handling different inside Tomcat

I've got an HttpClient instance that fetches a remote resource. I configure it to handle redirects.
HttpParams params = new BasicHttpParams();
params.setIntParameter(CoreConnectionPNames.SO_TIMEOUT,
SOCKET_TIMEOUT);
params.setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
CONNECTION_TIMEOUT);
params.setLongParameter(ClientPNames.CONN_MANAGER_TIMEOUT,
CONN_MANAGER_TIMEOUT_VALUE);
params.setParameter(ClientPNames.COOKIE_POLICY,
CookiePolicy.BROWSER_COMPATIBILITY);
params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, true);
params.setBooleanParameter(ClientPNames.REJECT_RELATIVE_REDIRECT,
false);
params.setIntParameter(ClientPNames.MAX_REDIRECTS, 4);
httpclient = new DefaultHttpClient(cm, params);
When I'm calling it from inside a webapp (Tomcat6) I get the 301 response. When I call it from JSE environment I get the 200 final response (redirects get handled). My first suspect was classloading issues, but printing out the source of HttpClient class shows that both times it's loaded from httpclient-4.2.5.jar
Any ideas how else I can debug this?

Run HttpClient with the context / wire logging turned on as described here and compare HTTP message exchanged in both environments.

The HttpClient instance was shared throughout the webapp, including SolrJ (Solr client), which set the "follow redirect" param to false. I figured this out by creating a copy of the RequestDirector with extra logging lines. I could have simply looked for all calls of HttpClient.getParams(). The more you know.

DefaultHttpClient timeout set to 10 seconds but only takes 1 second to timeout

I have this static function which basicly makes a connection to a webpage, send post data along with it and return the received response (JSON object)
The problem that I am having is that no matter what timeout i set, it very often gives a timeout when it is only trying 1 second, with the timeout being 6 seconds that should not happen.
public static String makeRequest(String path, String info) throws Exception
{
HttpParams httpParameters = new BasicHttpParams();
int timeoutConnection = 6000;
HttpConnectionParams.setConnectionTimeout(httpParameters, timeoutConnection);
int timeoutSocket = 6000;
HttpConnectionParams.setSoTimeout(httpParameters, timeoutSocket);
DefaultHttpClient httpclient = new DefaultHttpClient(httpParameters);
HttpPost httpost = new HttpPost(path);
StringEntity jsonobj = new StringEntity(info);
httpost.setEntity(jsonobj);
httpost.setHeader("Accept", "application/json");
httpost.setHeader("Content-type", "application/json");
ResponseHandler responseHandler = new BasicResponseHandler();
String response = httpclient.execute(httpost, responseHandler);
return response;
Now I have seen some issues like this but I could not find an answer that could help me.
Some say it is due to the fact that it is not threadsafe, however, I do not do multiple calls at the same time, it is all done in order. The issue evens occurs on the first try, which should not happen because of this reason as there are no multiple connection/httpposts yet, let alone from different threads.
However it does happen a lot lately, sometimes it did not occur for a few days or barely in those days.
I tried looking at the AndroidHttpClient, but it does not seem to support HttpPost, so that is not usefull either (or I am wrong with this?)
The data for both path and info is correct, tested it. Also my server does not have issues.
Some say it can be your network, but I have it on 3 wifi networks tested today. Strangly, on the internet connection of my mobile provider it does not or barely occur.
I have read in one answer that it may be because of the ISP changing header information. I have tried using different values for the user-agent but that also did not work.
I hope someone can be of help and for that i would be grateful as it really frustrates me that this keeps happening.

If TCP detects a connection reject (i.e. an incoming RST) before the timeout period, it is a failure. There is no point in waiting out the timeout in this case, and it is to be expected that a retry will only fail again the same way. The timeout is for the case when there is no response.