Develop Reference

Apache HttpClient Keep-Alive Strategy for active connections

In an Apache HttpClient with a PoolingHttpClientConnectionManager, does the Keep-Alive strategy change the amount of time that an active connection will stay alive until it will be removed from the connection pool? Or will it only close out idle connections?
For example, if I set my Keep-Alive strategy to return 5 seconds for every request, and I use the same connection to hit a single URL/route once every 2 seconds, will my keep-alive strategy cause this connection to leave the pool? Or will it stay in the pool, because the connection is not idle?

I just tested this and confirmed that the Keep-Alive strategy will only idle connections from the HttpClient's connection pool after the Keep-Alive duration has passed. The Keep-Alive duration determines whether or not the connection is idle, in fact - if the Keep-alive strategy says to keep connections alive for 10 seconds, and we receive responses from the server every 2 seconds, the connection will be kept alive for 10 seconds after the last successful response.
The test that I ran was as follows:
I set up an Apache HttpClient (using a PoolingHttpClientConnectionManager) with the following ConnectionKeepAliveStrategy:
return (httpResponse, httpContext) -> {
// Honor 'keep-alive' header
HeaderElementIterator it = new BasicHeaderElementIterator(
httpResponse.headerIterator(HTTP.CONN_KEEP_ALIVE));
while (it.hasNext()) {
HeaderElement he = it.nextElement();
String param = he.getName();
String value = he.getValue();
if (value != null && param.equalsIgnoreCase("timeout")) {
try {
return Long.parseLong(value) * 1000;
} catch(NumberFormatException ignore) {
}
}
}
if (keepAliveDuration <= 0) {
return -1; // the connection will stay alive indefinitely.
}
return keepAliveDuration * 1000;
};
}
I created an endpoint on my application which used the HttpClient to make a GET request to a URL behind a DNS.
I wrote a program to hit that endpoint every 1 second.
I changed my local DNS for the address that the HttpClient was sending GET requests to to point to a dummy URL that would not respond to requests. (This was done by changing my /etc/hosts file).
When I had set the keepAliveDuration to -1 seconds, even after changing the DNS to point to the dummy URL, the HttpClient would continuously send requests to the old IP address, despite the DNS change. I kept this test running for 1 hour and it continued to send requests to the old IP address associated with the stale DNS. This would happen indefinitely, as my ConnectionKeepAliveStrategy had been configured to keep the connection to the old URL alive indefinitely.
When I had set the keepAliveDuration to 10, after I had changed my DNS, I sent successful requests continuously, for about an hour. It wasn't until I turned off my load test and waited 10 seconds until we received a new connection. This means that the ConnectionKeepAliveStrategy removed the connection from the HttpClient's connection pool 10 seconds after the last successful response from the server.
Conclusion
By default, if an HttpClient does not receive a Keep-Alive header from a response it gets from a server, it assumes its connection to that server can be kept alive indefinitely, and will keep that connection in it's PoolingHttpClientConnectionManager indefinitely.
If you set a ConnectionKeepAliveStrategy like I did, then it will add a Keep-Alive header to the response from the server. Having a Keep-Alive header on the HttpClient response will cause the connection to leave the connection pool after the Keep-Alive duration has passed, after the last successful response from the server. This means that only idle connections are affected by the Keep-Alive duration, and "idle connections" are connections that haven't been used since the Keep-Alive duration has passed.

Embedded Jetty Closing Open Connections During Graceful Termination

I have written a basic embedded jetty server (with jersey to define endpoints), and need it to terminate gracefully (in response to a SIGINT). I would like all in-flight requests to finish processing and return results to the requestor (up until some timeout), while not accepting any new connections/requests.
The current behavior that I'm seeing is that in-flight requests are completed correctly, but the connections to clients are terminated (so they don't receive the responses to their requests).
When creating my Server instance, I set setStopAtShutdown to true and set a value for setStopTimeout. I also configure the StatisticsHandler. From what I can tell from the docs, this is what's required to enable graceful termination.
Here is my jetty server set up:
// For additional Jetty configuration options see:
// https://www.eclipse.org/jetty/documentation/current/embedding-jetty.html#_like_jetty_xml
public static Server createServer(
int port, int shutdownGracePeriodMillis, int httpServerThreads, int httpIdleTimeoutMillis) {
// Create server
QueuedThreadPool threadPool = new QueuedThreadPool();
threadPool.setMaxThreads(httpServerThreads);
Server server = new Server(threadPool);
HttpConfiguration config = new HttpConfiguration();
ServerConnector http = new ServerConnector(server, new HttpConnectionFactory(config));
http.setPort(port);
http.setIdleTimeout(httpIdleTimeoutMillis);
server.addConnector(http);
// Enable StatisticsHandler (required for graceful termination)
StatisticsHandler stats = new StatisticsHandler();
stats.setHandler(server.getHandler());
server.setHandler(stats);
ServerConnectionStatistics.addToAllConnectors(server);
// Configure graceful termination
server.setStopAtShutdown(true);
server.setStopTimeout(shutdownGracePeriodMillis);
// Associate Jersey with Jetty
JettyHttpContainer container =
ContainerFactory.createContainer(JettyHttpContainer.class, new AppResourceConfig());
server.setHandler(container);
return server;
}
Here is my jersey resource. It uses Thread.sleep to simulate some long-running request that needs to complete before the server shuts down.
#POST
#Consumes("application/json")
#Produces(MediaType.TEXT_PLAIN)
public String testEndpoint() {
for(int i = 0; i < 4; i++) {
try {
logger.info("Number: " + i);
Thread.sleep(5000);
} catch (Exception e) { }
}
logger.info("Request completed!");
return "Hello, world!\n";
}
I run the server and issue the following curl command
curl -v --header "Content-Type: application/json" \
--request POST \
--data '' \
http://localhost:4200/testEndpoint
When I send a SIGINT to the server process while this request is still waiting for a response I see on the server:
INFO c.a.a.s.resources.Resource - Number: 0
INFO c.a.a.s.resources.Resource - Number: 1
^C {{NOTE: SIGINT issued here}}
[Thread-1] INFO o.e.jetty.server.AbstractConnector - Stopped ServerConnector#32c726ee{HTTP/1.1,[http/1.1]}{0.0.0.0:4200}
INFO c.a.a.s.resources.AuctionResource - Number: 2
INFO c.a.a.s.resources.AuctionResource - Number: 3
INFO c.a.a.s.resources.AuctionResource - Request completed!
We can see from the server logs that the request is completed correctly. The curl command never gets its response of "Hello, world!" though (which it does in a regular not interrupted request).
The curl output looks like this:
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 4200 (#0)
> POST /testEndpoint HTTP/1.1
> Host: localhost:4200
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 0
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
I expected that I should get the correct reply from the service, but instead get an empty reply from the server. What further config is required to not close the connections of in-flight requests during graceful termination for an embedded jetty server?

How to set TCP keep Alive from HttpClient?

My Java application which resides in AWS private subnet connects to an http server via AWS Nat gateway. I am calling a POST request via HttpClient to the HTTP server. That request will take more than 10 minutes to complete. I have configured a socket time out and connection timeout of 1 hour as this this a background task . But the intermediate AWS NAT gateway will send back a RST packet after 300 secs [5 mins] and cause the connection to get resetted , there is no way i can increase the NAT gateway timeout. So i need to handle the problem from my application side.
My strategy is to use a TCP keep alive time which will send a packet say every 240 secs to keep the connection active. I have configured this
as below
CloseableHttpClient httpClient = HttpClients.createDefault()
HttpParams params = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(params, 3600000); //connection Timeout
HttpConnectionParams.setSoTimeout(params, 3600000); // Socket Time out
HttpConnectionParams.setSoKeepalive(params, true); //Enable Socket level keep alive time
and then call the post request via execute method
HttpPost post = new HttpPost("http://url");
HttpResponse response = httpClient.execute(post);
Since I am using a Linux system I have configured the server with following sysctl values:
sysctl -w net.ipv4.tcp_keepalive_time=240
sysctl -w net.ipv4.tcp_keepalive_intvl=240
sysctl -w net.ipv4.tcp_keepalive_probes=10
But while executing the program the keep alive is not enabled and connection fails as previous.
I have checked this with netstat -o option and as shown below keep alive is off
tcp 0 0 192.168.1.141:43770 public_ip:80 ESTABLISHED 18134/java off (0.00/0/0)
Is there any way i can set TCP keep alive from java code using httpclient . Also I can see HttpConnectionParams are deprecated. But I couldn't find any new class which can set keep alive

I have found a solution to the problem . Curious case is there is no way i can use some builder class in httpclient to pass socket keep alive . One method as i specified in the question is using HttpConnectionParams as below but this is not working and this class is now deprecated.
HttpParams params = httpClient.getParams();
HttpConnectionParams.setSoKeepalive(params, true);
So while checking apache http docs I can see that now connection parameters are passed to httpclient via RequestConfig class . Builders of this class provide solution to set connection_time_out and socket_time_out. But checking the socurce code of this I couldnt see an option to enable SocketKeepAlive which is what we want. So the only solution is directly creating a Socket using SocketBuilder class and pass that to the HttpClientBuilder.
Following is the working code
SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setSoTimeout(3600000).build(); //We need to set socket keep alive
RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(3600000).build();
CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).
setDefaultSocketConfig(socketConfig).build();
HttpPost post = new HttpPost(url.toString());
HttpResponse response = httpClient.execute(post);
While executing above i can see that keep alive is properly set in the socket based on the sysctl values i set in linux kernel
tcp 0 0 localip:48314 public_ip:443 ESTABLISHED 14863/java keepalive (234.11/0/0)
If some one has a better solution to enable Socket Keep alive from Requestconfig class or any other high level builder class i am open to suggestions.

Keeping an HTTP connection open but inactive for a long period is a bad design choice. HTTP is a request-response protocol, with the implication that requests and responses are quick.
Holding a connection open holds resources. From the perspective of the server (and network firewalls and routers) a client that opens a connection and begins a request (A POST in your case) but does not send any bytes for a long period is indistinguishable from a client that will never send any more data, because it is faulty or malicious (conducting a DOS attack). The server (and network hardware) is right to conclude that the right thing to do is to shutdown the connection and reclaim the resources used for it. You are trying to fight against correct behaviour that occurs for good reasons. Even if you manage to workaround the TCP shutdown you will find other problems, such as HTTP server timeouts and database timeouts.
You should instead be reconsidered the design of communication between the two components. That is, this looks like an XY Problem. You might consider
Having the client wait until it has a complete upload to perform before starting the POST.
Splitting the uploads into smaller, more frequent uploads.
Use a protocol other than HTTP.

The approach above with Socket worked beautifully with a reset of tcp_keepalive_intvl value below the AWS Network Load Balancer timeout. Using both, reset the NLB tcp idle timeout that allowed java hour+ connections.

Sometimes, if the configuration is overwritten, the configuration does not take effect.My initial modification of setDefaultSocketConfig in buildClient didn't take effect.Because it is overwritten by getConnectionManager()
public CloseableHttpClient buildClient() throws Exception {
HttpClientBuilder builder = HttpClientBuilder.create()
.setDefaultSocketConfig(SocketConfig.custom().setSoKeepAlive(true).build()) // did not work
.setConnectionManager(getConnectionManager())
.setRetryHandler(getRequestRetryHandler())
.setConnectionReuseStrategy(getConnectionReuseStrategy())
.setDefaultConnectionConfig(getConnectionConfig())
.setDefaultRequestConfig(getRequestConfig())
.setDefaultHeaders(getDefaultHeaders())
.setDefaultCredentialsProvider(getDefaultCredentialsProvider())
.disableContentCompression() // gzip is not needed. Use lz4 when compress=1
.setDefaultCookieStore(cookieStoreProvider.getCookieStore(properties))
.disableRedirectHandling();
String clientName = properties != null ? properties.getClientName() : null;
if (!Utils.isNullOrEmptyString(clientName)) {
builder.setUserAgent(clientName);
}
return builder.build();
And then I move the config to getConnectionManager()，and it work.
private PoolingHttpClientConnectionManager getConnectionManager()
throws CertificateException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException, IOException {
RegistryBuilder<ConnectionSocketFactory> registry = RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", PlainConnectionSocketFactory.getSocketFactory());
if (properties.getSsl()) {
HostnameVerifier verifier = "strict".equals(properties.getSslMode()) ? SSLConnectionSocketFactory.getDefaultHostnameVerifier() : NoopHostnameVerifier.INSTANCE;
registry.register("https", new SSLConnectionSocketFactory(getSSLContext(), verifier));
}
//noinspection resource
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(
registry.build(),
null,
null,
new IpVersionPriorityResolver(),
properties.getTimeToLiveMillis(),
TimeUnit.MILLISECONDS
);
connectionManager.setDefaultMaxPerRoute(properties.getDefaultMaxPerRoute());
connectionManager.setMaxTotal(properties.getMaxTotal());
connectionManager.setDefaultConnectionConfig(getConnectionConfig());
connectionManager.setDefaultSocketConfig(SocketConfig.custom().setSoKeepAlive(true).build());
return connectionManager;
}

Play 2.5 WebSocket Connection Build

I have an AWS server (medium) running in EU West, and there are roughly 250 devices connected but are also always reconnecting due to internet connectivity issues, but for some reason, the amount of TCP connections to the server grows until it reaches around 4300. Then no new connections are allowed to the server. I have confirmed that it is isolated to WebSocket requests and not regular HTTP requests.
WebSocket connections are kept per device in a Map with device UUID as key; it sometimes happens that a device will send a request for a new WS connection even though the server has a connection to the device. In this case, the current connection is closed, and an error is returned so that the device can retry the connection request.
Below is the code snippet from the Controller handling the connections using LegacyWebSocket. Connections are closed using out.close() as per https://www.playframework.com/documentation/2.5.x/JavaWebSockets#handling-websockets-using-callbacks
public LegacyWebSocket<String> create(String uuid) {
logger.debug("NEW WebSocket request from {}, creating new socket...", uuid);
if(webSocketMap.containsKey(uuid)){
logger.debug("WebSocket already exists for {}, closing existing connection", uuid);
webSocketMap.get(uuid).close();
logger.debug("Responding forbidden to force WS restart from device {}", uuid);
return WebSocket.reject(forbidden());
}
LegacyWebSocket<String> ws = WebSocket.whenReady((in, out) -> {
logger.debug("Adding downstream connection to webSocketMap-> {} webSocketMap.size() = {}",uuid, webSocketMap.size());
webSocketMap.put(uuid,out);
// For each event received on the socket,
in.onMessage(message->{
if(message.equals("ping")){
logger.debug("PING received from {} {}",uuid, message);
out.write("pong");
}
});
// When the socket is closed.
in.onClose(() -> {
logger.debug("onClose, removing for {}",uuid);
webSocketMap.remove(uuid);
});
});
return ws;
}
How can I ensure that Play Framework closes the TCP connection for closed WS connections?
The call that I use to check the amount of TCP connections is netstat -n -t | wc -l

Looks like a TCP keep-alive issue - i.e. that the TCP connections become stale because of connectivity issues on the client side and the server does not handle or clean up the stale connections in time before the limit is reached.
This link will help you configure the TCP keep-alive on your server to ensure that the stale connections are cleaned up in time.

linux network setting POST failing after exactly 2 hours

I have following test code running fine for a very long POST request (more than 2 hours):
URL postURL = new URL(url);
con = (HttpURLConnection) postURL.openConnection();
con.setUseCaches(false);
con.setDoOutput(true);
con.setDoInput(true);
con.setRequestMethod("POST");
OutputStream out = con.getOutputStream();
OutputStreamWriter wout = new OutputStreamWriter(out, "UTF-8");
wout.write(xmlRequest);
wout.flush();
out.close();
con.connect();
int httpResponseCode = HttpURLConnection.HTTP_SERVER_ERROR;
try {
httpResponseCode = con.getResponseCode();
} catch (IOException e) {
log(e.toString() + " Error retrieving the status code");
}
log("Failure status code " + httpResponseCode + " received: ");
I run this post from one host to another and it runs fine in all environments, except one exact linux redhat host -when i run this code from this host I got exception:
java.net.SocketException: recv() failed, errno = 104 Connection reset by peer Error retrieving the status code
Failure status code 500 received.
Target server host is the same host in all tests. So the difference is only in client caller host.
So I'm trying to understand what exact tcp setting on this linux machine is causing receive to fail after exactly 2 hours.
I agree to get any blame here for such "incorrect" using of the sending post and waiting for more then 2 hours for response;) but question is what causing this

google for 7200 seconds tcp redhat and among other things
tcp_keepalive_time (integer; default: 7200; since Linux 2.2)
The number of seconds a connection needs to be idle before TCP
begins sending out keep-alive probes. Keep-alives are only sent
when the SO_KEEPALIVE socket option is enabled. The default
value is 7200 seconds (2 hours). An idle connection is termi-
nated after approximately an additional 11 minutes (9 probes an
interval of 75 seconds apart) when keep-alive is enabled.
Note that underlying connection tracking mechanisms and applica-
tion timeouts may be much shorter.

the difference was that one client host was on different network segment and router was killing idle sessions after timeout.