I have written a basic embedded jetty server (with jersey to define endpoints), and need it to terminate gracefully (in response to a SIGINT). I would like all in-flight requests to finish processing and return results to the requestor (up until some timeout), while not accepting any new connections/requests.
The current behavior that I'm seeing is that in-flight requests are completed correctly, but the connections to clients are terminated (so they don't receive the responses to their requests).
When creating my Server instance, I set setStopAtShutdown to true and set a value for setStopTimeout. I also configure the StatisticsHandler. From what I can tell from the docs, this is what's required to enable graceful termination.
Here is my jetty server set up:
// For additional Jetty configuration options see:
// https://www.eclipse.org/jetty/documentation/current/embedding-jetty.html#_like_jetty_xml
public static Server createServer(
int port, int shutdownGracePeriodMillis, int httpServerThreads, int httpIdleTimeoutMillis) {
// Create server
QueuedThreadPool threadPool = new QueuedThreadPool();
threadPool.setMaxThreads(httpServerThreads);
Server server = new Server(threadPool);
HttpConfiguration config = new HttpConfiguration();
ServerConnector http = new ServerConnector(server, new HttpConnectionFactory(config));
http.setPort(port);
http.setIdleTimeout(httpIdleTimeoutMillis);
server.addConnector(http);
// Enable StatisticsHandler (required for graceful termination)
StatisticsHandler stats = new StatisticsHandler();
stats.setHandler(server.getHandler());
server.setHandler(stats);
ServerConnectionStatistics.addToAllConnectors(server);
// Configure graceful termination
server.setStopAtShutdown(true);
server.setStopTimeout(shutdownGracePeriodMillis);
// Associate Jersey with Jetty
JettyHttpContainer container =
ContainerFactory.createContainer(JettyHttpContainer.class, new AppResourceConfig());
server.setHandler(container);
return server;
}
Here is my jersey resource. It uses Thread.sleep to simulate some long-running request that needs to complete before the server shuts down.
#POST
#Consumes("application/json")
#Produces(MediaType.TEXT_PLAIN)
public String testEndpoint() {
for(int i = 0; i < 4; i++) {
try {
logger.info("Number: " + i);
Thread.sleep(5000);
} catch (Exception e) { }
}
logger.info("Request completed!");
return "Hello, world!\n";
}
I run the server and issue the following curl command
curl -v --header "Content-Type: application/json" \
--request POST \
--data '' \
http://localhost:4200/testEndpoint
When I send a SIGINT to the server process while this request is still waiting for a response I see on the server:
INFO c.a.a.s.resources.Resource - Number: 0
INFO c.a.a.s.resources.Resource - Number: 1
^C {{NOTE: SIGINT issued here}}
[Thread-1] INFO o.e.jetty.server.AbstractConnector - Stopped ServerConnector#32c726ee{HTTP/1.1,[http/1.1]}{0.0.0.0:4200}
INFO c.a.a.s.resources.AuctionResource - Number: 2
INFO c.a.a.s.resources.AuctionResource - Number: 3
INFO c.a.a.s.resources.AuctionResource - Request completed!
We can see from the server logs that the request is completed correctly. The curl command never gets its response of "Hello, world!" though (which it does in a regular not interrupted request).
The curl output looks like this:
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 4200 (#0)
> POST /testEndpoint HTTP/1.1
> Host: localhost:4200
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 0
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
I expected that I should get the correct reply from the service, but instead get an empty reply from the server. What further config is required to not close the connections of in-flight requests during graceful termination for an embedded jetty server?
Related
I have a problem. I am using WSO2 Micro Integration 4.1. I am sending request after "STATE_DESCRIPTION = Socket Timeout occurred after accepting the request headers and the request body, INTERNAL_STATE = REQUEST_DONE, DIRECTION = REQUEST, CAUSE_OF_ERROR = Connection between the client and the EI timeouts, HTTP_URL = /api/v1/test, HTTP_METHOD = POST, SOCKET_TIMEOUT = 180000, CLIENT_ADDRESS = /127.0.0.1:52694" is return my console and Error: socket hang up is return from postman. I tried http.connection.disable.keepalive=true in passthru-http.properties. I need help.
I tried http.connection.disable.keepalive=true in passthru-http.properties
As the description suggests this happened due to connection got time out. If you are having slow performing backend you can increase the timeout so that the server will wait until the response.
As an example following is the config to increase the synapse global timeout.
[synapse_properties]
'synapse.global_timeout_interval'=3000
Please note that you need to add the configurations on the deployment.toml file not in the passthru-http.properties file.
In that case if you want to make suggested change on the keep-alive settings, Do it as follows on the deployment.toml file.
[transport.http]
socket_timeout = 180000 # timeout in milliseconds
disable_connection_keepalive = true
connection_timeout = 90000 # in milliseconds
Apart from Check whether the 52694 port is accessible too as the server is localhost means that it is the same server as the MI is running.
In an Apache HttpClient with a PoolingHttpClientConnectionManager, does the Keep-Alive strategy change the amount of time that an active connection will stay alive until it will be removed from the connection pool? Or will it only close out idle connections?
For example, if I set my Keep-Alive strategy to return 5 seconds for every request, and I use the same connection to hit a single URL/route once every 2 seconds, will my keep-alive strategy cause this connection to leave the pool? Or will it stay in the pool, because the connection is not idle?
I just tested this and confirmed that the Keep-Alive strategy will only idle connections from the HttpClient's connection pool after the Keep-Alive duration has passed. The Keep-Alive duration determines whether or not the connection is idle, in fact - if the Keep-alive strategy says to keep connections alive for 10 seconds, and we receive responses from the server every 2 seconds, the connection will be kept alive for 10 seconds after the last successful response.
The test that I ran was as follows:
I set up an Apache HttpClient (using a PoolingHttpClientConnectionManager) with the following ConnectionKeepAliveStrategy:
return (httpResponse, httpContext) -> {
// Honor 'keep-alive' header
HeaderElementIterator it = new BasicHeaderElementIterator(
httpResponse.headerIterator(HTTP.CONN_KEEP_ALIVE));
while (it.hasNext()) {
HeaderElement he = it.nextElement();
String param = he.getName();
String value = he.getValue();
if (value != null && param.equalsIgnoreCase("timeout")) {
try {
return Long.parseLong(value) * 1000;
} catch(NumberFormatException ignore) {
}
}
}
if (keepAliveDuration <= 0) {
return -1; // the connection will stay alive indefinitely.
}
return keepAliveDuration * 1000;
};
}
I created an endpoint on my application which used the HttpClient to make a GET request to a URL behind a DNS.
I wrote a program to hit that endpoint every 1 second.
I changed my local DNS for the address that the HttpClient was sending GET requests to to point to a dummy URL that would not respond to requests. (This was done by changing my /etc/hosts file).
When I had set the keepAliveDuration to -1 seconds, even after changing the DNS to point to the dummy URL, the HttpClient would continuously send requests to the old IP address, despite the DNS change. I kept this test running for 1 hour and it continued to send requests to the old IP address associated with the stale DNS. This would happen indefinitely, as my ConnectionKeepAliveStrategy had been configured to keep the connection to the old URL alive indefinitely.
When I had set the keepAliveDuration to 10, after I had changed my DNS, I sent successful requests continuously, for about an hour. It wasn't until I turned off my load test and waited 10 seconds until we received a new connection. This means that the ConnectionKeepAliveStrategy removed the connection from the HttpClient's connection pool 10 seconds after the last successful response from the server.
Conclusion
By default, if an HttpClient does not receive a Keep-Alive header from a response it gets from a server, it assumes its connection to that server can be kept alive indefinitely, and will keep that connection in it's PoolingHttpClientConnectionManager indefinitely.
If you set a ConnectionKeepAliveStrategy like I did, then it will add a Keep-Alive header to the response from the server. Having a Keep-Alive header on the HttpClient response will cause the connection to leave the connection pool after the Keep-Alive duration has passed, after the last successful response from the server. This means that only idle connections are affected by the Keep-Alive duration, and "idle connections" are connections that haven't been used since the Keep-Alive duration has passed.
I use reactor-netty to request a set of URLs. Majority of URLs belong to the same hosts. reactor-netty seems to make a brand new TCP connection for every URL even if connection to the host is already established for the previous URL. Some servers drop new connections or start to respond slowly when hundreds of simultaneous connections established.
Sample of the code:
Flux.just(...)
.groupBy(link -> {
String host = "";
try {
host = new URL(link).getHost();
} catch (MalformedURLException e) {
LOGGER.warn("Cannot determine host {}", link, e);
}
return host;
})
.flatMap(group -> {
HttpClient client = HttpClient.create()
.keepAlive(true)
.tcpConfiguration(tcp -> tcp.host(group.key()));
return group.flatMap(link -> client.get()
.uri(link)
.response((resp, cont) -> resp.status().code() == 200 ? cont.aggregate().asString() : Mono.empty())
.doOnSubscribe(s -> LOGGER.debug("Requesting {}", link))
.timeout(Duration.ofMinutes(1))
.doOnError(e -> LOGGER.warn("Cannot get response from {}", link, e))
.onErrorResume(e -> Flux.empty())
.collect(Collectors.joining())
.filter(s -> StringUtils.isNotBlank(s)));
})
.blockLast();
In the log I see that local ports are different for the same remote host and sum of active and inactive connections are way higher than the number of distinct hosts. That's why I think that reactor-netty is not reusing already established connections.
DEBUG [2019-04-29 08:15:18,711] reactor-http-nio-10 r.n.r.PooledConnectionProvider: [id: 0xaed18e87, L:/192.168.1.183:56832 - R:capcp2.naad-adna.pelmorex.com/52.242.33.4:80] Releasing channel
DEBUG [2019-04-29 08:15:18,711] reactor-http-nio-10 r.n.r.PooledConnectionProvider: [id: 0xaed18e87, L:/192.168.1.183:56832 - R:capcp2.naad-adna.pelmorex.com/52.242.33.4:80] Channel cleaned, now 1 active connections and 239 inactive connections
...
DEBUG [2019-04-29 08:15:20,158] reactor-http-nio-10 r.n.r.PooledConnectionProvider: [id: 0xd6c6c5db, L:/192.168.1.183:56965 - R:capcp2.naad-adna.pelmorex.com/52.242.33.4:80] Releasing channel
DEBUG [2019-04-29 08:15:20,158] reactor-http-nio-10 r.n.r.PooledConnectionProvider: [id: 0xd6c6c5db, L:/192.168.1.183:56965 - R:capcp2.naad-adna.pelmorex.com/52.242.33.4:80] Channel cleaned, now 0 active connections and 240 inactive connections
Is it possible to request several URLs on the same host using keep-alive HTTP client through the same TCP connection to the host? If not, how do I restrict the number of simultaneous connections to the same host or perform requests to the same host sequentially (the next request only after receiving response to the previous one)?
I use Californium-SR6 release train.
Yes, reactor netty supports keep-alive, connection reuse, and connection pooling.
Note that .flatMap is a async operation that processes the inner streams in parallel. Therefore, when you call group.flatMap(... the inner requests will be executed in parallel. And since they are executed in parallel, multiple connections will need to be established.
If you want to execute requests to the same host sequentially, change your example to use group.concatMap instead of .flatMap.
If you want to still execute them in parallel, but limit the number of active requests to an individual host, then change your example to use one of the overloaded versions of .flatMap that takes a concurrency parameter.
Also, since you are using HttpClient.create(), your example uses the default global http connection pool. If you want more control over connection pooling, you can specify a different ConnectionProvider via HttpClient.create(ConnectionProvider).
My Java application which resides in AWS private subnet connects to an http server via AWS Nat gateway. I am calling a POST request via HttpClient to the HTTP server. That request will take more than 10 minutes to complete. I have configured a socket time out and connection timeout of 1 hour as this this a background task . But the intermediate AWS NAT gateway will send back a RST packet after 300 secs [5 mins] and cause the connection to get resetted , there is no way i can increase the NAT gateway timeout. So i need to handle the problem from my application side.
My strategy is to use a TCP keep alive time which will send a packet say every 240 secs to keep the connection active. I have configured this
as below
CloseableHttpClient httpClient = HttpClients.createDefault()
HttpParams params = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(params, 3600000); //connection Timeout
HttpConnectionParams.setSoTimeout(params, 3600000); // Socket Time out
HttpConnectionParams.setSoKeepalive(params, true); //Enable Socket level keep alive time
and then call the post request via execute method
HttpPost post = new HttpPost("http://url");
HttpResponse response = httpClient.execute(post);
Since I am using a Linux system I have configured the server with following sysctl values:
sysctl -w net.ipv4.tcp_keepalive_time=240
sysctl -w net.ipv4.tcp_keepalive_intvl=240
sysctl -w net.ipv4.tcp_keepalive_probes=10
But while executing the program the keep alive is not enabled and connection fails as previous.
I have checked this with netstat -o option and as shown below keep alive is off
tcp 0 0 192.168.1.141:43770 public_ip:80 ESTABLISHED 18134/java off (0.00/0/0)
Is there any way i can set TCP keep alive from java code using httpclient . Also I can see HttpConnectionParams are deprecated. But I couldn't find any new class which can set keep alive
I have found a solution to the problem . Curious case is there is no way i can use some builder class in httpclient to pass socket keep alive . One method as i specified in the question is using HttpConnectionParams as below but this is not working and this class is now deprecated.
HttpParams params = httpClient.getParams();
HttpConnectionParams.setSoKeepalive(params, true);
So while checking apache http docs I can see that now connection parameters are passed to httpclient via RequestConfig class . Builders of this class provide solution to set connection_time_out and socket_time_out. But checking the socurce code of this I couldnt see an option to enable SocketKeepAlive which is what we want. So the only solution is directly creating a Socket using SocketBuilder class and pass that to the HttpClientBuilder.
Following is the working code
SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setSoTimeout(3600000).build(); //We need to set socket keep alive
RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(3600000).build();
CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).
setDefaultSocketConfig(socketConfig).build();
HttpPost post = new HttpPost(url.toString());
HttpResponse response = httpClient.execute(post);
While executing above i can see that keep alive is properly set in the socket based on the sysctl values i set in linux kernel
tcp 0 0 localip:48314 public_ip:443 ESTABLISHED 14863/java keepalive (234.11/0/0)
If some one has a better solution to enable Socket Keep alive from Requestconfig class or any other high level builder class i am open to suggestions.
Keeping an HTTP connection open but inactive for a long period is a bad design choice. HTTP is a request-response protocol, with the implication that requests and responses are quick.
Holding a connection open holds resources. From the perspective of the server (and network firewalls and routers) a client that opens a connection and begins a request (A POST in your case) but does not send any bytes for a long period is indistinguishable from a client that will never send any more data, because it is faulty or malicious (conducting a DOS attack). The server (and network hardware) is right to conclude that the right thing to do is to shutdown the connection and reclaim the resources used for it. You are trying to fight against correct behaviour that occurs for good reasons. Even if you manage to workaround the TCP shutdown you will find other problems, such as HTTP server timeouts and database timeouts.
You should instead be reconsidered the design of communication between the two components. That is, this looks like an XY Problem. You might consider
Having the client wait until it has a complete upload to perform before starting the POST.
Splitting the uploads into smaller, more frequent uploads.
Use a protocol other than HTTP.
The approach above with Socket worked beautifully with a reset of tcp_keepalive_intvl value below the AWS Network Load Balancer timeout. Using both, reset the NLB tcp idle timeout that allowed java hour+ connections.
Sometimes, if the configuration is overwritten, the configuration does not take effect.My initial modification of setDefaultSocketConfig in buildClient didn't take effect.Because it is overwritten by getConnectionManager()
public CloseableHttpClient buildClient() throws Exception {
HttpClientBuilder builder = HttpClientBuilder.create()
.setDefaultSocketConfig(SocketConfig.custom().setSoKeepAlive(true).build()) // did not work
.setConnectionManager(getConnectionManager())
.setRetryHandler(getRequestRetryHandler())
.setConnectionReuseStrategy(getConnectionReuseStrategy())
.setDefaultConnectionConfig(getConnectionConfig())
.setDefaultRequestConfig(getRequestConfig())
.setDefaultHeaders(getDefaultHeaders())
.setDefaultCredentialsProvider(getDefaultCredentialsProvider())
.disableContentCompression() // gzip is not needed. Use lz4 when compress=1
.setDefaultCookieStore(cookieStoreProvider.getCookieStore(properties))
.disableRedirectHandling();
String clientName = properties != null ? properties.getClientName() : null;
if (!Utils.isNullOrEmptyString(clientName)) {
builder.setUserAgent(clientName);
}
return builder.build();
And then I move the config to getConnectionManager()，and it work.
private PoolingHttpClientConnectionManager getConnectionManager()
throws CertificateException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException, IOException {
RegistryBuilder<ConnectionSocketFactory> registry = RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", PlainConnectionSocketFactory.getSocketFactory());
if (properties.getSsl()) {
HostnameVerifier verifier = "strict".equals(properties.getSslMode()) ? SSLConnectionSocketFactory.getDefaultHostnameVerifier() : NoopHostnameVerifier.INSTANCE;
registry.register("https", new SSLConnectionSocketFactory(getSSLContext(), verifier));
}
//noinspection resource
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(
registry.build(),
null,
null,
new IpVersionPriorityResolver(),
properties.getTimeToLiveMillis(),
TimeUnit.MILLISECONDS
);
connectionManager.setDefaultMaxPerRoute(properties.getDefaultMaxPerRoute());
connectionManager.setMaxTotal(properties.getMaxTotal());
connectionManager.setDefaultConnectionConfig(getConnectionConfig());
connectionManager.setDefaultSocketConfig(SocketConfig.custom().setSoKeepAlive(true).build());
return connectionManager;
}
I have an AWS server (medium) running in EU West, and there are roughly 250 devices connected but are also always reconnecting due to internet connectivity issues, but for some reason, the amount of TCP connections to the server grows until it reaches around 4300. Then no new connections are allowed to the server. I have confirmed that it is isolated to WebSocket requests and not regular HTTP requests.
WebSocket connections are kept per device in a Map with device UUID as key; it sometimes happens that a device will send a request for a new WS connection even though the server has a connection to the device. In this case, the current connection is closed, and an error is returned so that the device can retry the connection request.
Below is the code snippet from the Controller handling the connections using LegacyWebSocket. Connections are closed using out.close() as per https://www.playframework.com/documentation/2.5.x/JavaWebSockets#handling-websockets-using-callbacks
public LegacyWebSocket<String> create(String uuid) {
logger.debug("NEW WebSocket request from {}, creating new socket...", uuid);
if(webSocketMap.containsKey(uuid)){
logger.debug("WebSocket already exists for {}, closing existing connection", uuid);
webSocketMap.get(uuid).close();
logger.debug("Responding forbidden to force WS restart from device {}", uuid);
return WebSocket.reject(forbidden());
}
LegacyWebSocket<String> ws = WebSocket.whenReady((in, out) -> {
logger.debug("Adding downstream connection to webSocketMap-> {} webSocketMap.size() = {}",uuid, webSocketMap.size());
webSocketMap.put(uuid,out);
// For each event received on the socket,
in.onMessage(message->{
if(message.equals("ping")){
logger.debug("PING received from {} {}",uuid, message);
out.write("pong");
}
});
// When the socket is closed.
in.onClose(() -> {
logger.debug("onClose, removing for {}",uuid);
webSocketMap.remove(uuid);
});
});
return ws;
}
How can I ensure that Play Framework closes the TCP connection for closed WS connections?
The call that I use to check the amount of TCP connections is netstat -n -t | wc -l
Looks like a TCP keep-alive issue - i.e. that the TCP connections become stale because of connectivity issues on the client side and the server does not handle or clean up the stale connections in time before the limit is reached.
This link will help you configure the TCP keep-alive on your server to ensure that the stale connections are cleaned up in time.