I m using Jetty's http2client to make synchronous calls to my server, my sample program is a follows,
Client part
Security.addProvider(new OpenSSLProvider());
SslContextFactory sslContextFactory = new SslContextFactory(true);
sslContextFactory.setProvider("Conscrypt");
sslContextFactory.setProtocol("TLSv1.3");
HTTP2Client http2Client = new HTTP2Client();
http2Client.setConnectTimeout(5000);
http2Client.setIdleTimeout(5000);
HttpClient httpClient = new org.eclipse.jetty.client.HttpClient(new HttpClientTransportOverHTTP2(http2Client), sslContextFactory);
httpClient.setMaxConnectionsPerDestination(20);
httpClient.setMaxRequestsQueuedPerDestination(100);
httpClient.setConnectTimeout(5000);
httpClient.addBean(sslContextFactory);
httpClient.start();
Request Part
Request request = httpClient.POST("my url goes here");
request.header(HttpHeader.CONTENT_TYPE, "application/json");
request.content(new StringContentProvider("xmlRequest PayLoad goes here","utf-8"));
ContentResponse response = request.send();
String res = new String(response.getContent());
I need to instrument to get metrics like number of connections per destination, number of requests per connections, number of failed transactions, etc.
My application runs in a server where using wireshark or any other tcp tool is restricted. so I need to get this data within java. Enabling debug logs of jetty is not viable as it writes GBs of data.
Is there a way to get these metrics either by some util or by java reflection?
Thanks in Advance
http2Client.setMaxConcurrentPushedStreams(1000);
This is way too big, it's unlikely the server will push 1000 concurrent streams.
http2Client.setConnectTimeout(30);
http2Client.setIdleTimeout(5);
The timeouts are measured in milliseconds, so these values are way too small.
I also recommend the idle timeout to be a larger value than 5000 milliseconds, something like 20000-30000 is typically better.
String res = new String(response.getContent());
This is wrong, as you don't take into account the response charset.
Use instead response.getContentAsString().
As for the metrics, you can use JMX and extract a number of metrics using a JMX console (or via standard JMX APIs).
To setup JMX for HttpClient you can do this:
MBeanServer mbeanServer = ManagementFactory.getPlatformMBeanServer();
MBeanContainer mbeanContainer = new MBeanContainer(mbeanServer);
httpClient.addBean(mbeanContainer);
The code above will export the HttpClient components to JMX and there you can query the various components for the metrics you are interested in.
Related
We want to migrate all our apache-httpclient-4.x code to java-http-client code to reduce dependencies. While migrating them, i ran into the following issue under java 11:
How to set the socket timeout in Java HTTP Client?
With apache-httpclient-4.x we can set the connection timeout and the socket timeout like this:
DefaultHttpClient httpClient = new DefaultHttpClient();
int timeout = 5; // seconds
HttpParams httpParams = httpClient.getParams();
httpParams.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, timeout * 1000);
httpParams.setParameter(CoreConnectionPNames.SO_TIMEOUT, timeout * 1000);
With java-http-client i can only set the connection timeout like this:
HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.build()
But i found no way to set the socket timeout. Is there any way or an open issue to support that in the future?
You can specify it at the HttpRequest.Builder level via the timeout method:
HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.build();
HttpRequest httpRequest = HttpRequest.newBuilder()
.uri(URI.create("..."))
.timeout(Duration.ofSeconds(5)) //this
.build();
httpClient.send(httpRequest, HttpResponse.BodyHandlers.ofString());
If you've got connected successfully but not able to receive a response at the desired amount of time, java.net.http.HttpTimeoutException: request timed out will be thrown (in contrast with java.net.http.HttpConnectTimeoutException: HTTP connect timed out which will be thrown if you don't get a successful connection).
There doesn't seem to be a way to specify a timeout on the flow of packets (socket timeout) on the Java Http Client.
I found an enhancement request on OpenJDK which seems to cover this possibility - https://bugs.openjdk.org/browse/JDK-8258397
Content from the link
The HttpClient lets you set a connection timeout (HttpClient.Builder) and a request timeout (HttpRequest.Builder). However the request timeout will be cancelled as soon as the response headers have been read. There is currently no timeout covering the reception of the body.
A possibility for the caller is to make use of the CompletableFuture API (get/join will accept a timeout, or CF::orTimeout can be called).
IIRC - in that case, it will still be the responsibility of the caller to cancel the request. We might want to reexamine and possibility change that.
The disadvantage here is that some of our BodyHandlers (ofPublisher, ofInputStream) will return immediately - so the CF API won't help in this case.
This might be a good thing (or not).
Another possibility could be to add a body timeout on HttpRequest.Builder. This would then cover all cases - but do we really want to timeout in the case of ofInputStream or ofPublisher if the caller doesn't read the body fast enough?
I am getting 504 Gateway timeout error from my GET method call to another service.
Recently I added a fix by increasing the timeout period but that didn't help.
This is what I have tried
public void getUserInformation(final Integer userId) {
HttpClient httpClient = getBasicAuthDefaultHttpClient();
HttpGet httpGet = new HttpGet("http://xxxx/users/"+userId);
httpGet.addHeader("userid", userid);
httpGet.addHeader("secret", secret);
try {
HttpResponse response = httpClient.execute(httpGet);
HttpEntity entity = response.getEntity();
if (entity != null && HttpStatus.OK.value() ==
response.getStatusLine().getStatusCode()) {
ObjectMapper objectMapper = new ObjectMapper();
userInfo = objectMapper.readValue(entity.getContent(),
UserInfo.class);
} else {
logger.error("Call to the service failed: response code:
{}", response.getStatusLine().getStatusCode());
}
} catch (Exception e) {
logger.error("Exception: "+ e);
}
}
public HttpClient getBasicAuthDefaultHttpClient() {
CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials creds = new
UsernamePasswordCredentials(user, password);
provider.setCredentials(AuthScope.ANY, creds);
//Fix to avoid HTTP 504 ERROR (GATEWAY TIME OUT ERROR) for ECM calls
RequestConfig.Builder requestBuilder = RequestConfig.custom();
requestBuilder.setConnectTimeout(30 * 1000);
requestBuilder.setConnectionRequestTimeout(30 * 1000);
HttpClientBuilder builder = HttpClientBuilder.create();
builder.setDefaultRequestConfig(requestBuilder.build());
builder.setDefaultCredentialsProvider(provider).build();
return builder.build();
}
I am calling this process within a loop to process records, this works for most of the records but fails for few userId's in that.
But what I noticed is everything will work fine when I run only the failed records, not sure whats the problem in this case.
I thought of calling the method again when I receive 504 to invoke it again hoping to receive 200 next time.
Not sure is this the good idea. Any advice would be greatly appreciated.
According to the description of the 504 Gateway Timeout status code, it is returned when you have a chain of servers that communicate to process the request and one of the nodes (not the server you are calling but some later one) is not able to process the request in a timely fashion.
I would presume that the situation you are in could be depicted as follows.
CLIENT -> USERS SERVICE -> SOME OTHER SERVICE
The problem is that SOME OTHER SERVICE is taking too long to process your request. The USERS SERVICE gives up at some point in time and returns you this specific status code to indicate that.
As far as I know, there is little you could do to mitigate the problem. You need to get in touch with the owners of the USERS SERVICE and ask them to increase their timeout or the owners of SOME OTHER SERVICE and ask them to improve their performance.
As for why such an error could occur from time to time. It is possible that you, in combination with other clients, are transitively overloading SOME OTHER SERVICE, causing it to process requests slower and slower. Or it could be that SOME OTHER SERVICE has throttling or rate limiting enabled to prevent Denial of Service attacks. By making too many requests to the USERS SERVICE it is possible that you are consuming the quota it has.
Of course, all of these are speculations, without knowing you actual scenario.
I faced the same sometime back, below are the checks i did to resolve this. I will add more details to the above analogy.
Client-> Users Service -> Some Other Service
Client checks:
httpclient - I see you are using RequestConfig.Builder to set the client timeout.(that will do in your case)
jerseyclient - If someone uses jersey client it can be configured as How to set the connection and read timeout with Jersey 2.x?
'Some Other Service' checks:
If throttling/rate limiting is set to avoid DOS attacks. Then you need to increase the timeouts on Some Other Service. I used tomcat server on AWS: Changed the idle timeout in your yaml file
metadata:
annotations:
#below for openshift which worked for me
haproxy.router.openshift.io/timeout:20000
#below for kubernetes timeout in ELB
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout:20000
Also changed the connector timeout on tomcat
<Connector connectionTimeout="20000" port="8080" protocol="HTTP/1.1" redirectPort="8443"/>
Voila! It worked for me.
How do I enable secure connections with CometD?
I have an app that is working when I use an "http" protocol for the BayeuxServer. If I switch to "https", I get failed handshakes.
What is the correct way to use a secure connection in CometD?
This is via the Java Client.
Here is the error:
{failure={exception=java.lang.NullPointerException, message={ext={ack=true}, supportedConnectionTypes=[long-polling], channel=/meta/handshake, id=4, version=1.0}, connectionType=long-polling}, channel=/meta/handshake, id=4, subscription=null, successful=false}
I do not see any exceptions on the server (ie, the null pointer is not in our code), and if I use HTTP, it works fine.
I've pieced together the following for the Java client side:
SslContextFactory sslContextFactory = new SslContextFactory();
sslContextFactory.setTrustAll(true); // only interacting with our backend, so accept self-signed certs
WebSocketClient webSocketClient = new WebSocketClient(sslContextFactory);
webSocketClient.start();
ClientTransport wsTransport = new JettyWebSocketTransport(null, null, webSocketClient);
HttpClient httpClient = new HttpClient(sslContextFactory);
httpClient.start();
ClientTransport httpTransport = new LongPollingTransport(null, httpClient);
I believe that will do it.
I still need to figure out how to configure the server side cometd to accept the secure connections. I am using the Spring setup.
The answer to the server side is: Its a pain in the ass.
Here is how you can get it working with the jetty maven plugin:
http://juplo.de/configure-https-for-jetty-maven-plugin-9-0-x/#comment-53352
Architecture is midTier Liberty server that receives http requests and brokers to various back ends, some REST, some just JSON. When I configure for SSL (only thru envVars which is quite cool) ... it appears I get a full handShake w/every request. Additionally, the server side uses a different thread with each request (may be related). This is Liberty so it is multiThreaded. Servlet has static ref to POJO that does all apache httpClient work. Not using HttpClientContext (in this case). Basic flow is at end (struggling w/formatting for post legality)
EnvVars are:
-Djavax.net.ssl.keyStore=/root/lWasServers/certs/zosConnKey.jks
-Djavax.net.ssl.trustStore=/root/lWasServers/certs/zosConnTrust.jks
-Djavax.net.ssl.keyStorePassword=fredpwd
-Dhttp.maxConnections=40
Looked at many similar problems, but again, right now this flow does not use client context. Hoping I'm missing something simple. Code being appended on first response as I continue to struggle here w/FF in RHEL.
private static PoolingHttpClientConnectionManager cm = null ;
private static CloseableHttpClient httpClient = null ;
// ....
cm = new PoolingHttpClientConnectionManager();
cm.setMaxTotal(512);
cm.setDefaultMaxPerRoute(256) ;
httpClient = HttpClients.custom().setConnectionManager(cm).build() ;
// ...
responseBody = httpClient.execute(httpGet, responseHandler);
If a persistent HTTP connection is stateful and is associated with a particular security context or identity, such as SSL key or NTLM user name, HttpClient tries to make sure this connection cannot be accidentally re-used within a different security context or by a different user. Usually the most straight-forward way of letting HttpClient know that requests are logically related and belong to the same session is by executing those requests with the same HttpContext instance. See HttpClient tutorial for details. One can also disable connection state tracking if HttpClient can only be accessed by a single user or within the same security context. Use with caution.
OK, while I'm not exactly an expert at reading the ssl trace, I do believe I have resolved it. I am on a thread but that is controlled by the server. I now pass the HttpSession in and keep a reference to the HttpClientConnection that I now create for each session. I pool these HttpClientConnection objects (rudimentary pooling, basically just get/release). So all calls w/in an http session use the same HttpClientContext. Now it appears that I am NOT handShaking all the time. There may have been a better way to do it, but this does indeed work, I have a few gremlins to look into (socket timeouts in < 1 millisecond?) ... but I'm confident that I'm non longer handShaking with each request (only each time I end up creating a new context) ... so this is all good. Thanks,
these days i'm struggling with a quite weird issue regarding Apache HttpClient and threads.
The point is that I have a HttpClient shared by all the threads and the use it to execute an HttpPut request to upload a small file (8k aprox.). Well with a small amount of threads everything is allright and the times are good (200-600 milliseconds), but when we start increasing the number of concurrent threads the times are awful (8 seconds).
We checked the server to ensure the problem wasn't there. Using jmeter with the same load (1000 threads in a second) we got response times of milliseconds!!
The implentation uses a thread-safe connection manager:
PoolingHttpClientConnectionManager httpConnectionManager = new PoolingHttpClientConnectionManager();
httpConnectionManager.setMaxTotal(5000);
httpConnectionManager.setDefaultMaxPerRoute(5000);
HttpClient httpClient = HttpClientBuilder.create()
.setConnectionManager(httpConnectionManager)
.build();
And the threads run the following code:
HttpPut put = new HttpPut(urlStr);
put.setConfig(RequestConfig.custom()
.setExpectContinueEnabled(true)
.setStaleConnectionCheckEnabled(false)
.setRedirectsEnabled(true).build());
put.setEntity(new FileEntity(new
File("image.tif")));
put.setHeader("Content-Type", "image/tiff");
put.setHeader("Connection", "keep-alive");
HttpResponse response = httpClient.execute(put, HttpClientContext.create());
It looks like if there was a shared resource that has a huge impact when there is a high load.
Looking at the sourcecode of Apache Jmeter I don't see relevant differences respect this code.
Any idea guys?
You need to turn on debugging on the client side in order to do the following:
verify that the pool of 5000 is actually being used to great depth. The logger will display the changing totals for "available remain in pool" and the number of the current Pool entry being used.
verify that you are immediately clean-up and RETURN to pool the entry. Remember to close your resources ( Streams used to access response, Entity objects )
CloseableHttpResponse response
case PUT:
HttpPut httpPut = new HttpPut(url);
httpPut.setProtocolVersion(new ProtocolVersion("HTTP", 1,1));
httpPut.setConfig(this.config);
httpPut.setEntity(new StringEntity(data, ContentType.APPLICATION_JSON));
response = httpClient.execute(httpPut, context);
httpPut.releaseConnection();
More info