I am getting 504 Gateway timeout error from my GET method call to another service.
Recently I added a fix by increasing the timeout period but that didn't help.
This is what I have tried
public void getUserInformation(final Integer userId) {
HttpClient httpClient = getBasicAuthDefaultHttpClient();
HttpGet httpGet = new HttpGet("http://xxxx/users/"+userId);
httpGet.addHeader("userid", userid);
httpGet.addHeader("secret", secret);
try {
HttpResponse response = httpClient.execute(httpGet);
HttpEntity entity = response.getEntity();
if (entity != null && HttpStatus.OK.value() ==
response.getStatusLine().getStatusCode()) {
ObjectMapper objectMapper = new ObjectMapper();
userInfo = objectMapper.readValue(entity.getContent(),
UserInfo.class);
} else {
logger.error("Call to the service failed: response code:
{}", response.getStatusLine().getStatusCode());
}
} catch (Exception e) {
logger.error("Exception: "+ e);
}
}
public HttpClient getBasicAuthDefaultHttpClient() {
CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials creds = new
UsernamePasswordCredentials(user, password);
provider.setCredentials(AuthScope.ANY, creds);
//Fix to avoid HTTP 504 ERROR (GATEWAY TIME OUT ERROR) for ECM calls
RequestConfig.Builder requestBuilder = RequestConfig.custom();
requestBuilder.setConnectTimeout(30 * 1000);
requestBuilder.setConnectionRequestTimeout(30 * 1000);
HttpClientBuilder builder = HttpClientBuilder.create();
builder.setDefaultRequestConfig(requestBuilder.build());
builder.setDefaultCredentialsProvider(provider).build();
return builder.build();
}
I am calling this process within a loop to process records, this works for most of the records but fails for few userId's in that.
But what I noticed is everything will work fine when I run only the failed records, not sure whats the problem in this case.
I thought of calling the method again when I receive 504 to invoke it again hoping to receive 200 next time.
Not sure is this the good idea. Any advice would be greatly appreciated.
According to the description of the 504 Gateway Timeout status code, it is returned when you have a chain of servers that communicate to process the request and one of the nodes (not the server you are calling but some later one) is not able to process the request in a timely fashion.
I would presume that the situation you are in could be depicted as follows.
CLIENT -> USERS SERVICE -> SOME OTHER SERVICE
The problem is that SOME OTHER SERVICE is taking too long to process your request. The USERS SERVICE gives up at some point in time and returns you this specific status code to indicate that.
As far as I know, there is little you could do to mitigate the problem. You need to get in touch with the owners of the USERS SERVICE and ask them to increase their timeout or the owners of SOME OTHER SERVICE and ask them to improve their performance.
As for why such an error could occur from time to time. It is possible that you, in combination with other clients, are transitively overloading SOME OTHER SERVICE, causing it to process requests slower and slower. Or it could be that SOME OTHER SERVICE has throttling or rate limiting enabled to prevent Denial of Service attacks. By making too many requests to the USERS SERVICE it is possible that you are consuming the quota it has.
Of course, all of these are speculations, without knowing you actual scenario.
I faced the same sometime back, below are the checks i did to resolve this. I will add more details to the above analogy.
Client-> Users Service -> Some Other Service
Client checks:
httpclient - I see you are using RequestConfig.Builder to set the client timeout.(that will do in your case)
jerseyclient - If someone uses jersey client it can be configured as How to set the connection and read timeout with Jersey 2.x?
'Some Other Service' checks:
If throttling/rate limiting is set to avoid DOS attacks. Then you need to increase the timeouts on Some Other Service. I used tomcat server on AWS: Changed the idle timeout in your yaml file
metadata:
annotations:
#below for openshift which worked for me
haproxy.router.openshift.io/timeout:20000
#below for kubernetes timeout in ELB
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout:20000
Also changed the connector timeout on tomcat
<Connector connectionTimeout="20000" port="8080" protocol="HTTP/1.1" redirectPort="8443"/>
Voila! It worked for me.
Related
I'm using google-http-client and google-http-client-apache-v2 libraries to make a POST request behind a proxy.
// 1.- Setting ssl and proxy
HttpClientBuilder builder = HttpClientBuilder.create();
SSLContext sslContext = SslUtils.getTlsSslContext();
SslUtils.initSslContext(sslContext, GoogleUtils.getCertificateTrustStore(), SslUtils.getPkixTrustManagerFactory());
builder.setSSLSocketFactory(new SSLConnectionSocketFactory(sslContext));
builder.setProxy(new HttpHost(host, port));
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(new AuthScope(host, port), new UsernamePasswordCredentials(user, pass));
builder.setDefaultCredentialsProvider(credentialsProvider);
// 2.- Build request
HttpTransport httpTransport = new ApacheHttpTransport(builder.build());
HttpRequestFactory factory = httpTransport.createRequestFactory(credential);
HttpContent httpContent = new ByteArrayContent("application/json", "{}")
HttpRequest request = factory.buildRequest("POST", new GenericUrl(url), httpContent);
// 3.- Execute request
HttpResponse httpResponse = request.execute();
That request produces a NonRepeatableRequestException:
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar!/:4.5.13]
at com.google.api.client.http.apache.v2.ApacheHttpRequest.execute(ApacheHttpRequest.java:73) ~[google-http-client-apache-v2-1.39.2.jar!/:?]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.39.2.jar!/:1.39.2]
at
...
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity.
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:225) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar!/:4.5.13]
at com.google.api.client.http.apache.v2.ApacheHttpRequest.execute(ApacheHttpRequest.java:73) ~[google-http-client-apache-v2-1.39.2.jar!/:?]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.39.2.jar!/:1.39.2]
It seems like ApacheHttpRequest wraps ByteArrayContent that is repeatable (see JavaDoc) inside a ContentEntity that is non-repeatable.
Debuging execution inside google libraries, proxy is returning "407 Proxy Authentication Required", then it tries to repeat the request (guess that including the credentials) and that exception arises because ContentEntity used by google library is non-repeatable.
Is there any way to avoid handshake with proxy including credentials in first request to avoid reuse of the entity?
Is there any way to tell google libraries that uses a repeatable entity?
Tryed with follwing library versions:
google-api-client-1.31.5
google-http-client-jackson2-1.39.2
google-oauth-client-1.31.5
google-http-client-apache-v2-1.39.2
google-http-client-1.39.2
httpclient-4.5.13
httpcore-4.4.14
It is correct that the library returns with the error saying "your request is not retryable." It is working as intended.
POST requests are fundamentally considered non-retryable, as they are most likely to have a server store data. For example, a server is recommended to return 201 (Created) as a response when the server successfully created one or more resources. Retrying a POST request may end up inserting, uploading, or posting data multiple times. This is why sometimes web browsers show the following prompt to avoid "a duplicate credit card transaction":
A potential retry logic for POST should be implemented at the user application level, not at the library level.
In your case, the cause of the error is that you are not authorized to use the proxy. Therefore, you need to authenticate with the proxy first before attempting to use it, and then send (or re-send) a POST request.
UPDATES for the questions asked later in the comment as well as in the GitHub issue.
Why is the library who tries to repeat the request? (failling on a POST request).
The question reads weird, so I'm not sure what you're asking. Anyways, the library is designed to intentionally not repeat a request for POST. For GET, it's a different story.
Why the library have the same behaviour (retrying the request) with a GET request? (but in this case sucessfully because GET request do not have entity and do not matters if it is repeatable or not).
GET is by its nature considered a repeatable request. See this doc for example to understand the nature of the difference of GET and POST.
GET requests are only used to request data (not modify)
POST is used to send data to a server to create/update a resource.
.
GET
POST
BACK button/Reload
Harmless
Data will be re-submitted (the browser should alert the user that the data are about to be re-submitted)
Why if I change the entity, as show in workaround, to make it repeatable, the POST request works successfully through the proxy for which you say I'm not authorized to use?
You programmed your app to repeat the request when it fails at the application level through the use of the Apache API. Nothing prevents you from whatever you want with the Apache library. And of course, if we change the Google library to do what you are trying to do, it is technically possible to make it work that way. However, what I am saying is that it is wrong for the library to do so. And lastly, auth is not really relevant; it's just one kind of many failures you may encounter. For POST, in almost all cases, it doesn't make sense to automatically re-send the request regardless of which kind of error you encounter.
If as you say I'm not authorized to use the proxy:
You are not authorized to use the server for the initial request. That's why you get 407 Proxy Authentication Required from the proxy server. A client most likely needs to check the returned Proxy-Authenticate value and take an appropriate action to figure out the credentials. What action it needs to take depends on the value of the header, as explained in the doc:
This status is sent with a Proxy-Authenticate header that contains information on how to authorize correctly.
The form of the credentials you provide may not be the final form the proxy may expect. Often, your initial credentials are used to obtain the final form of the credentials that the server wants. Later once you have obtained them, the client will have to provide these credentials in subsequent requests. In any case, the truth is that, the server did return 407, saying "I'm denying your request, because authentication is required."
UPDATE2
Apache HttpClient is retrying the request
Yes, of course. And you manually programmed your app to allow Apache HttpClient to re-send a request for POST (which may be a viable workaround for you but this shouldn't be generalized for other cases).
Now I see what you are missing and where you have a wrong idea. When interacting with a proxy (or a non-proxy) that requires auth, generally you (whether it is you or the Apache library) will have to make at least two requests. First, you try without sending any sensitive information (why would you disclose your information upfront to someone who cannot be trusted? Even if you trust them, you don't really know if they are going to need your info at all. Moreover, even so, you don't know how correctly you should present your sensitive info). That first request may (or may not) fail with an error like "407 Proxy Authentication Required" (people call this that the server is "challenging" you), and based on what kinds of challenges the server gives you, you will need to take the right action to prepare an auth header for the second request. And the Apache library does that for you.
despite I provide the credentials
What did you expect that calling .setDefaultCredentialsProvider() would do? It doesn't do what you are currently thinking. The Apache library does nothing about your password in the first request. As I said earlier, in the end, you need to provide the right form of credentials that the server wants after checking the value of Proxy-Authenticate, which tells you how you should correctly auth with the server. That is why generally you have to repeat a request. If all these sound alien to you, please take a moment to read this introductory doc to understand how this challenge-based HTTP auth framework works. (The doc makes a note that it will explain only with the "Basic" scheme for educational purposes, but note that there are other non-basic schemes.)
Workaround I posted on github in case it helps someone:
As workaround what I'm trying is to wrap ApacheHttpTransport in CustomApacheHttpTransport, which delegate the result of methods to ApacheHttpTransport except for buildRequest method.
This buildRequest method in CustomApacheHttpTransport builds a custom request of type CustomApacheHttpRequest.
public final class CustomApacheHttpTransport extends HttpTransport {
private ApacheHttpTransport apacheHttpTransport;
public CustomApacheHttpTransport (HttpClient httpClient) {
this.apacheHttpTransport = new ApacheHttpTransport(httpClient);
}
#Override
protected LowLevelHttpRequest buildRequest (String method, String url) {
HttpRequestBase requestBase;
if (method.equals("DELETE")) {
requestBase = new HttpDelete(url);
} else if (method.equals("GET")) {
requestBase = new HttpGet(url);
} else if (method.equals("HEAD")) {
requestBase = new HttpHead(url);
} else if (method.equals("PATCH")) {
requestBase = new HttpPatch(url);
} else if (method.equals("POST")) {
..
}
return new CustomApacheHttpRequest(apacheHttpTransport.getHttpClient(), requestBase);
}
}
This custom request is like ApacheHttpRequest except for when it is executed it creates a custom entity, CustomContentEntity, which will be repeatable depending on whether the request content supports retries.
final class CustomApacheHttpRequest extends LowLevelHttpRequest {
private final HttpClient httpClient;
private final HttpRequestBase request;
private RequestConfig.Builder requestConfig;
CustomApacheHttpRequest (HttpClient httpClient, HttpRequestBase request) {
this.httpClient = httpClient;
this.request = request;
this.requestConfig = RequestConfig.custom().setRedirectsEnabled(false).setNormalizeUri(false).setStaleConnectionCheckEnabled(false);
}
...
#Override
public LowLevelHttpResponse execute () throws IOException {
if (this.getStreamingContent() != null) {
Preconditions.checkState(request instanceof HttpEntityEnclosingRequest, "Apache HTTP client does not support %s requests with content.", request.getRequestLine().getMethod());
CustomContentEntity entity = new CustomContentEntity(this.getContentLength(), this.getStreamingContent());
entity.setContentEncoding(this.getContentEncoding());
entity.setContentType(this.getContentType());
if (this.getContentLength() == -1L) {
entity.setChunked(true);
}
((HttpEntityEnclosingRequest) request).setEntity(entity);
}
request.setConfig(requestConfig.build());
return new CustomApacheHttpResponse(request, httpClient.execute(request));
}
}
The key in CustomContentEntity is isRepeatable method wich do not returns always false as ContentEntity does.
final class CustomContentEntity extends AbstractHttpEntity {
private final long contentLength;
private final StreamingContent streamingContent;
CustomContentEntity (long contentLength, StreamingContent streamingContent) {
this.contentLength = contentLength;
this.streamingContent = streamingContent;
}
#Override
public boolean isRepeatable () {
return ((HttpContent) streamingContent).retrySupported();
}
...
}
Also I have to create CustomApacheHttpResponse as response for CustomApacheHttpRequest because ApacheHttpResponse is package-private (CustomApacheHttpResponse is exactly like ApacheHttpResponse).
We want to migrate all our apache-httpclient-4.x code to java-http-client code to reduce dependencies. While migrating them, i ran into the following issue under java 11:
How to set the socket timeout in Java HTTP Client?
With apache-httpclient-4.x we can set the connection timeout and the socket timeout like this:
DefaultHttpClient httpClient = new DefaultHttpClient();
int timeout = 5; // seconds
HttpParams httpParams = httpClient.getParams();
httpParams.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, timeout * 1000);
httpParams.setParameter(CoreConnectionPNames.SO_TIMEOUT, timeout * 1000);
With java-http-client i can only set the connection timeout like this:
HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.build()
But i found no way to set the socket timeout. Is there any way or an open issue to support that in the future?
You can specify it at the HttpRequest.Builder level via the timeout method:
HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.build();
HttpRequest httpRequest = HttpRequest.newBuilder()
.uri(URI.create("..."))
.timeout(Duration.ofSeconds(5)) //this
.build();
httpClient.send(httpRequest, HttpResponse.BodyHandlers.ofString());
If you've got connected successfully but not able to receive a response at the desired amount of time, java.net.http.HttpTimeoutException: request timed out will be thrown (in contrast with java.net.http.HttpConnectTimeoutException: HTTP connect timed out which will be thrown if you don't get a successful connection).
There doesn't seem to be a way to specify a timeout on the flow of packets (socket timeout) on the Java Http Client.
I found an enhancement request on OpenJDK which seems to cover this possibility - https://bugs.openjdk.org/browse/JDK-8258397
Content from the link
The HttpClient lets you set a connection timeout (HttpClient.Builder) and a request timeout (HttpRequest.Builder). However the request timeout will be cancelled as soon as the response headers have been read. There is currently no timeout covering the reception of the body.
A possibility for the caller is to make use of the CompletableFuture API (get/join will accept a timeout, or CF::orTimeout can be called).
IIRC - in that case, it will still be the responsibility of the caller to cancel the request. We might want to reexamine and possibility change that.
The disadvantage here is that some of our BodyHandlers (ofPublisher, ofInputStream) will return immediately - so the CF API won't help in this case.
This might be a good thing (or not).
Another possibility could be to add a body timeout on HttpRequest.Builder. This would then cover all cases - but do we really want to timeout in the case of ofInputStream or ofPublisher if the caller doesn't read the body fast enough?
Architecture is midTier Liberty server that receives http requests and brokers to various back ends, some REST, some just JSON. When I configure for SSL (only thru envVars which is quite cool) ... it appears I get a full handShake w/every request. Additionally, the server side uses a different thread with each request (may be related). This is Liberty so it is multiThreaded. Servlet has static ref to POJO that does all apache httpClient work. Not using HttpClientContext (in this case). Basic flow is at end (struggling w/formatting for post legality)
EnvVars are:
-Djavax.net.ssl.keyStore=/root/lWasServers/certs/zosConnKey.jks
-Djavax.net.ssl.trustStore=/root/lWasServers/certs/zosConnTrust.jks
-Djavax.net.ssl.keyStorePassword=fredpwd
-Dhttp.maxConnections=40
Looked at many similar problems, but again, right now this flow does not use client context. Hoping I'm missing something simple. Code being appended on first response as I continue to struggle here w/FF in RHEL.
private static PoolingHttpClientConnectionManager cm = null ;
private static CloseableHttpClient httpClient = null ;
// ....
cm = new PoolingHttpClientConnectionManager();
cm.setMaxTotal(512);
cm.setDefaultMaxPerRoute(256) ;
httpClient = HttpClients.custom().setConnectionManager(cm).build() ;
// ...
responseBody = httpClient.execute(httpGet, responseHandler);
If a persistent HTTP connection is stateful and is associated with a particular security context or identity, such as SSL key or NTLM user name, HttpClient tries to make sure this connection cannot be accidentally re-used within a different security context or by a different user. Usually the most straight-forward way of letting HttpClient know that requests are logically related and belong to the same session is by executing those requests with the same HttpContext instance. See HttpClient tutorial for details. One can also disable connection state tracking if HttpClient can only be accessed by a single user or within the same security context. Use with caution.
OK, while I'm not exactly an expert at reading the ssl trace, I do believe I have resolved it. I am on a thread but that is controlled by the server. I now pass the HttpSession in and keep a reference to the HttpClientConnection that I now create for each session. I pool these HttpClientConnection objects (rudimentary pooling, basically just get/release). So all calls w/in an http session use the same HttpClientContext. Now it appears that I am NOT handShaking all the time. There may have been a better way to do it, but this does indeed work, I have a few gremlins to look into (socket timeouts in < 1 millisecond?) ... but I'm confident that I'm non longer handShaking with each request (only each time I end up creating a new context) ... so this is all good. Thanks,
I have an app that makes http requests to a remote server. I do this with the following code:
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("myURL");
try {
ArrayList<BasicNameValuePair> postVariables = new ArrayList<BasicNameValuePair>(2);
postVariables.add(new BasicNameValuePair("key","value"));
httpPost.setEntity(new UrlEncodedFormEntity(postVariables));
HttpResponse response = httpClient.execute(httpPost);
String responseString = EntityUtils.toString(response.getEntity());
if (responseString.contains("\"success\":true")){
//this means the request succeeded
} else {
//failed
}
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
This goes really well, but one of our customers has set up an APN that requires requests to go via a certain proxy server. If I add the following to the request this works, the request gets rerouted via the proxy to the server:
HttpHost httpHost = new HttpHost("proxyURL",8080);
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, httpHost);
So far so good, however, I use a library that makes some http requests as well. The library's code is not accesible for me, so I can't add those two lines to the code. I contacted the creators of that library, and they told me it should be possible to set up the android environment so that all requests will automatically go through the proxy. Is there something like that? I didn't find anything on google.
I'm basically looking for a way to set the above two lines as a standard for all http requests. Please note that the APN does not set the proxy as a default for the entire phone, so apps will have to do this manually (and yes that means the majority of the apps don't work on that customer's phone).
It's been a year or two since I've needed to use it, but if I remember correctly, you can use the System.setProperty(String, String) in order to set an environment-wide setting for your application to route all HTTP traffic through a proxy. The properties that you should need to set are "http.proxyHost" and "http.proxyPort" and then use your HttpClient normally without specifying a proxy because the VM will handle routing requests.
Docs for more information about what I'm talking about can be found here: ProxySelector (just so you know what keys to use) and here for documentation about the actual System.setProperty(String, String) function
If that doesn't work for you, let me know and I'll try to dig out my old code that set a system-level proxy. BTW, it's really only "system-level" since each app runs in it's own Dalvik so you won't impact other app's network communications.
I'm working on an application that uses HttpResponse for network calls.
Occasionally (about 30% of the time) network calls seem to fail with a 502 returning from the server. When checking the server logs, It appears that the server never received the request.
The logic doesn't seem bad, however, and it does work most of the time. I am communicating to the server via JSON.
HttpClient client = new DefaultHttpClient();
HttpPost method = new HttpPost( BASE_URL + endpoint )
...add json entity
method.addHeader( "content-type", "application/json" );
method.addHeader( "accept", "application/json" );
....call below function w client/method
protected HttpResponse invoke( HttpClient client, HttpUriRequest method ) throws ServiceException {
HttpResponse response = null;
try {
response = client.execute( method );
} catch ( //catch statements here ....)
int status = response.getStatusLine().getStatusCode();
//status = 502
}
The interesting thing about this call failing is that it's the exact same call across devices, and the server it's hitting is 20 feet away, although it does happen on a different server located in another state.
I can't see why it works sometimes but not others.
Run wire shark on both sender and receiver . Observe sends and receives. You should be able to see the cause of the problem.
If you are getting a 502 error it suggests an intermediate device ( proxy,web filter,etc ) is blocking or " eating " your HTTP post .
Hate to say it, but this just sounds like an infrastructure issue. Some networking problem between your device, and the server. The code itself looks fine