Develop Reference

Using google-http-client and google-http-client-apache-v2 behind a proxy produces NonRepeatableRequestException

I'm using google-http-client and google-http-client-apache-v2 libraries to make a POST request behind a proxy.
// 1.- Setting ssl and proxy
HttpClientBuilder builder = HttpClientBuilder.create();
SSLContext sslContext = SslUtils.getTlsSslContext();
SslUtils.initSslContext(sslContext, GoogleUtils.getCertificateTrustStore(), SslUtils.getPkixTrustManagerFactory());
builder.setSSLSocketFactory(new SSLConnectionSocketFactory(sslContext));
builder.setProxy(new HttpHost(host, port));
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(new AuthScope(host, port), new UsernamePasswordCredentials(user, pass));
builder.setDefaultCredentialsProvider(credentialsProvider);
// 2.- Build request
HttpTransport httpTransport = new ApacheHttpTransport(builder.build());
HttpRequestFactory factory = httpTransport.createRequestFactory(credential);
HttpContent httpContent = new ByteArrayContent("application/json", "{}")
HttpRequest request = factory.buildRequest("POST", new GenericUrl(url), httpContent);
// 3.- Execute request
HttpResponse httpResponse = request.execute();
That request produces a NonRepeatableRequestException:
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar!/:4.5.13]
at com.google.api.client.http.apache.v2.ApacheHttpRequest.execute(ApacheHttpRequest.java:73) ~[google-http-client-apache-v2-1.39.2.jar!/:?]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.39.2.jar!/:1.39.2]
at
...
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity.
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:225) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar!/:4.5.13]
at com.google.api.client.http.apache.v2.ApacheHttpRequest.execute(ApacheHttpRequest.java:73) ~[google-http-client-apache-v2-1.39.2.jar!/:?]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.39.2.jar!/:1.39.2]
It seems like ApacheHttpRequest wraps ByteArrayContent that is repeatable (see JavaDoc) inside a ContentEntity that is non-repeatable.
Debuging execution inside google libraries, proxy is returning "407 Proxy Authentication Required", then it tries to repeat the request (guess that including the credentials) and that exception arises because ContentEntity used by google library is non-repeatable.
Is there any way to avoid handshake with proxy including credentials in first request to avoid reuse of the entity?
Is there any way to tell google libraries that uses a repeatable entity?
Tryed with follwing library versions:
google-api-client-1.31.5
google-http-client-jackson2-1.39.2
google-oauth-client-1.31.5
google-http-client-apache-v2-1.39.2
google-http-client-1.39.2
httpclient-4.5.13
httpcore-4.4.14

It is correct that the library returns with the error saying "your request is not retryable." It is working as intended.
POST requests are fundamentally considered non-retryable, as they are most likely to have a server store data. For example, a server is recommended to return 201 (Created) as a response when the server successfully created one or more resources. Retrying a POST request may end up inserting, uploading, or posting data multiple times. This is why sometimes web browsers show the following prompt to avoid "a duplicate credit card transaction":
A potential retry logic for POST should be implemented at the user application level, not at the library level.
In your case, the cause of the error is that you are not authorized to use the proxy. Therefore, you need to authenticate with the proxy first before attempting to use it, and then send (or re-send) a POST request.
UPDATES for the questions asked later in the comment as well as in the GitHub issue.
Why is the library who tries to repeat the request? (failling on a POST request).
The question reads weird, so I'm not sure what you're asking. Anyways, the library is designed to intentionally not repeat a request for POST. For GET, it's a different story.
Why the library have the same behaviour (retrying the request) with a GET request? (but in this case sucessfully because GET request do not have entity and do not matters if it is repeatable or not).
GET is by its nature considered a repeatable request. See this doc for example to understand the nature of the difference of GET and POST.
GET requests are only used to request data (not modify)
POST is used to send data to a server to create/update a resource.
.
GET
POST
BACK button/Reload
Harmless
Data will be re-submitted (the browser should alert the user that the data are about to be re-submitted)
Why if I change the entity, as show in workaround, to make it repeatable, the POST request works successfully through the proxy for which you say I'm not authorized to use?
You programmed your app to repeat the request when it fails at the application level through the use of the Apache API. Nothing prevents you from whatever you want with the Apache library. And of course, if we change the Google library to do what you are trying to do, it is technically possible to make it work that way. However, what I am saying is that it is wrong for the library to do so. And lastly, auth is not really relevant; it's just one kind of many failures you may encounter. For POST, in almost all cases, it doesn't make sense to automatically re-send the request regardless of which kind of error you encounter.
If as you say I'm not authorized to use the proxy:
You are not authorized to use the server for the initial request. That's why you get 407 Proxy Authentication Required from the proxy server. A client most likely needs to check the returned Proxy-Authenticate value and take an appropriate action to figure out the credentials. What action it needs to take depends on the value of the header, as explained in the doc:
This status is sent with a Proxy-Authenticate header that contains information on how to authorize correctly.
The form of the credentials you provide may not be the final form the proxy may expect. Often, your initial credentials are used to obtain the final form of the credentials that the server wants. Later once you have obtained them, the client will have to provide these credentials in subsequent requests. In any case, the truth is that, the server did return 407, saying "I'm denying your request, because authentication is required."
UPDATE2
Apache HttpClient is retrying the request
Yes, of course. And you manually programmed your app to allow Apache HttpClient to re-send a request for POST (which may be a viable workaround for you but this shouldn't be generalized for other cases).
Now I see what you are missing and where you have a wrong idea. When interacting with a proxy (or a non-proxy) that requires auth, generally you (whether it is you or the Apache library) will have to make at least two requests. First, you try without sending any sensitive information (why would you disclose your information upfront to someone who cannot be trusted? Even if you trust them, you don't really know if they are going to need your info at all. Moreover, even so, you don't know how correctly you should present your sensitive info). That first request may (or may not) fail with an error like "407 Proxy Authentication Required" (people call this that the server is "challenging" you), and based on what kinds of challenges the server gives you, you will need to take the right action to prepare an auth header for the second request. And the Apache library does that for you.
despite I provide the credentials
What did you expect that calling .setDefaultCredentialsProvider() would do? It doesn't do what you are currently thinking. The Apache library does nothing about your password in the first request. As I said earlier, in the end, you need to provide the right form of credentials that the server wants after checking the value of Proxy-Authenticate, which tells you how you should correctly auth with the server. That is why generally you have to repeat a request. If all these sound alien to you, please take a moment to read this introductory doc to understand how this challenge-based HTTP auth framework works. (The doc makes a note that it will explain only with the "Basic" scheme for educational purposes, but note that there are other non-basic schemes.)

Workaround I posted on github in case it helps someone:
As workaround what I'm trying is to wrap ApacheHttpTransport in CustomApacheHttpTransport, which delegate the result of methods to ApacheHttpTransport except for buildRequest method.
This buildRequest method in CustomApacheHttpTransport builds a custom request of type CustomApacheHttpRequest.
public final class CustomApacheHttpTransport extends HttpTransport {
private ApacheHttpTransport apacheHttpTransport;
public CustomApacheHttpTransport (HttpClient httpClient) {
this.apacheHttpTransport = new ApacheHttpTransport(httpClient);
}
#Override
protected LowLevelHttpRequest buildRequest (String method, String url) {
HttpRequestBase requestBase;
if (method.equals("DELETE")) {
requestBase = new HttpDelete(url);
} else if (method.equals("GET")) {
requestBase = new HttpGet(url);
} else if (method.equals("HEAD")) {
requestBase = new HttpHead(url);
} else if (method.equals("PATCH")) {
requestBase = new HttpPatch(url);
} else if (method.equals("POST")) {
..
}
return new CustomApacheHttpRequest(apacheHttpTransport.getHttpClient(), requestBase);
}
}
This custom request is like ApacheHttpRequest except for when it is executed it creates a custom entity, CustomContentEntity, which will be repeatable depending on whether the request content supports retries.
final class CustomApacheHttpRequest extends LowLevelHttpRequest {
private final HttpClient httpClient;
private final HttpRequestBase request;
private RequestConfig.Builder requestConfig;
CustomApacheHttpRequest (HttpClient httpClient, HttpRequestBase request) {
this.httpClient = httpClient;
this.request = request;
this.requestConfig = RequestConfig.custom().setRedirectsEnabled(false).setNormalizeUri(false).setStaleConnectionCheckEnabled(false);
}
...
#Override
public LowLevelHttpResponse execute () throws IOException {
if (this.getStreamingContent() != null) {
Preconditions.checkState(request instanceof HttpEntityEnclosingRequest, "Apache HTTP client does not support %s requests with content.", request.getRequestLine().getMethod());
CustomContentEntity entity = new CustomContentEntity(this.getContentLength(), this.getStreamingContent());
entity.setContentEncoding(this.getContentEncoding());
entity.setContentType(this.getContentType());
if (this.getContentLength() == -1L) {
entity.setChunked(true);
}
((HttpEntityEnclosingRequest) request).setEntity(entity);
}
request.setConfig(requestConfig.build());
return new CustomApacheHttpResponse(request, httpClient.execute(request));
}
}
The key in CustomContentEntity is isRepeatable method wich do not returns always false as ContentEntity does.
final class CustomContentEntity extends AbstractHttpEntity {
private final long contentLength;
private final StreamingContent streamingContent;
CustomContentEntity (long contentLength, StreamingContent streamingContent) {
this.contentLength = contentLength;
this.streamingContent = streamingContent;
}
#Override
public boolean isRepeatable () {
return ((HttpContent) streamingContent).retrySupported();
}
...
}
Also I have to create CustomApacheHttpResponse as response for CustomApacheHttpRequest because ApacheHttpResponse is package-private (CustomApacheHttpResponse is exactly like ApacheHttpResponse).

How to cache the httpclient object in Java?

Using Apache HttpClient 4.5.x in my client webapp which connects to (and log in to) another (say main) server webapp.
The relationship between these 2 webapps is many to many - meaning for some user's request in client webapp, it has to login as another user + make rest call, in the server webapp. So some separation of cookiestores is needed and there's no way (is there?) to get/set the cookie store after creating a httpclient instance, so each request thread received in client webapp does something like this (and need to optimize):
HttpClient client = HttpClientBuilder.create().setDefaultCookieStore(new BasicCookieStore()).build();
//Now POST to login end point and get back JSESSIONID cookie and then make one REST call, and then the client object goes out of scope when the request ends.
I was hoping to ask on the best practice of caching the httpclient instance object as its heavy and is supposed to be reused for at least multiple requests, if not for the whole client webapp as a static singleton.
Specifically, I was hoping for advice on which of these (if any) approaches would constitute a best practice:
Use a static ConcurrentHashMap to cache the httpclient and its associated basiccookiestore for each "user" in client webapp, and to login only when the contained cached cookie is near to its expiry time. Not sure about memory usage, and un/rarely-used httpclient stay in memory without eviction.
Cache only the Cookie (somehow), but recreate a new httpclient object whenever the need arises to use that cookie for a rest call. This saves the prior call to login until the cookie expires, but no reuse of htptclient.
PooledConnectionManager - but can't find examples easily, though might require devising an eviction strategy, max number of threads etc. (so can be complex).
Is there any better way of doing this ? Thanks.
References:
http://hc.apache.org/httpclient-3.x/performance.html
Generally it is recommended to have a single instance of HttpClient
per communication component or even per application
Similar at java httpclient 4.x performance guide to resolve issue

Using concurrent hash map would be the simplest way to achieve what you want to do.
Additionally, if you are using Spring, you might want to create a bean for holding the HTTP client.

Why would you want to do all this? One can assign a different CookieStore on a per request basis by using a local HttpContext.
If needed one can maintain a map of CookieStore instances per unique user.
CloseableHttpClient httpclient = HttpClients.createDefault();
CookieStore cookieStore = new BasicCookieStore();
// Create local HTTP context
HttpClientContext localContext = HttpClientContext.create();
// Bind custom cookie store to the local context
localContext.setCookieStore(cookieStore);
HttpGet httpget = new HttpGet("http://httpbin.org/cookies");
System.out.println("Executing request " + httpget.getRequestLine());
// Pass local context as a parameter
CloseableHttpResponse response = httpclient.execute(httpget, localContext);
try {
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
List<Cookie> cookies = cookieStore.getCookies();
for (int i = 0; i < cookies.size(); i++) {
System.out.println("Local cookie: " + cookies.get(i));
}
EntityUtils.consume(response.getEntity());
} finally {
response.close();
}

How do OkHttp caches work?

I recently made a change from my existing implementation of creating a new client for every request to one standard singleton client.
Now as i began reading into the docs, i found out that there's also a thing called cache which is being used. There was some content which said that having multiple caches trying to access the same cache directory may end up causing them stomping over each other and causing crashes.
The docs I'm looking at are in their github repo:
okhttp recipes for regular issues faced
My questions:
I'm not specifically setting any cache or cache control for my client. Just a few timeout values and a connection pool. Will there be any caching done by default if i use a new request and response object for each call i make?
client = new Okhttpclient.Builder()
. connectTimeout (10000,TimeUnit.MILLISECONDS)
.readTimeout(15000,TimeUnit.MILLISECONDS)
. connectionPool (new ConnectionPool ())
.build();
The above client is set as singleton and returned to all calling servlets. The request is created as
Request req = new Request.Builder()
.url(someurl)
.get()
.build();
Response res = client.newCall(req).execute();
If so, will there be issues as the mentioned stomping part. I don't need caching at all as mostly I'm just writing stuff to another server and when I'm reading i do need it to be the current values and not caches one... So do i need to explicitly set the cache-control set to force-network or will my default no specified setting be the same?
EDIT: this is the excerpt from the Response caching part of the docs
To cache responses, you'll need a cache directory that you can read
and write to, and a limit on the cache's size. The cache directory
should be private, and untrusted applications should not be able to
read its contents!
It is an error to have multiple caches accessing the same cache
directory simultaneously. Most applications should call new
OkHttpClient() exactly once, configure it with their cache, and use
that same instance everywhere. Otherwise the two cache instances will
stomp on each other, corrupt the response cache, and possibly crash
your program.

OkHttp cache directory is set for each client instance. What the doc is telling you is that you shouldn't configure multiple clients to use the same cache directory. Caching has to be enabled explicitly, and isn't enabled in the code snippet in your question.
Having configured caching on the client instance, you can control response caching for each request. See Cache.

Programmatically keep HTTP Session Alive without browser

For one of our requirements I am talking between two servers using HTTP protocol. The interaction is long running, where a user might not interact with the other site for pretty long intervals.
When they come to the page, the log in into the remote site. Every time user tried to interact with the remote site, internally I make a HTTP call (authetication is done based on sessionId).
I was wondering if there is a way to also refresh the session and ensure that it does not expire.
As per my limited understanding, browser handles this by passing keep-alive in header or cookie (which I don't understand completely). Can anyone suggest a programmatic way in Java to achieve keep-alive behavior

1.
<session-config>
<session-timeout>-1</session-timeout>
</session-config>
Simply paste this piece if code in your deployment descriptor (DD).
If you want to keep your session alive for a particular duration of time replace -1 with any positive numeric value.
Time specified here is in minutes.
2.
If you want to change session timeout value for a particular session instance without affecting the timeout length of any other session in the application :
session.setMaxInactiveInterval(30*60);
**********************
Note :
1.In DD, the time specified is in minutes.
2.If you do it programatically, the time specified is in seconds.
Hope this helps :)

I guess below code can help you, if you can pass JSESSIONID cookie then your container will take responsibility to keep the session alive if its same session that might be created from some other browser.
find the link below that explained a lot.
Click Here
Code snippet from the link above
BasicCookieStore cookieStore = new BasicCookieStore();
BasicClientCookie cookie = new BasicClientCookie("JSESSIONID", "97E3D1012B3802114FA0A844EDE7B2ED");
cookie.setDomain("localhost");
cookie.setPath("/CookieTest");
cookieStore.addCookie(cookie);
HttpClient client = HttpClientBuilder.create().setDefaultCookieStore(cookieStore).build();
final HttpGet request = new HttpGet("http://localhost:1234/CookieTest/CookieTester");
HttpResponse response = client.execute(request);

Take a look of Apache HttpClient and see its tutorial. HttpClient supports keep alive headers and other features that should enable you to programmatically make an HTTP call.

How to use HttpClient with multithreaded operation?

I've to do an application that performs a Login POST request in a certain host, then navigates some pages, finds and retrieves some data.
Becase the website resouce is protected by session, so I have to login the website first before I can do some operation such as get or post some data.
My question is because HttpClient is not thread-safe, how can I create only one HttpClient instance but threads can perform on it safely?
Remember that the underlying connection must login first before it can be used to operate.

Here is an answer: http://pro-programmers.blogspot.com/2009/06/apache-httpclient-multi-threads.html

You can make HttpClient thread safe by specifying a thread safe client manager.
API : http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/tsccm/ThreadSafeClientConnManager.html
http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/DefaultHttpClient.html#DefaultHttpClient%28org.apache.http.conn.ClientConnectionManager%29
Example : http://thinkandroid.wordpress.com/2009/12/31/creating-an-http-client-example/