Develop Reference

Using google-http-client and google-http-client-apache-v2 behind a proxy produces NonRepeatableRequestException

I'm using google-http-client and google-http-client-apache-v2 libraries to make a POST request behind a proxy.
// 1.- Setting ssl and proxy
HttpClientBuilder builder = HttpClientBuilder.create();
SSLContext sslContext = SslUtils.getTlsSslContext();
SslUtils.initSslContext(sslContext, GoogleUtils.getCertificateTrustStore(), SslUtils.getPkixTrustManagerFactory());
builder.setSSLSocketFactory(new SSLConnectionSocketFactory(sslContext));
builder.setProxy(new HttpHost(host, port));
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(new AuthScope(host, port), new UsernamePasswordCredentials(user, pass));
builder.setDefaultCredentialsProvider(credentialsProvider);
// 2.- Build request
HttpTransport httpTransport = new ApacheHttpTransport(builder.build());
HttpRequestFactory factory = httpTransport.createRequestFactory(credential);
HttpContent httpContent = new ByteArrayContent("application/json", "{}")
HttpRequest request = factory.buildRequest("POST", new GenericUrl(url), httpContent);
// 3.- Execute request
HttpResponse httpResponse = request.execute();
That request produces a NonRepeatableRequestException:
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar!/:4.5.13]
at com.google.api.client.http.apache.v2.ApacheHttpRequest.execute(ApacheHttpRequest.java:73) ~[google-http-client-apache-v2-1.39.2.jar!/:?]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.39.2.jar!/:1.39.2]
at
...
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity.
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:225) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar!/:4.5.13]
at com.google.api.client.http.apache.v2.ApacheHttpRequest.execute(ApacheHttpRequest.java:73) ~[google-http-client-apache-v2-1.39.2.jar!/:?]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.39.2.jar!/:1.39.2]
It seems like ApacheHttpRequest wraps ByteArrayContent that is repeatable (see JavaDoc) inside a ContentEntity that is non-repeatable.
Debuging execution inside google libraries, proxy is returning "407 Proxy Authentication Required", then it tries to repeat the request (guess that including the credentials) and that exception arises because ContentEntity used by google library is non-repeatable.
Is there any way to avoid handshake with proxy including credentials in first request to avoid reuse of the entity?
Is there any way to tell google libraries that uses a repeatable entity?
Tryed with follwing library versions:
google-api-client-1.31.5
google-http-client-jackson2-1.39.2
google-oauth-client-1.31.5
google-http-client-apache-v2-1.39.2
google-http-client-1.39.2
httpclient-4.5.13
httpcore-4.4.14

It is correct that the library returns with the error saying "your request is not retryable." It is working as intended.
POST requests are fundamentally considered non-retryable, as they are most likely to have a server store data. For example, a server is recommended to return 201 (Created) as a response when the server successfully created one or more resources. Retrying a POST request may end up inserting, uploading, or posting data multiple times. This is why sometimes web browsers show the following prompt to avoid "a duplicate credit card transaction":
A potential retry logic for POST should be implemented at the user application level, not at the library level.
In your case, the cause of the error is that you are not authorized to use the proxy. Therefore, you need to authenticate with the proxy first before attempting to use it, and then send (or re-send) a POST request.
UPDATES for the questions asked later in the comment as well as in the GitHub issue.
Why is the library who tries to repeat the request? (failling on a POST request).
The question reads weird, so I'm not sure what you're asking. Anyways, the library is designed to intentionally not repeat a request for POST. For GET, it's a different story.
Why the library have the same behaviour (retrying the request) with a GET request? (but in this case sucessfully because GET request do not have entity and do not matters if it is repeatable or not).
GET is by its nature considered a repeatable request. See this doc for example to understand the nature of the difference of GET and POST.
GET requests are only used to request data (not modify)
POST is used to send data to a server to create/update a resource.
.
GET
POST
BACK button/Reload
Harmless
Data will be re-submitted (the browser should alert the user that the data are about to be re-submitted)
Why if I change the entity, as show in workaround, to make it repeatable, the POST request works successfully through the proxy for which you say I'm not authorized to use?
You programmed your app to repeat the request when it fails at the application level through the use of the Apache API. Nothing prevents you from whatever you want with the Apache library. And of course, if we change the Google library to do what you are trying to do, it is technically possible to make it work that way. However, what I am saying is that it is wrong for the library to do so. And lastly, auth is not really relevant; it's just one kind of many failures you may encounter. For POST, in almost all cases, it doesn't make sense to automatically re-send the request regardless of which kind of error you encounter.
If as you say I'm not authorized to use the proxy:
You are not authorized to use the server for the initial request. That's why you get 407 Proxy Authentication Required from the proxy server. A client most likely needs to check the returned Proxy-Authenticate value and take an appropriate action to figure out the credentials. What action it needs to take depends on the value of the header, as explained in the doc:
This status is sent with a Proxy-Authenticate header that contains information on how to authorize correctly.
The form of the credentials you provide may not be the final form the proxy may expect. Often, your initial credentials are used to obtain the final form of the credentials that the server wants. Later once you have obtained them, the client will have to provide these credentials in subsequent requests. In any case, the truth is that, the server did return 407, saying "I'm denying your request, because authentication is required."
UPDATE2
Apache HttpClient is retrying the request
Yes, of course. And you manually programmed your app to allow Apache HttpClient to re-send a request for POST (which may be a viable workaround for you but this shouldn't be generalized for other cases).
Now I see what you are missing and where you have a wrong idea. When interacting with a proxy (or a non-proxy) that requires auth, generally you (whether it is you or the Apache library) will have to make at least two requests. First, you try without sending any sensitive information (why would you disclose your information upfront to someone who cannot be trusted? Even if you trust them, you don't really know if they are going to need your info at all. Moreover, even so, you don't know how correctly you should present your sensitive info). That first request may (or may not) fail with an error like "407 Proxy Authentication Required" (people call this that the server is "challenging" you), and based on what kinds of challenges the server gives you, you will need to take the right action to prepare an auth header for the second request. And the Apache library does that for you.
despite I provide the credentials
What did you expect that calling .setDefaultCredentialsProvider() would do? It doesn't do what you are currently thinking. The Apache library does nothing about your password in the first request. As I said earlier, in the end, you need to provide the right form of credentials that the server wants after checking the value of Proxy-Authenticate, which tells you how you should correctly auth with the server. That is why generally you have to repeat a request. If all these sound alien to you, please take a moment to read this introductory doc to understand how this challenge-based HTTP auth framework works. (The doc makes a note that it will explain only with the "Basic" scheme for educational purposes, but note that there are other non-basic schemes.)

Workaround I posted on github in case it helps someone:
As workaround what I'm trying is to wrap ApacheHttpTransport in CustomApacheHttpTransport, which delegate the result of methods to ApacheHttpTransport except for buildRequest method.
This buildRequest method in CustomApacheHttpTransport builds a custom request of type CustomApacheHttpRequest.
public final class CustomApacheHttpTransport extends HttpTransport {
private ApacheHttpTransport apacheHttpTransport;
public CustomApacheHttpTransport (HttpClient httpClient) {
this.apacheHttpTransport = new ApacheHttpTransport(httpClient);
}
#Override
protected LowLevelHttpRequest buildRequest (String method, String url) {
HttpRequestBase requestBase;
if (method.equals("DELETE")) {
requestBase = new HttpDelete(url);
} else if (method.equals("GET")) {
requestBase = new HttpGet(url);
} else if (method.equals("HEAD")) {
requestBase = new HttpHead(url);
} else if (method.equals("PATCH")) {
requestBase = new HttpPatch(url);
} else if (method.equals("POST")) {
..
}
return new CustomApacheHttpRequest(apacheHttpTransport.getHttpClient(), requestBase);
}
}
This custom request is like ApacheHttpRequest except for when it is executed it creates a custom entity, CustomContentEntity, which will be repeatable depending on whether the request content supports retries.
final class CustomApacheHttpRequest extends LowLevelHttpRequest {
private final HttpClient httpClient;
private final HttpRequestBase request;
private RequestConfig.Builder requestConfig;
CustomApacheHttpRequest (HttpClient httpClient, HttpRequestBase request) {
this.httpClient = httpClient;
this.request = request;
this.requestConfig = RequestConfig.custom().setRedirectsEnabled(false).setNormalizeUri(false).setStaleConnectionCheckEnabled(false);
}
...
#Override
public LowLevelHttpResponse execute () throws IOException {
if (this.getStreamingContent() != null) {
Preconditions.checkState(request instanceof HttpEntityEnclosingRequest, "Apache HTTP client does not support %s requests with content.", request.getRequestLine().getMethod());
CustomContentEntity entity = new CustomContentEntity(this.getContentLength(), this.getStreamingContent());
entity.setContentEncoding(this.getContentEncoding());
entity.setContentType(this.getContentType());
if (this.getContentLength() == -1L) {
entity.setChunked(true);
}
((HttpEntityEnclosingRequest) request).setEntity(entity);
}
request.setConfig(requestConfig.build());
return new CustomApacheHttpResponse(request, httpClient.execute(request));
}
}
The key in CustomContentEntity is isRepeatable method wich do not returns always false as ContentEntity does.
final class CustomContentEntity extends AbstractHttpEntity {
private final long contentLength;
private final StreamingContent streamingContent;
CustomContentEntity (long contentLength, StreamingContent streamingContent) {
this.contentLength = contentLength;
this.streamingContent = streamingContent;
}
#Override
public boolean isRepeatable () {
return ((HttpContent) streamingContent).retrySupported();
}
...
}
Also I have to create CustomApacheHttpResponse as response for CustomApacheHttpRequest because ApacheHttpResponse is package-private (CustomApacheHttpResponse is exactly like ApacheHttpResponse).

Jersey/JAX-RS Client throwing 400 Bad Request

I have a RESTful Java web service that I built using Jersey. The client for it defines a resource with the following method:
#Override
public String saveWidget(Widget widget) {
return webResource.path("user").type(MediaType.APPLICATION_JSON).entity(widget).post(String.class, Widget.class);
}
Then, a driver using this client:
public class Driver {
public static void main(String[] args) {
WidgetClient client;
WidgetClientBuilder builder = new WidgetClientBuilder();
client = builder.withUri("http://localhost:8080/myapi").build();
Widget w = getSomehow();
String widgetUri = client.getWidgetResource().saveWidget(w);
System.out.println("Widget was saved URI was returned: " + widgetUri);
}
}
When I run this I get:
Exception in thread "main" com.sun.jersey.api.client.UniformInterfaceException: POST http://localhost:8080/myapi/widget returned a response status of 400 Bad Request
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:688)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570)
at com.my.myapi.WidgetResource.saveWidget(WidgetResource.java:27)
at com.my.myapi.Driver.main(Driver.java:32)
I know the service endpoint is valid because I can hit it from another (non-Java) web client without issues. This means that either my Widget instance is malformed or that there is something with my Java client method (saveWidget). I ruled out my w Widget being bad by serializing it into JSON, and then copying it into my non-Java web client and POSTing to the same endpoint (no issues arose). So this tells me I have the client method configured wrong. Any ideas?

This is regarding making a call POST call using Jersey client.
For jersey client, default client configuration uses ChunkedEncoding and gzip. This can be checked in request headers for POST call. Content length of payload (JSON String or any object mapper pojo) and request headers received by post call i.e. header name CONTENT-LENGTH, CONTENT-ENCODING. If there is difference, POST call might return 400 bad request. (Something like unable to process JSON). To solve this, you can disable ChunkedEncoding, gzip encoding. Code snippet for the same:
clientConfiguration.setChunkedEncodingEnabled(false);
clientConfiguration.setGzipEnabled(false);
Client client = (new JerseyClientBuilder(environment)).using(clientConfiguration).using(environment).build("HTTP_CLIENT");
WebTarget webTarget = client.target(endpoint);
Response response = webTarget.path(path).request(MediaType.APPLICATION_JSON).post(Entity.json(jsonString));

.post(String.class, Widget.class);
You appear to be posting a Class object, not a Widget object.

How can I access a 'Web App' built in Google Apps Script from Android?

I can successfully access Google Drive and Spreadsheet functionality from my application.
So I have an authorised instance of com.google.api.client.auth.oauth2.Credential.
Now I wish to execute a Google Apps Script that is deployed as a 'Web App'. This will also require authentication to run. This script runs in the browser if I hit the endpoint and am authenticated.
Here's some psuedo code :
String url = "https://script.google.com/a/macros/mycompany.com/s/xxxx/dev";
GenericUrl webAppEndPoint = new GenericUrl(url);
final HttpTransport httpTransport = AndroidHttp.newCompatibleTransport();
HttpRequestFactory requestFactory = httpTransport.createRequestFactory(currentCredential);
// Do POST for service
String requestBody = URLEncoder.encode("{\"name\":\"John Smith\",\"company\":\"Virginia Company\",\"pdf\":\""+getPdfBase64()+"\"}", "UTF-8");
HttpRequest postRequest =requestFactory.buildPostRequest(new GenericUrl(url), ByteArrayContent.fromString(null, requestBody));
postRequest.getHeaders().setAccept("application/json");
postRequest.setFollowRedirects(true);
postRequest.setLoggingEnabled(true);
HttpResponse postResponse = postRequest.execute();
If I run the code I get the following error : com.google.api.client.http.HttpResponseException: HttpResponseException 405 Method Not Allowed
UPDATE : So - originally i was POSTing to the wrong URL ( i'd copied the redirected URL from a browser instead of the script URL )
The POST is now successful ( authentication included ) using the above code, but it still doesn't handle the GET redirect after submission. I can work with this now but it would be good to be able to get a response from the server.
I think that com.google.api.client.http.HttpRequest doesn't handle authenticated POST redirects properly.

Your pseudocode isn’t very illuminating; to really see what’s going on you’d need to show the actual HTTP traffic. I should say though that a 302 redirect to a specified redict_uri is a normal part of the OAuth 2 authentication flow.

1) you cant call an apps script with authentication. You need to publish it as anonymous access as a contentService.
2)you are also calling the wrong url. Call the service url not the redirected one that you get in the browser.

Using HttpClient to send JSON query string in a URL to a web service (Java)

I have a web service that I built... what I am trying to do now is send a simple request that contains a json query string from a Tapestry web app to that web service. I searched around and most people say to use Apache HttpClient to achieve this. Along with HttpClient I am using URIBuilder.
The Json object that I am trying to send looks like this
{"user":{"userEmail":"jdoe#gmail.com","firstName":"John","lastName":"Doe","phone":"203- 555-5555"},"password":"dead"}
*I realize the issues with the password being sent in plain text etc...
The url that works (tested by manually entering in a web browser and this web service already services an Android client and an iOS client) looks like this
http:// ##.##.###.##/createuser?json={"user":{"userEmail":"jdoe#gmail.com","firstName":"John","lastName":"Doe","phone":"203-555-5555"},"password":"dead"}
Here is the HttpClient code that I have mashed together from google'ing around trying to figure out why this wont work. Essentially what I am trying to do is create a URI with URIBuilder and then construct an HttpPost or HttpGet object with the newly built URI. But something is going wrong in the URIBuilding process. When I debug, an exception gets thrown when I try to set all the aspects of the URI.
Object onSuccess() throws ClientProtocolException, IOException, URISyntaxException{
// json = {"user":{"userEmail":"jdoe#gmail.com","firstName":"John","lastName":"Doe","phone":"203- 555-5555"},"password":"dead"}
String json = user.toJson();
URIBuilder builder = new URIBuilder();
// Error gets thrown when I step over the next line
builder.setScheme("http").setHost("##.###.##.###").setPort(8080).setPath("createuser").setQuery("json=" +json);
URI uri = builder.build();
HttpPost request = new HttpPost(uri);
DefaultHttpClient httpClient = new DefaultHttpClient();
String tmp = request.getURI().toString();
HttpResponse response = httpClient.execute(request);
index.setResponse(EntityUtils.toString(response.getEntity()));
return index;
The error that comes back when I step over the line that I commented in the code is
[ERROR] TapestryModule.RequestExceptionHandler Processing of request failed with uncaught exception:org.apache.http.client.utils.URLEncodedUtils.parse(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/util/List;
java.lang.NoSuchMethodError:org.apache.http.client.utils.URLEncodedUtils.parse(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/util/List;
I have tried a lot of other combinations of methods and objects to get this request to send off to the server correctly and nothing seems to work. Hopefully I am overlooking something relatively simple.
Thanks in advance for any guidance you can provide.

You most likely have the wrong version or two versions of the apache httpcomponents on your classpath. If you are running Tapestry it will print out all packages on the classpath on the error page. Investigate there, find which httpcomponents is loaded, figure out where it comes from and fix it.
If this does not work, you should share some of your runtime environment with us. Which servlet engine, running from which IDE or are you running from the command line. Are you using Maven? If so share your pom. Etc.

Java web service client generated in Netbeans - getting Http Status Code 307

I use Netbeans to generate web service client code, client-style JAX-WS, so i can invoke a web service API.
However, when I invoke the web service API, I get the exception:
com.sun.xml.internal.ws.client.ClientTransportException: The server sent HTTP status code 307: Temporary Redirect
Why do I get this? What is the workaround? I know the problem isn't with the web service itself, because I can get responses fine via soapUI and .Net.

Faced the same problem about a month ago.
Web service client classes were generated using Apache CXF and web service returned HTTP
status 307, which led to the same exception.
Invocation of the same web service method using soapUI with property Follow Redirects set to true was successful and returned needed data.
After googling awhile, it looked like there is no property to enable following redirects in the JAX-WS for this.
So, below is the code which is currently working, though I'm not sure it is compliant with any standards:
Supposing generated client classes looks like:
// generated service class
public class MyWebServiceClient extends javax.xml.ws.Service {
// ...
private final QName portName = "...";
// ...
public RetrieveMyObjects getRetrieveMyObjects() {
return super.getPort(portName, RetrieveMyObject.class);
}
// ...
}
// generated port interface
// annotations here
public interface RetrieveMyObjects {
// annotations here
List<MyObject> getAll();
}
Now, upon executing following code:
MyWebServiceClient wsClient = new MyWebServiceClient("wsdl/location/url/here.wsdl");
RetrieveMyObjectsPort retrieveMyObjectsPort = wsClient.getRetrieveMyObjects();
wsClient should return instance which is both instance of RetrieveMyObjects & javax.xml.ws.BindingProvider interfaces. It is not stated anywhere on the surface of JAX-WS, but it seems that a lot of code is based on that fact. One can re-assure him\herself by executing something like:
if(!(retrieveMyObjectsPort instanceof javax.xml.ws.BindingProvider)) {
throw new RuntimeException("retrieveMyObjectsPort is not instance of " + BindingProvider.class + ". Redirect following as well as authentication is not possible");
}
Now, when we are sure that retrieveMyObjectsPort is instance of javax.xml.ws.BindingProvider we can send plain HTTP POST request to it, simulating SOAP request (though it looks incredibly incorrect & ugly, but this works in my case and I didn't find anything better while googling) and check whether web service will send redirect status as a response:
// defined somewhere before
private static void checkRedirect(final Logger logger, final BindingProvider bindingProvider) {
try {
final URL url = new URL((String) bindingProvider.getRequestContext().get(ENDPOINT_ADDRESS_PROPERTY));
logger.trace("Checking WS redirect: sending plain POST request to {}", url);
final HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setInstanceFollowRedirects(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "text/html; charset='UTF-8'");
connection.setDoOutput(true);
if(connection.getResponseCode() == 307) {
final String redirectToUrl = connection.getHeaderField("location");
logger.trace("Checking WS redirect: setting new endpoint url, plain POST request was redirected with status {} to {}", connection.getResponseCode(), redirectToUrl);
bindingProvider.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, redirectToUrl);
}
} catch(final Exception e) {
logger.warn("Checking WS redirect: failed", e);
}
}
// somewhere at the application start
checkRedirect(logger, (BindingProvider) retrieveMyObjectsPort);
Now, what this method does is: it takes BindingProvider.ENDPOINT_ACCESS_PROPERTY of retrieveMyObjectsPort i.e. the url to which this port method will be sending SOAP requests and sends plain HTTP POST request as described above. Then it checks whether response status is 307 - Temporary Redirect (other statuses like 302 or 301 may also be included) and if it is, gets the URL to which web service is redirecting and sets new endpoint for the specified port.
In my case this checkRedirect method is called once for each web service port interface and then everything seems to work fine:
Redirect is checked on url like http://example.com:50678/restOfUrl
Web service redirects to url like https://example.com:43578/restOfUrl (please note that web service client authentication is present) - endpoint of a port is set to that url
Next web service requests executed via that port are successful
Disclaimer: I'm quite new to webservices and this is what I managed to achieve due to the lack of solutions for this questions, so please correct me if something is wrong here.
Hope this helps

Yes I know this post is old, but I've had similar errors, and thought maybe somebody would benefit from my solution.
the one that plagued me the most was:
com.sun.xml.ws.client.ClientTransportException: The server sent HTTP status code 200: OK
Which turns out to mean an incomplete response header. Apparently jax-ws does some kind of validation that includes validating the HTTP headers as well. And the server I was using was just sending an empty header.
It worked like a charm after adding 'application/soap+xml' to the Content-Type header.