Google HTTP Client Library for Java throws HttpResponseException: 301 Moved Permanently - java

I have a problem with Google HTTP Client Library for Java (1.22.0).
This is my code
String url = "http://gazetapraca.pl/ogl/2502758";
GenericUrl genericUrl = new GenericUrl(url);
ApacheHttpTransport apacheHttpTransport = new ApacheHttpTransport();
HttpRequest httpRequest = apacheHttpTransport.createRequestFactory().buildGetRequest(genericUrl);
httpRequest.setFollowRedirects(true);
HttpResponse httpResponse = httpRequest.execute();
and httpRequest.execute() throws
com.google.api.client.http.HttpResponseException: 301 Moved Permanently
Below is follow from Wireshark
GET /ogl/2502758 HTTP/1.1
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
Host: gazetapraca.pl
Connection: Keep-Alive
HTTP/1.1 301 Moved Permanently
Date: Sat, 26 Nov 2016 22:15:52 GMT
Server: Apache
Location: /ogl/2502758/pakowacz+-+mile+widziane+panie
Content-Length: 0
Set-Cookie: JSESSIONID_JOBS=2f1TffY6JYcb6zvBSrQ72fds7rfdsSnHM3sefw6D31Lfr434bnkDmdLQJLvLFZ6zkYBF!-12116034235597; path=/; HttpOnly
Content-Language: pl
P3P: CP="NOI DSP COR NID PSAo OUR IND"
Vary: User-Agent
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
GET /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie HTTP/1.1
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
Host: gazetapraca.pl
Connection: Keep-Alive
Cookie: JSESSIONID_JOBS=2f1TffY6JYcb6zvBSrQ72fds7rfdsSnHM3sefw6D31Lfr434bnkDmdLQJLvLFZ6zkYBF!-12116034235597
HTTP/1.1 301 Moved Permanently
Date: Sat, 26 Nov 2016 22:15:52 GMT
Server: Apache
Location: /ogl/2502758/pakowacz+-+mile+widziane+panie
Content-Length: 0
Content-Language: pl
P3P: CP="NOI DSP COR NID PSAo OUR IND"
Vary: User-Agent
Keep-Alive: timeout=2, max=99
Connection: Keep-Alive
and repeat a few times.
Maybe the problem is with url, because location is /ogl/2502758/pakowacz+-+mile+widziane+panie and next request method get is /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie.
In other software and library everything is working (google chrome browser, postman - addon to chrome, JSOUP - java library).
Does anyone have an idea how to solve the problem?

This is not your library's fault.
To understand why this problem is occurring, we must first understand the "error" message associated with your problem:
com.google.api.client.http.HttpResponseException: 301 Moved Permanently
So, what does this mean? Well, the last part of the error message, the description says "301 Moved Permanently". What that is referring to is an HTTP Status Code. An HTTP Status Code indicates what the outcome of a specific request is. In this case, the status code was 301, which according to RFC protocol means:
The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs.
So, this means that the URL that you are using is no longer valid, and that you have to use the new URL given to you by the Location response header. Now, it seems that the library that you're using is smart enough to detect this, and initializes a new request to the new URL. That's great and all, but your library that you are using, is incorrectly escaping the url provided by the Location header, and using that for the new request (turning /ogl/2502758/pakowacz+-+mile+widziane+panie into /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie), and the server receiving this request recognizes that those to paths are not the same (even though they should be. So, the server sends another 301 response, telling the client (the library in this case) to use the un-escaped URL instead of the escaped one, even though they should be the same.
Now, why is your library doing this? It turns out that, according to RFC spec, the '+' character is reserved for URIs. That means that that character, along with other characters are only intended for use in URIs for their intended purpose. Therefore, it is not standard to include the '+' character in URIs, unless it is used for a very specific purpose, which it looks like is not the case.
So, this all means that you cannot blame the library for this error, you can only blame the people who developed this site.
The reason that this works in your browser and other places is because those clients do not seem to be escaping the requested URL for you before sending it to the server.

Related

Getting error -> Invalid header field name, with 32 [duplicate]

I am new to Flutter and I am trying to call my ASP.NET server web API.
From the logs on my server, everything goes fine but Android Studio throws an exception: "invalid header field name".
Here is the code in dart:
import 'package:http/http.dart' as http;
...
_getService() async {
String result;
try {
var url = 'http://192.168.1.14:34263/api/Mobile/test/1';
Future<http.Response> response = http.get( url );
result = response.toString();
} catch(exception){
result = exception.toString();
debugPrint(result);
}
...
}
Here is the response header (obtained via Chrome):
Access-Control-Allow-Headers:accept, authorization, Content-Type
Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, PATCH, DELETE
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: WWW-Authenticate
Cache-Control: no-cache
Content-Encoding: deflate
Content-Length:79
Content-Type: application/xml; charset=utf-8
Date: Thu, 08 Mar 2018 01:01:25 GMT
Expires:-1
Pragma:no-cache
Server:MyTestServer
X-Content-Type-Options:NOSNIFF
X-Permitted-Cross-Domain-Policies:master-only
X-SourceFiles:=?UTF-8?BDpcTXlJbmNyZWRpYmxlRHJlc3NpbmdcTXlJbmNyZWRpYmxlRHJlc3NpbmdcTXlJbmNyZWRpYmxlRHJlc3NpbmdcYXBpXE1vYmlsZVxjb3Vjb3VcMQ==?=
X-XSS-Protection:1;mode=block
Here is the answer which is returned:
<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">test</string>
Can anyone tell me what am I doing wrong?
Many thanks
Ok, I finally found out, by debugging the code.
In fact, my server added a series of field names in the response's header (via the Web.config) and the last character of one of these field names was a space.
As a result, the http_parser.dart threw an exception since spaces are no authorized characters in header field name.
Nothing was detected by Chrome (or any browser) nor by Postman.
I had similar problem and after some heavy debugging
I removed these headers from nginx:
#add_header X−Content−Type−Options nosniff;
#add_header X−Frame−Options SAMEORIGIN;
#add_header X−XSS−Protection 1;
and it works fine. So most likely it's backend - header related issue

404 error when uploading file to GCS with query string on Production

I am using flash to upload a file to GCS.
I create a upload url using the following successUrl
"/cms/requestManager?ps={"1":{"action":"Upload Audio Recording","sendId":1,"data":{"fileName":"testQuestionAudioRecording","category":"testQuestionRecording"}},"jsInstanceID":"ahdzfmRldi1keW5hY3RpdmVzb2Z0d2FyZXITCxIKSlNJbnN0YW5jZRjKm-kBDKIBBmRzLmNtcw","userActive":true}"
When I send the file upload it comes back with a 404 error.
I check in the appengine instance logs for 404 errors and there are none.
The file I tried to upload does show up in the GCS bucket.
In the documentation for blobstoreService.createUploadUrl() it does not mention that query strings are not allowed. I would also expect that if there were something wrong with the successUrl that an IllegalArgumentException would be thrown.
If I go directly to the successUrl it will respond correctly with JSON data.
This is the link if you want to test
On my local test server it works fine. So I am seeing this issue only on production.
If I take out the query string, it will work. If I add a simpler query string like /cms/requestManager?blah=blah it will not work.
Does anyone have any ideas what is going on?
Is there a way to see what is happening on at /_ah/upload?
Could it be possible that it is 404ing on my end but not logging it? If so, what can I do find that out?
Here is the post and response headers and content.
Response Header:
HTTP/1.1 404 Not Found
Content-Length: 0
Date: Fri, 06 Mar 2015 00:25:25 GMT
Server: UploadServer ("Built on Feb 18 2015 18:10:26 (1424311826)")
Content-Type: text/html; charset=UTF-8
Alternate-Protocol: 80:quic,p=0.08
Request Headers:
POST /_ah/upload/AMmfu6a-yuYk7Gm1YMtqLt-GgyizWYLXRStoAZ49FCNNRHIVXMFrRh0Jo1aCSXf4c8uXrjJnCQwvSq7cFdVeI1v5J59jQeVfwvR6STMbs0hBw-GncTI1JHgE4NItQd2JQGgyHIqTasjR4lK_5g8-M0Nf1YdWr29by5Mskk07tpSLNUE0mEW2IOjlWP_Usa7ObJMFdElVqIsgd5a_Bq198AB9oprS6DMHkiYKsdJGCTmHe93w_PEoi-XiROlHSpEsi8TLwQUPPC86iqQsn6th4OGoRudOcdAVhFxCq1VJjXx2frv_Lm0khSxpmOU2nQu8gsbP-IaAgVWMe-0UUgsLYdODjqJhgkr-IB1h05F50pAXTgzImQHyv7ygvOhHO0zGUBAiH1a072LJa7EiN1TXcBYcDfbbMikR5lBdaqfqHccx23RZrDiyGfR3kwMiXmgqFy5BJY5Ht9ZoYlfST76laeM5v2PSheMqweHIZQATBZ0Zxojzv1mFr3ECasfwEY98ambjjrbgqysPwg56B-EYDCwUBAV5FvzY6hIGKnhjun9-rorrx0CuKww/ALBNUaYAAAAAVPj2QZXMlxhUs1-Au6wnswd3PTvLwL_G/ HTTP/1.1
Host: dev-dynactivesoftware.appspot.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:36.0) Gecko/20100101 Firefox/36.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.7,es;q=0.3
Accept-Encoding: gzip, deflate
Cookie: ahdzfmRldi1keW5hY3RpdmVzb2Z0d2FyZXITCxIKSlNJbnN0YW5jZRjKm-kBDKIBBmRzLmNtcw=1425601513386
Connection: keep-alive
appid = dev-dynactivesoftware
successUrl length = 256
Link to blobstoreService.createUploadUrl documentation
UPDATE:
I wrote some stuff into our system that allowed for data that was in the query string to be put into the url. So instead of
/requestManager?ps={} it would be
/requestManager/My Action/lsdkjfalsdkfjaldkfjaldkjf
It also 404d so I changed it to have no spaces in the url.
/requestManager/MyAction/lsdkjfalsdkfjaldkfjaldkjf
and it did not 404.
Could it be that the upload service is having problems using successUrls that have special characters in it, like spaces and ?.
The successUrl that you send into blobstoreService.createUploadUrl() needs to be url encoded first. So I modified the code that gets the uploadUrl to be the following:
String successUrl = URLEncoder.encode("/requestManager/My Action/askldjfkajsdflkasdkjh", "UTF8");
String uploadUrl = blobstoreService.createUploadUrl(successUrl);
I also had to decode the url later so that it was usable.
It seems to me that if the successUrl has characters that need to be encoded (because the service will return a 404 if it doesn't) then it should throw "java.lang.IllegalArgumentException - If successPath was not valid."
It would also be nice if the uploadServer would return something with the 404 message to indicate that successUrl 404d and this is the url we tried. At that point we would be able to see what is going on a lot faster.
I guess the even better fix would be to just put the code that the dev server runs on live because the dev environment worked perfect.

URL encoding (umlauts) with redirecting (302 Moved Temporarily)

I am working on a servlet that takes a unique identifier as an input and either returns a document or sends a 302 Moved Temporarily redirect, if the document can be found on another server. The other server is a linux box with apache and exposes files through webdav. I have no control over this server.
Redirecting works fine, as long as there are no special characters (German umlauts) in the URL.
A problematic ressource looks like this http://webfolder.company.com/projekt/übersicht.pdf
When I copy and paste this URL into my browser (Chrome), I can open the file with no problem (the umlaut stays in the URL).
But when I send the redirect from the server side, the URL gets encoded into http://webfolder.company.com/projekt/%C3%BCbersicht.pdf and for some reasons I don't understand, this leads to a 404 - Not Found.
My servlet implementation gets the URL from the database (MySQL, UTF-8) and uses HttpServletResponse.sendRedirect method.
response.sendRedirect(remoteRessource);
(remoteRessource is a String). Using curl I can see the Location header being set, but Chrome (and Firefox) encode the URL into http://webfolder.company.com/projekt/%C3%BCbersicht.pdf, which leads to the 404 error.
< HTTP/1.1 302 Moved Temporarily
< Date: Wed, 06 Mar 2013 20:50:16 GMT
< Server: Apache-Coyote/1.1
< Location: http://webfolder.company.com/projekt/übersicht.pdf
< Content-Type: text/html;charset=UTF-8
< Content-Length: 0
< Set-Cookie: JSESSIONID=FA670C0DDA55593368A3285F54894153; Path=/; HttpOnly
< Vary: Accept-Encoding
What is the proper way to provide a redirect URL as-is to a browser that does not get encoded in any way? Thanks.
Fix
Thanks to Julian Reschke's answer, I found out about the different encodings used by both servers. I now let Java/the URI class handle the encodings and things work fine.
URI uri = new URI(schema, host, path, null);
String location = uri.toASCIIString();
response.sendRedirect(location);
A likely reason is that the server running Apache+WebDAV expects an encoding different from UTF.8. ISO-8859-1 maybe?
You have to use i18n package for internationalization of characters instead of sending localized characters [in this case german umlaut]
Take a look into http://docs.oracle.com/javase/1.5.0/docs/api/java/net/URLEncoder.html
which is what's happening in your case
And browse through here for encoding and here for decoding

HTTPUrlConnection and application/octet-stream content-type response handling

I'm writing an Android media player that uses Java's HTTPUrlConnection class to access URL's. I was recently sent a bug report related to the following URL:
"http://listen.theradio.cc"
This URL redirects to "http://listen.theradio.cc/theradiocc.pls" which returns a playlist in PLS format. The problem I'm facing is that my application usually determines what to do with a URL best on the content-type header field. The URL I posted above returns a content-type of "application/octet-stream", which can be anything. So instead of trying to parse the returned playlist my application tries to play the URL (which fails, obviously). Is there any other way to effectively determine what type of content a URL is returning in a situation like this? Should I attempt to obtain an InputStream and check the first few lines of returned content?
Anything can be application/octet-stream thus it is misleading as the server does not return the expected content type for play-list.
Its better to fix in the server side.
Additionally doing a HEAD request it should be possible to know the information for the URL.
Making a curl request to http://listen.theradio.cc/theradiocc.pls returns following.
curl -i http://listen.theradio.cc/theradiocc.pls
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 26 Apr 2012 03:44:42 GMT
Content-Type: application/octet-stream
Content-Length: 111
Last-Modified: Tue, 15 Nov 2011 23:57:04 GMT
Connection: keep-alive
Accept-Ranges: bytes
[playlist]
NumberOfEntries=1
File1=http://theradio.cc:8000/trcc-stream
Title1=TheRadio.CC
Length1=-1
Version=2
So things needs to be fixed and properly fixed!

Httpclient / JSONObject

I am trying to autologin sending a JSONObject. Im gettign the reposnse as 302 Moved Temporarily which means I should redirect to another url. But my response.toString() shows "Location: /". Below is the code.
String input_text = "https://www.hautelook.com/v3/credential";
HttpPost httpost = new HttpPost(input_text);
String data = "{\"screen_resolution\":{\"height\":1080,\"width\":1920}}";
JSONObject jo=new JSONObject();
jo.put("email","sfhgfjk");
jo.put("passsword","dfjhsdkj");
jo.put("meta",data);
StringEntity se = new StringEntity( "JSON: " + json.toString());
se.setContentEncoding("UTF-8");
se.setContentEncoding(new BasicHeader(HTTP.CONTENT_TYPE, "application/json"));
httpost.setEntity(se);
HttpResponse response = httpclient.execute(httpost);
entity = response.getEntity();
What would be wrong?
This is the response.
response HTTP/1.1 302 Moved Temporarily [Server: nginx, Content-Type: text/html,
Location: /, Content-Encoding: gzip, Content-Length: 20, Expires: Thu, 16 Feb 2
012 19:07:55 GMT, Cache-Control: max-age=0, no-cache, no-store, Pragma: no-cache
, Date: Thu, 16 Feb 2012 19:07:55 GMT, Connection: keep-alive, Set-Cookie: PHPSE
SSID=vmoqeksits8ccukvnf7k4rdv75; path=/]
You can always verify if this is correct manually by issuing the request via CURL, or even your browser. For example, typing https://www.hautelook.com/v3/credential into a browser location bar causes a redirect to https://www.hautelook.com (with a nice little login dialog being shown by jQuery). So you know at least, that the behavior is consistent.
This can mean one of several things:
The endpoint you are using is incorrect (this is probably not the
case)
The authentication information you are supplying is incorrect (also
unlikely, because we would expect a 401 unauthorized in that case)
The way you are passing the authentication information is incorrect.
Without knowing more about the API its hard to say, but you should consult the docs again to ensure you are making the call correctly.
* EDIT*
Ok, tested with REST client and there are some things to correct in your code:
Change 'passsword' to 'password'
Change the line:
Original:
new StringEntity( "JSON: " + json.toString())
To:
new StringEntity(json.toString())
This should allow the request through, though I'm still not sure this is the correct endpoint, since I get back an HTML page. One last thing, its always good to remove your API credentials before posting your code to SO. I'm including a screenshot of the request below:
The default redirect strategy used by HttpClient 4.x honors restrictions on automatic redirection of entity enclosing methods such as POST and PUT imposed by the HTTP specification. Per requirements of the HTTP specification 302 Moved Temporarily, 301 Moved Permanently and 307 Temporary Redirect status codes may not be handled automatically for POST and PUT methods without an explicit confirmation by the user.
HttpClient 4.2 can be configured to use LaxRedirectStrategy that handles all types of redirects automatically regardless of the restrictions imposed by the specification. With earlier versions one can implement a custom redirect strategy as described here: Httpclient 4, error 302. How to redirect? (as suggested by Bob Kuhar).
At the same time I have to say that 'Location: /' header looks somewhat suspicious and even automatic redirect to that location may not necessarily produce the desired effect.
response HTTP/1.1 302 Moved Temporarily means, some redirection happened.
One of the example case is, Single sign on (or) Authorization required. Without authorization cookie when you try to access the resource using URL you may be redirected for Authorization with response as 302.
You could just configure your HTTPClient to follow redirects. Prior to 4.x, it was just a method on the HTTPMethod. Add...
HttpPost httpost = new HttpPost(input_text);
httpost.setFollowRedirect( true );
...but this wasn't pure enough or something and they changed it in 4.x. I haven't tried it since, so am reluctant to post code. But this question has come up and been answered here before. Maybe this helps you? Httpclient 4, error 302. How to redirect?

Categories

Resources