Develop Reference

HTTP Accept header makes response invalid

I am trying to make a Java servlet that will download a file to be opened in a browser window (or downloaded if the browser can't view it). As an example, I'm trying to fetch and view a PNG image in my browser.
I tried sending a request with no headers (except Host: localhost) via Fiddler, and I got back the image I was looking for.
When I try to view the image in my browser with the same URL, I get an invalid (transparent) image back. I tried adding the headers from my browser (Chrome) to Fiddler, and it looks like it fails when I add the following header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
When I remove that header, or change it to Accept: */*, the image loads again. I looked at the raw response in Fiddler and found there is a small difference in the actual data sent back.
This is what my servlet's code looks like. I'm not manually checking the Accept header for anything:
byte[] data = getData();
response.setContentType("image/png");
response.setHeader("Content-disposition", "inline; filename=\"my_image.png\"");
response.setContentLength(data.length);
OutputStream output = response.getOutputStream();
output.write(data);
output.close();
Why does Chrome's default Accept header make my response unreadable? How can I make my servlet ignore that header and send back the same data every time?
EDIT:
This is what Fiddler shows in the "Raw" view when I request the servlet without the Accept header (good):
HTTP/1.1 200 OK
x-xr-bookmark: ad70ff81-7415-4d46-954d-b79f98056729
content-disposition: inline; filename="image.PNG"
content-type: image/png
content-length: 82726
date: Tue, 21 Mar 2017 19:49:32 GMT
connection: close
PNG
IHDR 5- sRGB gAMA
a pHYs t t fx IDATx^ {p\ } &M $mg I K; N3 t`
*** FIDDLER: RawDisplay truncated at 128 characters. Right-click to disable truncation. ***
And this is what Fiddler shows when I request it with the Accept header mentioned above (bad):
HTTP/1.1 200 OK
x-xr-bookmark: 5a1b1e6d-6b68-42ac-a2cb-545811f9a879
content-disposition: inline; filename="image.PNG"
content-type: image/png
date: Tue, 21 Mar 2017 19:49:46 GMT
connection: close
Transfer-Encoding: chunked
258ec
PNG
IHDR 5- sRGB gAMA
a pHYs t t fx IDATx^ {p\ } &M $mg I K; N3
*** FIDDLER: RawDisplay truncated at 128 characters. Right-click to disable truncation. ***

Google HTTP Client Library for Java throws HttpResponseException: 301 Moved Permanently

I have a problem with Google HTTP Client Library for Java (1.22.0).
This is my code
String url = "http://gazetapraca.pl/ogl/2502758";
GenericUrl genericUrl = new GenericUrl(url);
ApacheHttpTransport apacheHttpTransport = new ApacheHttpTransport();
HttpRequest httpRequest = apacheHttpTransport.createRequestFactory().buildGetRequest(genericUrl);
httpRequest.setFollowRedirects(true);
HttpResponse httpResponse = httpRequest.execute();
and httpRequest.execute() throws
com.google.api.client.http.HttpResponseException: 301 Moved Permanently
Below is follow from Wireshark
GET /ogl/2502758 HTTP/1.1
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
Host: gazetapraca.pl
Connection: Keep-Alive
HTTP/1.1 301 Moved Permanently
Date: Sat, 26 Nov 2016 22:15:52 GMT
Server: Apache
Location: /ogl/2502758/pakowacz+-+mile+widziane+panie
Content-Length: 0
Set-Cookie: JSESSIONID_JOBS=2f1TffY6JYcb6zvBSrQ72fds7rfdsSnHM3sefw6D31Lfr434bnkDmdLQJLvLFZ6zkYBF!-12116034235597; path=/; HttpOnly
Content-Language: pl
P3P: CP="NOI DSP COR NID PSAo OUR IND"
Vary: User-Agent
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
GET /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie HTTP/1.1
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
Host: gazetapraca.pl
Connection: Keep-Alive
Cookie: JSESSIONID_JOBS=2f1TffY6JYcb6zvBSrQ72fds7rfdsSnHM3sefw6D31Lfr434bnkDmdLQJLvLFZ6zkYBF!-12116034235597
HTTP/1.1 301 Moved Permanently
Date: Sat, 26 Nov 2016 22:15:52 GMT
Server: Apache
Location: /ogl/2502758/pakowacz+-+mile+widziane+panie
Content-Length: 0
Content-Language: pl
P3P: CP="NOI DSP COR NID PSAo OUR IND"
Vary: User-Agent
Keep-Alive: timeout=2, max=99
Connection: Keep-Alive
and repeat a few times.
Maybe the problem is with url, because location is /ogl/2502758/pakowacz+-+mile+widziane+panie and next request method get is /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie.
In other software and library everything is working (google chrome browser, postman - addon to chrome, JSOUP - java library).
Does anyone have an idea how to solve the problem?

This is not your library's fault.
To understand why this problem is occurring, we must first understand the "error" message associated with your problem:
com.google.api.client.http.HttpResponseException: 301 Moved Permanently
So, what does this mean? Well, the last part of the error message, the description says "301 Moved Permanently". What that is referring to is an HTTP Status Code. An HTTP Status Code indicates what the outcome of a specific request is. In this case, the status code was 301, which according to RFC protocol means:
The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs.
So, this means that the URL that you are using is no longer valid, and that you have to use the new URL given to you by the Location response header. Now, it seems that the library that you're using is smart enough to detect this, and initializes a new request to the new URL. That's great and all, but your library that you are using, is incorrectly escaping the url provided by the Location header, and using that for the new request (turning /ogl/2502758/pakowacz+-+mile+widziane+panie into /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie), and the server receiving this request recognizes that those to paths are not the same (even though they should be. So, the server sends another 301 response, telling the client (the library in this case) to use the un-escaped URL instead of the escaped one, even though they should be the same.
Now, why is your library doing this? It turns out that, according to RFC spec, the '+' character is reserved for URIs. That means that that character, along with other characters are only intended for use in URIs for their intended purpose. Therefore, it is not standard to include the '+' character in URIs, unless it is used for a very specific purpose, which it looks like is not the case.
So, this all means that you cannot blame the library for this error, you can only blame the people who developed this site.
The reason that this works in your browser and other places is because those clients do not seem to be escaping the requested URL for you before sending it to the server.

how to get file name from the server request?

I'm trying http requests and create http get request to server. Server must return .zip file. here is the code:
url = new URL(urlToRead);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
is = conn.getInputStream();
then I want to write it to file like this:
r = 1;
while(r > 0){
r = is.read(buf);
if(r > 0)
fos.write(buf, 0, r);
}
but to create fileoutputstream i want to use the file name provided by server. I've found that server answer has the file name and all the structure looks like this:
HTTP/1.1 200 OK
Date: Mon, 07 Apr 2003 14:51:19 GMT
Server: Apache/1.3.20 (Win32) PHP/4.3.0
Last-Modified: Mon, 07 Apr 2003 14:51:00 GMT
Accept-Ranges: bytes
Content-Length: 673
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/zip
Content-Disposition: attachment; filename=test.zip
Pragma: no-cache
....(zip content)
how to get filename?

You can use the HttpUrlConnection object's getHeaderField() method to read the Content-Disposition header. Also make sure you handle the case when it is not present.

Have you had a look at the methods that HttpURLConnection offers?
Especially the ones that involve headers?

if u have servletRequest object(lets say obj) then
i think this will help
obj.getRequestURL() ;

404 error when uploading file to GCS with query string on Production

I am using flash to upload a file to GCS.
I create a upload url using the following successUrl
"/cms/requestManager?ps={"1":{"action":"Upload Audio Recording","sendId":1,"data":{"fileName":"testQuestionAudioRecording","category":"testQuestionRecording"}},"jsInstanceID":"ahdzfmRldi1keW5hY3RpdmVzb2Z0d2FyZXITCxIKSlNJbnN0YW5jZRjKm-kBDKIBBmRzLmNtcw","userActive":true}"
When I send the file upload it comes back with a 404 error.
I check in the appengine instance logs for 404 errors and there are none.
The file I tried to upload does show up in the GCS bucket.
In the documentation for blobstoreService.createUploadUrl() it does not mention that query strings are not allowed. I would also expect that if there were something wrong with the successUrl that an IllegalArgumentException would be thrown.
If I go directly to the successUrl it will respond correctly with JSON data.
This is the link if you want to test
On my local test server it works fine. So I am seeing this issue only on production.
If I take out the query string, it will work. If I add a simpler query string like /cms/requestManager?blah=blah it will not work.
Does anyone have any ideas what is going on?
Is there a way to see what is happening on at /_ah/upload?
Could it be possible that it is 404ing on my end but not logging it? If so, what can I do find that out?
Here is the post and response headers and content.
Response Header:
HTTP/1.1 404 Not Found
Content-Length: 0
Date: Fri, 06 Mar 2015 00:25:25 GMT
Server: UploadServer ("Built on Feb 18 2015 18:10:26 (1424311826)")
Content-Type: text/html; charset=UTF-8
Alternate-Protocol: 80:quic,p=0.08
Request Headers:
POST /_ah/upload/AMmfu6a-yuYk7Gm1YMtqLt-GgyizWYLXRStoAZ49FCNNRHIVXMFrRh0Jo1aCSXf4c8uXrjJnCQwvSq7cFdVeI1v5J59jQeVfwvR6STMbs0hBw-GncTI1JHgE4NItQd2JQGgyHIqTasjR4lK_5g8-M0Nf1YdWr29by5Mskk07tpSLNUE0mEW2IOjlWP_Usa7ObJMFdElVqIsgd5a_Bq198AB9oprS6DMHkiYKsdJGCTmHe93w_PEoi-XiROlHSpEsi8TLwQUPPC86iqQsn6th4OGoRudOcdAVhFxCq1VJjXx2frv_Lm0khSxpmOU2nQu8gsbP-IaAgVWMe-0UUgsLYdODjqJhgkr-IB1h05F50pAXTgzImQHyv7ygvOhHO0zGUBAiH1a072LJa7EiN1TXcBYcDfbbMikR5lBdaqfqHccx23RZrDiyGfR3kwMiXmgqFy5BJY5Ht9ZoYlfST76laeM5v2PSheMqweHIZQATBZ0Zxojzv1mFr3ECasfwEY98ambjjrbgqysPwg56B-EYDCwUBAV5FvzY6hIGKnhjun9-rorrx0CuKww/ALBNUaYAAAAAVPj2QZXMlxhUs1-Au6wnswd3PTvLwL_G/ HTTP/1.1
Host: dev-dynactivesoftware.appspot.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:36.0) Gecko/20100101 Firefox/36.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.7,es;q=0.3
Accept-Encoding: gzip, deflate
Cookie: ahdzfmRldi1keW5hY3RpdmVzb2Z0d2FyZXITCxIKSlNJbnN0YW5jZRjKm-kBDKIBBmRzLmNtcw=1425601513386
Connection: keep-alive
appid = dev-dynactivesoftware
successUrl length = 256
Link to blobstoreService.createUploadUrl documentation
UPDATE:
I wrote some stuff into our system that allowed for data that was in the query string to be put into the url. So instead of
/requestManager?ps={} it would be
/requestManager/My Action/lsdkjfalsdkfjaldkfjaldkjf
It also 404d so I changed it to have no spaces in the url.
/requestManager/MyAction/lsdkjfalsdkfjaldkfjaldkjf
and it did not 404.
Could it be that the upload service is having problems using successUrls that have special characters in it, like spaces and ?.

The successUrl that you send into blobstoreService.createUploadUrl() needs to be url encoded first. So I modified the code that gets the uploadUrl to be the following:
String successUrl = URLEncoder.encode("/requestManager/My Action/askldjfkajsdflkasdkjh", "UTF8");
String uploadUrl = blobstoreService.createUploadUrl(successUrl);
I also had to decode the url later so that it was usable.
It seems to me that if the successUrl has characters that need to be encoded (because the service will return a 404 if it doesn't) then it should throw "java.lang.IllegalArgumentException - If successPath was not valid."
It would also be nice if the uploadServer would return something with the 404 message to indicate that successUrl 404d and this is the url we tried. At that point we would be able to see what is going on a lot faster.
I guess the even better fix would be to just put the code that the dev server runs on live because the dev environment worked perfect.

HTTPUrlConnection and application/octet-stream content-type response handling

I'm writing an Android media player that uses Java's HTTPUrlConnection class to access URL's. I was recently sent a bug report related to the following URL:
"http://listen.theradio.cc"
This URL redirects to "http://listen.theradio.cc/theradiocc.pls" which returns a playlist in PLS format. The problem I'm facing is that my application usually determines what to do with a URL best on the content-type header field. The URL I posted above returns a content-type of "application/octet-stream", which can be anything. So instead of trying to parse the returned playlist my application tries to play the URL (which fails, obviously). Is there any other way to effectively determine what type of content a URL is returning in a situation like this? Should I attempt to obtain an InputStream and check the first few lines of returned content?

Anything can be application/octet-stream thus it is misleading as the server does not return the expected content type for play-list.
Its better to fix in the server side.
Additionally doing a HEAD request it should be possible to know the information for the URL.
Making a curl request to http://listen.theradio.cc/theradiocc.pls returns following.
curl -i http://listen.theradio.cc/theradiocc.pls
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 26 Apr 2012 03:44:42 GMT
Content-Type: application/octet-stream
Content-Length: 111
Last-Modified: Tue, 15 Nov 2011 23:57:04 GMT
Connection: keep-alive
Accept-Ranges: bytes
[playlist]
NumberOfEntries=1
File1=http://theradio.cc:8000/trcc-stream
Title1=TheRadio.CC
Length1=-1
Version=2
So things needs to be fixed and properly fixed!