Develop Reference

HTTP Accept header makes response invalid

I am trying to make a Java servlet that will download a file to be opened in a browser window (or downloaded if the browser can't view it). As an example, I'm trying to fetch and view a PNG image in my browser.
I tried sending a request with no headers (except Host: localhost) via Fiddler, and I got back the image I was looking for.
When I try to view the image in my browser with the same URL, I get an invalid (transparent) image back. I tried adding the headers from my browser (Chrome) to Fiddler, and it looks like it fails when I add the following header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
When I remove that header, or change it to Accept: */*, the image loads again. I looked at the raw response in Fiddler and found there is a small difference in the actual data sent back.
This is what my servlet's code looks like. I'm not manually checking the Accept header for anything:
byte[] data = getData();
response.setContentType("image/png");
response.setHeader("Content-disposition", "inline; filename=\"my_image.png\"");
response.setContentLength(data.length);
OutputStream output = response.getOutputStream();
output.write(data);
output.close();
Why does Chrome's default Accept header make my response unreadable? How can I make my servlet ignore that header and send back the same data every time?
EDIT:
This is what Fiddler shows in the "Raw" view when I request the servlet without the Accept header (good):
HTTP/1.1 200 OK
x-xr-bookmark: ad70ff81-7415-4d46-954d-b79f98056729
content-disposition: inline; filename="image.PNG"
content-type: image/png
content-length: 82726
date: Tue, 21 Mar 2017 19:49:32 GMT
connection: close
PNG
IHDR 5- sRGB gAMA
a pHYs t t fx IDATx^ {p\ } &M $mg I K; N3 t`
*** FIDDLER: RawDisplay truncated at 128 characters. Right-click to disable truncation. ***
And this is what Fiddler shows when I request it with the Accept header mentioned above (bad):
HTTP/1.1 200 OK
x-xr-bookmark: 5a1b1e6d-6b68-42ac-a2cb-545811f9a879
content-disposition: inline; filename="image.PNG"
content-type: image/png
date: Tue, 21 Mar 2017 19:49:46 GMT
connection: close
Transfer-Encoding: chunked
258ec
PNG
IHDR 5- sRGB gAMA
a pHYs t t fx IDATx^ {p\ } &M $mg I K; N3
*** FIDDLER: RawDisplay truncated at 128 characters. Right-click to disable truncation. ***

java - do json remote procedure call(RPC) from client

I'm working on a java application. I need to call a remote api method. Suppose I have this information: remote_ip, remote_port, remote_method_name and some key-value data to post. I need to post my data to remote server through TCP protocol. I tested Sockets in this way, but not working:
Socket socket = new Socket(remote_ip, remote_port);
BufferedWriter wr = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), "UTF8"));
String params = URLEncoder.encode("key1", "UTF-8")
+ "=" + URLEncoder.encode(value1, "UTF-8");
params += "&" + URLEncoder.encode("key2", "UTF-8")
+ "=" + URLEncoder.encode(value2, "UTF-8");
wr.write("POST " + remote_method_name + " HTTP/1.0\r\n");
wr.write("Content-Length: " + params.length() + "\r\n");
wr.write("Content-Type: application/x-www-form-urlencoded\r\n");
wr.write("\r\n");
wr.write(params);
wr.flush();
Could any one tell me how can I call api method in the correct way?
I want to do it without any third-party library if possible.
Any help would be gratefully appreciated.

First of all, even though you say you want to use raw tcp sockets, you're very clearly trying to make an HTTP rest request. It will be way easier and more appropriate to use an http client for that. I you don't want to use third-party libraries, use the built-in HttpUrlConnection (example usage).
Another advantage is that using an http client will give you a clear(er) error message.
Second, are you sure about that content-type? If you're trying to submit json, normally the header to set would be Content-Type: application/json.
Third, if you're getting 404 not found, I'd bet the url you're posting to is incorrect. Double-check the domain and baseurl with whoever gave you the specs for this API. Right now your URL is essentially http://remote_ip:remote_port/remote_method_name which is quite unlikely to be correct.

i think the reason is that the 'remote_method_name' you provided is wrong.
since you are making a http call, here is a brief example for you to refer.
for the page you are reading now, the request should be:
curl -v 'http://stackoverflow.com/questions/40171522/java-do-json-remote-procedure-callrpc-from-client'
* Trying 151.101.193.69...
* Connected to stackoverflow.com (151.101.193.69) port 80 (#0)
> GET /questions/40171522/java-do-json-remote-procedure-callrpc-from-client HTTP/1.1
> Host: stackoverflow.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Last-Modified: Fri, 21 Oct 2016 09:01:29 GMT
< X-Frame-Options: SAMEORIGIN
< X-Request-Guid: 405a2900-543b-4a97-8c62-8fa9019ab934
< Content-Length: 77809
< Accept-Ranges: bytes
< Date: Fri, 21 Oct 2016 09:18:59 GMT
< Via: 1.1 varnish
< Age: 0
< Connection: keep-alive
< X-Served-By: cache-ams4437-AMS
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1477041539.483029,VS0,VE95
< X-DNS-Prefetch-Control: off
< Set-Cookie: prov=aef7ece4-db49-60e0-3209-a2a2830d8749; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
<
<!DOCTYPE html>
<html itemscope itemtype="http://schema.org/QAPage">
......

Incorrect Java HttpClient's response stream

In my application I need to parse a website and save some data from ir to the database. I am using HttpClient to get the page content. My code looks like this:
HttpClient client = new DefaultHttpClient();
System.out.println(doc.getUrl());
HttpGet contentGet= new HttpGet(siteUrl + personUrl);
HttpResponse response = client.execute(contentGet);
String html = convertStreamToString(response.getEntity().getContent());
/*
parse the page
*/
/***********************************************************************/
public static String convertStreamToString(InputStream is) throws Exception {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
return sb.toString();
}
I am doing this in a loop - I try to get content of some pages (their structure is the same). Sometimes it works fine, but unfortunately, my response in many cases is a sequence of similar trash liek this:
�=�v7���9�Hdz$�d7/�$�st��؎I��X^�$A6t_D���!gr�����C^��k#��MQ�2�d�8�]
I
I don't know where is the problem, please help me.
I have displayed headers of all responses that I got. For correct ones, there are:
Server : nginx/1.0.13
Date : Sat, 23 Mar 2013 21:50:31 GMT
Content-Type : text/html; charset=utf-8
Transfer-Encoding : chunked
Connection : close
Vary : Accept-Encoding
Expires : Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control : no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma : no-cache
Set-Cookie : pfSC=1; path=/; domain=.profeo.pl
Set-Cookie : pfSCvp=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.profeo.pl
For incorrect ones:
Server : nginx/1.2.4
Date : Sat, 23 Mar 2013 21:50:33 GMT
Content-Type : text/html
Transfer-Encoding : chunked
Connection : close
Set-Cookie : pfSCvp=3cff2422fd8f9b6e57e858d3883f4eaf; path=/; domain=.profeo.pl
Content-Encoding : gzip
Any other suggestions? My guess is that this gzip encoding is a problem here, but what can I do about it?

This probably has to do with some websites using a different character encoding in their response than your JVM default. To convert from a raw byte stream, like those provided by InputStreams, to a character stream (or a String), you have to choose a character encoding. HTTP responses can use different encodings, but they'll typically tell you what encoding they're using. You could do this manually by finding the "Content-Encoding" header of the HttpResponse, but your library provides a utility for doing this, since it's a common need. It's found in the EntityUtils class, and you can use it like so:
String html = EntityUtils.toString(response.getEntity());
You'll have to add
import org.apache.http.util.EntityUtils;
to the top of your file for that to work.
If that doesn't help, another possibility is that some of the URLs you're retrieving are binary, not textual, in which case the things you're trying to do don't make sense. If that's the case, you can possibly try to distinguish between the textual responses and the binary responses by checking Content-Type header, like so:
boolean isTextual = response.getFirstHeader("Content-Type").getValue().startsWith("text");
NEW MATERIAL:
After looking at the HTTP headers you added to your question, my best guess is that this is being caused by gzip compression of the responses. You can find more info on how to deal with that in this question, but the short version is that you should try using ContentEncodingHttpClient instead of DefaultHttpClient.
Another edit: ContentEncodingHttpClient is now deprecated, and you're supposed to use DecompressingHttpClient instead.

You need a httpclient which don't use compression.
I use this HttpClientBuilder.create().disableContentCompression().build() httpclient

Http-Server How to Create Request-Headers and response-Headers

SOS SOS SOS PLEASE!!!
I have created a primitive HttpServer in java which listens on port 80 and Uses Get method to open a file etc (127.0.0.1/index.html). ow i want create request headers (Accept,Accept Language,User-Agent) and response headers (Content-Length and Cache-Control) from HTTP/1.1 (RFC 2616) protocol.
Can you help me how to do that...You will save my life!!!!!!!!
Thanks!

Headers are just lines following initial GET/POST/* operation. Last header is separated from the content by an empty line. So all you need to do (both on the client and server sides) is to write a few lines into the request/response before the content.
HTTP/1.0 200 OK
Date: Fri, 31 Dec 1999 23:59:59 GMT
Content-Type: text/html
Content-Length: 1354
<html>
<body>
...
(more file contents)
P.S. Java has a built-in HTTP server, did you know that?
com.sun.net.HttpServer:
HttpServer httpServer = HttpServer.create(new InetSocketAddress(port), 5);
httpServer.createContext("/", new MyRequestHandler());
httpServer.setExecutor(Executors.newCachedThreadPool());
httpServer.start();

Counting Bytes sent/received over HTTP

In Java,
How could the bytes sent and received over an active HTTP connection be counted?
I want to display some statistics like:
Bytes Sent : xxxx Kb
Bytes Received : xxxx Kb
Duration : hh:mm

It is difficult to see how you could decorate HttpConnection to count the raw byte data. You could reimplement HTTP using sockets, but this would have to be pretty important to go to those lengths.
Sample response from stackoverflow.com:
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Expires: Mon, 21 Sep 2009 11:46:48 GMT
Vary: Accept-Encoding
Server: Microsoft-IIS/7.0
Date: Mon, 21 Sep 2009 11:46:48 GMT
Content-Length: 19452
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>...remaining data....
HTTP requests are reasonably simple, so you might try reconstructing it using the information that is exposed. Something of the form:
// TODO: edge cases, error handling, header delimiter, etc.
int byteCount = 0;
int headerIndex = 0;
while (true) {
String key = httpConnection.getHeaderFieldKey(headerIndex);
if (key == null)
break;
String value = httpConnection.getHeaderField(headerIndex++);
byteCount += key.getBytes("US-ASCII").length
+ value.getBytes("US-ASCII").length + 2;
}
byteCount += httpConnection.getHeaderFieldInt("Content-Length",
Integer.MIN_VALUE);
That implementation is incomplete and untested. You'd need to study up on the details of the HTTP protocol to ensure the accuracy of your results.

Wrap the HTTPURLConnection's getInputStream() and getOutputStream() in your own streams that count the bytes that go through them. Or even better use the Apache Commons IO library - they have counting stream implementations in there.