Develop Reference

HttpsUrlConnection response returned from servlet contains extra 'b''0''\r\n' characters when read through python library

I am using HttpsURLConnection to call a server and return the response returned from the HttpsURLConnection from my servlet. I am copying the response from HttpssURLConnection to HttpServletresponse using streams, copying bytes from the httpconnection response input stream to the response's output stream, checking the end by seeing if read returns < 0.
Following is the code for copying the response. The variable response is of type HttpServletResponse and the variable httpCon is of type HttpsURLConnection.
InputStream responseStream = httpCon.getInputStream();
if (responseStream != null)
{
OutputStream os = response.getOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = responseStream.read(buffer)) >= 0)
{
os.write(buffer, 0, len);
}
os.flush();
os.close();
}
On the client side, I am using python requests library to read the response.
What I am seeing that if I use the curl to test my servlet, I am getting the proper response json, response = u'{"key":"value"}'.
If i read it from the requests python, it is putting some extra characters in the response , the response looks like the following
response = u'b0\r\n{"key":"value"}\r\n0\r\n\r\n'
Both the strings are unicode. But the second one has extra characters.
Same resonse if I try from curl/Postman restclient, I am able to get it properly. But from python requests, it is not working. I tried another livetest library in python, with that also, it is not working and the response has same characters. I also tried to change the accept-encoding header but it did not have any effect.
Because of this, I am not able to parse the json.
I don't want to change the client to parse this kind of string.
Can I change something on the server so that it will work correctly?

Did the response contain the below header "Transfer-Encoding: chunked"?
The response should be in Chunked transfer encoding
https://en.wikipedia.org/wiki/Chunked_transfer_encoding.
In this case, you get \r\n0\r\n\r\n at the end of the response is as expected since it is terminating symbol of this encoding. I guest curl/Postman just help us to handle Chunked transfer encoding, so you can't find these chunked symbols.

Incorrect Java HttpClient's response stream

In my application I need to parse a website and save some data from ir to the database. I am using HttpClient to get the page content. My code looks like this:
HttpClient client = new DefaultHttpClient();
System.out.println(doc.getUrl());
HttpGet contentGet= new HttpGet(siteUrl + personUrl);
HttpResponse response = client.execute(contentGet);
String html = convertStreamToString(response.getEntity().getContent());
/*
parse the page
*/
/***********************************************************************/
public static String convertStreamToString(InputStream is) throws Exception {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
return sb.toString();
}
I am doing this in a loop - I try to get content of some pages (their structure is the same). Sometimes it works fine, but unfortunately, my response in many cases is a sequence of similar trash liek this:
�=�v7���9�Hdz$�d7/�$�st��؎I��X^�$A6t_D���!gr�����C^��k#��MQ�2�d�8�]
I
I don't know where is the problem, please help me.
I have displayed headers of all responses that I got. For correct ones, there are:
Server : nginx/1.0.13
Date : Sat, 23 Mar 2013 21:50:31 GMT
Content-Type : text/html; charset=utf-8
Transfer-Encoding : chunked
Connection : close
Vary : Accept-Encoding
Expires : Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control : no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma : no-cache
Set-Cookie : pfSC=1; path=/; domain=.profeo.pl
Set-Cookie : pfSCvp=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.profeo.pl
For incorrect ones:
Server : nginx/1.2.4
Date : Sat, 23 Mar 2013 21:50:33 GMT
Content-Type : text/html
Transfer-Encoding : chunked
Connection : close
Set-Cookie : pfSCvp=3cff2422fd8f9b6e57e858d3883f4eaf; path=/; domain=.profeo.pl
Content-Encoding : gzip
Any other suggestions? My guess is that this gzip encoding is a problem here, but what can I do about it?

This probably has to do with some websites using a different character encoding in their response than your JVM default. To convert from a raw byte stream, like those provided by InputStreams, to a character stream (or a String), you have to choose a character encoding. HTTP responses can use different encodings, but they'll typically tell you what encoding they're using. You could do this manually by finding the "Content-Encoding" header of the HttpResponse, but your library provides a utility for doing this, since it's a common need. It's found in the EntityUtils class, and you can use it like so:
String html = EntityUtils.toString(response.getEntity());
You'll have to add
import org.apache.http.util.EntityUtils;
to the top of your file for that to work.
If that doesn't help, another possibility is that some of the URLs you're retrieving are binary, not textual, in which case the things you're trying to do don't make sense. If that's the case, you can possibly try to distinguish between the textual responses and the binary responses by checking Content-Type header, like so:
boolean isTextual = response.getFirstHeader("Content-Type").getValue().startsWith("text");
NEW MATERIAL:
After looking at the HTTP headers you added to your question, my best guess is that this is being caused by gzip compression of the responses. You can find more info on how to deal with that in this question, but the short version is that you should try using ContentEncodingHttpClient instead of DefaultHttpClient.
Another edit: ContentEncodingHttpClient is now deprecated, and you're supposed to use DecompressingHttpClient instead.

You need a httpclient which don't use compression.
I use this HttpClientBuilder.create().disableContentCompression().build() httpclient

Restlet Can't parse payload of a GET request

I am using Restlet in java as a simple client to make calls to a RESTful service provided by tastypie (python django)
I am monitoring the payload through tcpmon and I am seeing the following payload get returned however i can't get at the data in the payload.
Client calls
ClientResource resource = new ClientResource("http://localhost/someplace/");
String rep = resource.get(String.class);
Payload
HTTP/1.0 200 OK
Date: Thu, 12 Jul 2012 20:22:12 GMT
Server: WSGIServer/0.1 Python/2.7.1
Vary: Cookie
Content-Type: application/json; charset=utf-8
Set-Cookie: sessionid=63c5ea23113073e489cb8920819f37d; expires=Thu, 26-Jul-2012 20:22:12 GMT; httponly; Max-Age=1209600; Path=/
{"jsonData": "someData"}
When i debug into restlet i notice that the InboundWay.createEntity(..) is setting the data to EmptyRepresentation due to the lack of length, chunkEncoding, or connection headers. Why is this? Why can't I just stream the data?
Does any one know why this is happening? Is there a better client? So far tried jersey and Resteasy with limited success. (jeresy has a bug where it is trying to mark and reset a autoCloseSteam, and resteasy was just a pain to use as a client) I was hoping not to have to write an HTTPClient.
Did a bit more digging it looks like RESTLET is doing something quite silly in my oppinion.
It is looking a the response headers and trying to find if it knows the size, or if its chunkedEncoding, or if the Connection:closed is specified. Otherwise it does NOT try to parse the payload ... Does anyone know why this is? I didn't realize that those headers are required in any way. Why can't we use the ClosingInputStream when the connection close is not specified...
Restlet Code
public Representation createInboundEntity(Series<Parameter> headers) {
Representation result = null;
long contentLength = HeaderUtils.getContentLength(headers);
boolean chunkedEncoding = HeaderUtils.isChunkedEncoding(headers);
// In some cases there is an entity without a content-length header
boolean connectionClosed = HeaderUtils.isConnectionClose(headers);
// Create the representation
if ((contentLength != Representation.UNKNOWN_SIZE && contentLength != 0)
|| chunkedEncoding || connectionClosed) {
InputStream inboundEntityStream = getInboundEntityStream(
contentLength, chunkedEncoding);
ReadableByteChannel inboundEntityChannel = getInboundEntityChannel(
contentLength, chunkedEncoding);
...

Http Post not posting data

I'm trying to post some data from a Java client using sockets. It talks to localhost running php code, that simply spits out the post params sent to it.
Here is Java Client:
public static void main(String[] args) throws Exception {
Socket socket = new Socket("localhost", 8888);
String reqStr = "testString";
String urlParameters = URLEncoder.encode("myparam="+reqStr, "UTF-8");
System.out.println("Params: " + urlParameters);
try {
Writer out = new OutputStreamWriter(socket.getOutputStream(), "UTF-8");
out.write("POST /post3.php HTTP/1.1\r\n");
out.write("Host: localhost:8888\r\n");
out.write("Content-Length: " + Integer.toString(urlParameters.getBytes().length) + "\r\n");
out.write("Content-Type: text/html\r\n\n");
out.write(urlParameters);
out.write("\r\n");
out.flush();
InputStream inputstream = socket.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
String string = null;
while ((string = bufferedreader.readLine()) != null) {
System.out.println("Received " + string);
}
} catch(Exception e) {
e.printStackTrace();
} finally {
socket.close();
}
}
This is how post3.php looks like:
<?php
$post = $_REQUEST;
echo print_r($post, true);
?>
I expect to see an array (myparams => "testString") as the response. But its not passing post args to server.
Here is output:
Received HTTP/1.1 200 OK
Received Date: Thu, 25 Aug 2011 20:25:56 GMT
Received Server: Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/0.9.8r DAV/2 PHP/5.3.6
Received X-Powered-By: PHP/5.3.6
Received Content-Length: 10
Received Content-Type: text/html
Received
Received Array
Received (
Received )
Just a FYI, this setup works for GET requests.
Any idea whats going on here?

As Jochen and chesles rightly point out, you are using the wrong Content-Type: header - it should indeed be application/x-www-form-urlencoded. However there are several other issues as well...
The last header should be seperated from the body by a blank line between the headers and the body. This should be a complete CRLF (\r\n), in your code it is just a new line (\n). This is an outright protocol violation and I'm a little surprised you haven't just got a 400 Bad Request back from the server, although Apache can be quite forgiving in this respect.
You should specify Connection: close to ensure that you are not left hanging around with open sockets, the server will close the connection as soon as the request is complete.
The final CRLF sequence is not required. PHP is intelligent enough to sort this out by itself, but other server languages and implementations may not be...
If you are working with any standardised protocol in it's raw state, you should always start by at least scanning over the RFC.
Also, please learn to secure your Apache installs...

It looks like you are trying to send data in application/x-www-form-urlencoded format, but you are setting the Content-Type to text/html.

Use
out.write("Content-Type: application/x-www-form-urlencoded\n\n");
instead. As this page states:
The Content-Length and Content-Type headers are critical because they tell the web server how many bytes of data to expect, and what kind, identified by a MIME type.
For sending form data, i.e. data in the format key=value&key2=value2 use application/x-www-form-urlencoded. It doesn't matter if the value contains HTML, XML, or other data; the server will interpret it for you and you'll be able to retrieve the data as usual in the $_POST or $_REQUEST arrays on the PHP end.
Alternatively, you can send your data as raw HTML, XML, etc. using the appropriate Content-Type header, but you then have to retrieve the data manually in PHP by reading the special file php://input:
<?php
echo file_get_contents("php://input");
?>
As an aside, if you're using this for anything sufficiently complex, I would strongly recommend the use of an HTTP client library like HTTPClient.

How do I differentiate between request and response in an HTTPURLConnection?

Hi I need to evaluate if a server supports gzip compression.
Therefore I want to set gzip as Accept-Encoding in my request and evaluate if the response of the server containts "Content-Encoding: gzip".
However I am not quite clear on how to do this with HTTPURLConnection. Especially when to query my Connection object about the response. I currently do:
HttpURLConnection con = (HttpURLConnection) feedurl.openConnection();
con.setRequestProperty("Accept-Encoding", "gzip");
System.out.println("Current Accept-Encoding: "+con.getRequestProperty("Accept-Encoding"));
//check the response for the content-size
float feedsize = con.getContentLength()/1024f;
//the server uses transfer-encoding=chunked and does not specify
//a content length
if(con.getContentLength()==-1)
{
//count the content size manually
CountingInputStream counter = new CountingInputStream(con.getInputStream());
while(counter.read()!=-1)
{}
feedsize=counter.getCount()/1024f;
}
con.disconnect();

Have a look at this OReilly article. It illustrates how to generate the request, and how to interrogate the response and then create the appropriate stream (normal or gzipped) dependent on what's being returned.