Develop Reference

Change header-body-separator with HttpURLConnection (from java.net)

I am sending a simple POST request with the built-in HttpURLConnection class, but I want to change the way Java separates the headers from the body (look at the outputs of tcpflow -a port 80 below). Here is the code:
// Create HttpURLConnection object
URL url = new URL("http://httpbin.org/post");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// Set request method
connection.setRequestMethod("POST");
// Write body and "Content-Length" header
String body = "This+is+the+body+of+the+post+request.";
connection.setRequestProperty("Content-Length",
String.valueOf(body.length()));
connection.setDoOutput(true);
connection.getOutputStream().write(body.getBytes("US-ASCII"));
// Send request
int responseCode = connection.getResponseCode();
When I execute that code and look at what Java actually sends using tcpflow -a port 80 (prints all requests/responses on port 80), I see the following (I cut away the response):
192.168.178.113.54654-054.225.177.165.00080: POST /post HTTP/1.1
User-Agent: Java/9
Host: httpbin.org
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-type: application/x-www-form-urlencoded
Content-Length: 37
192.168.178.113.54654-054.225.177.165.00080: This is the body of the post request.
The headers are correct, the body is correct. But I can see that the body is transferred in a separate connection. I know that this is a problem with java.net.HttpURLConnection because when I try the same with Apache's HttpClient, tcpflow -a port 80 gives me:
192.168.178.113.39708-054.243.202.193.00080: POST /post HTTP/1.1
Content-Length: 37
Host: httpbin.org
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.3 (Java/9)
Accept-Encoding: gzip,deflate
This+is+the+body+of+the+post+request.
Here the body is sent with the headers, separated with just /r/n/r/n. I would like the Java library (java.net.HttpURLConnection) to do the same. Is that possible?
EDIT: I found out that the reason the server rejected my requests was not that the body was in a different packet (packet, not connection, as Julian Reschke pointed out) than the headers but just that I sent the wrong data facepalm.

Encoded Http Request/Response body

I've built an Android proxy server passing http request and responses using Java Sockets.
The proxy is working, all content in browser is passing through it. However I would be able to read requests/responses but their body seems to be encoded:
GET http://m.onet.pl/ HTTP/1.1
Host: m.onet.pl
Proxy-Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Linux; Android 4.4.4; XT1039 Build/KXB21.14-L1.56) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36
DNT: 1
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,pl;q=0.4
Cookie: onet_ubi=201509221839473724130028; onetzuo_ticket=9AEDF08D278EC7965FF6A20BABD36EF0010012ED90FDD127C16068426F8B65A5D81A000000000000000050521881000000; onet_cid=dd6df83b3a8c33cd497d1ec3fcdea91b; __gfp_64b=2Mp2U1jvfJ3L9f.y6CbKfJ0oVfA7pVdBYfT58G1nf7T.p7; ea_uuid=201509221839478728300022; onet_cinf=1; __utma=86187972.1288403231.1442939988.1444999380.1445243557.40; __utmb=86187972.13.10.1445243557; __utmc=86187972; __utmz=86187972.1442939988.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
�����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
So both in request and response a lot of "���" occurs. I didn't find any info about http encoding. What is it ? How can I properly read body ?
Assuming it might be GZIPed message I tried:
while ((count = externalServerInputReader.read(buf, 0, buf.length)) != -1)
{
String stream = new String(buf, 0 , count);
proxyOutputStream.write(buf, 0, count);
if (stream.contains("content-encoding: gzip")) {
ByteArrayInputStream bais = new ByteArrayInputStream(buf);
GZIPInputStream gzis = new GZIPInputStream(bais);
InputStreamReader reader = new InputStreamReader(gzis);
BufferedReader in = new BufferedReader(reader);
String readed;
while ((readed = in.readLine()) != null) {
Log.d("Hello", "UnGzip: " + readed);
}
}
}
proxyOutputStream.flush();
However I get error on ungzipping attempt.
unknown format (magic number 5448)

I tried your sample request by saving it to "/tmp/req" and replaying it using cat /tmp/req | nc m.onet.pl 80. The server sent back a gzip encoded response, which I could tell from the response header content-encoding: gzip. In the case where the response is gzip encoded, you could decompress it in Java using java.util.zip.GZIPInputStream. Note that the user agent in your example is also advertising support for "deflate" and "sdch" too, so you may also get responses with those encodings. The "deflate" encoding can be decompressed using java.util.zip.InflaterInputStream. I'm not aware of any built in support for sdch, so you would need to find or write a library to decompress that - see this other Stack Overflow question for a possible starting point: "Java SDCH compressor/decompressor".
To address the updated part of your question where you added a stab at using GZIPInputStream, the most immediate issue is that you should only gunzip the stream after the HTTP response headers have ended. The simplest thing to do would be to wait for "\r\n\r\n" to come across the underlying InputStream (not a Reader) and then run the data starting with the next byte on through a single GZIPInputStream. That should probably work for the example you gave - I successfully decoded the replayed response I got using gunzip -c. For thoroughness, there are some other issues that will keep this from working as a general solution for arbitrary websites, but I think it will be enough to get you started. (Some examples: 1) you might miss a "content-encoding" header because you are splitting the response into chunks of length buf.length. 2) Responses which use chunked encoding would need to be de-chunked. 3) Keep-alive responses would necessitate that you track when the response ends rather than waiting for end of stream.)

jquery ajax and java server, lost data

i have this ajax function that looks like so
$.ajax({
type: "POST",
url: "http://localhost:55556",
data: "lots and lots of pie",
cache: false,
success: function(result)
{
alert("sent");
},
failure: function()
{
alert('An Error has occured, please try again.');
}
});
and a server that looks like so
clientSocket = AcceptConnection();
inp = new BufferedReader(new InputStreamReader (clientSocket.getInputStream()));
String requestString = inp.readLine();
BufferedReader ed = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
while(true){
String tmp = inp.readLine();
System.out.println(tmp);
}
now the odd thing is when i send my ajax my server gets by using system.out
Host: localhost:55556
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 20
Origin: null
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
the question is where is the data that i sent through, where is lots of pie?

The data should come after a blank line after the header lines, but I think the problem is that the data does not end with a newline character, and therefore, you cannot read it with the .readLine() method.
While looping through the header lines, you could look for the "Content-Length" line and get the length of the data. When you have reached the blank line, stop using .readLine(). Instead switch to reading one character at a time, reading the number of characters specified by the "Content-Length" header. I think you can find example code for this in this answer.
If you can, I suggest you use a library to help with this. I think the Apache HTTP Core library can help with this. See this answer.

Java socket connection hangs 60 seconds before reading last line

I have developed manually a soap connection via Java sockets (it was a very simple soap request and Axis was giving lots of build problem).
To achieve this, I basically copied the HTTP header I was getting out of Soap UI when , and coded the follwing:
String hostname = "aaaaa";
int port = 11111;
InetAddress addr = InetAddress.getByName(hostname);
Socket sock = new Socket(addr, port);
sock.setSoTimeout(100000);
BufferedWriter wr = new BufferedWriter(new OutputStreamWriter(sock.getOutputStream(),"UTF-8"));
wr.write("POST " + "http://aaaa:11111/servicePath" + " HTTP/1.1\r\n");
wr.write("Host: aaaaa:11111\r\n");
wr.write("Accept-Encoding: gzip,deflate\r\n");
wr.write("Content-Length: " + soapXml.length() + "\r\n");
wr.write("Content-Type: text/xml; charset=\"UTF-8\"\r\n");
//wr.write("Connection: Keep-Alive\r\n");
wr.write("SOAPAction: \"/someSoapAction\"\r\n");
wr.write("User-Agent: Apache-HttpClient/4.1.1 (java 1.5)\r\n");
wr.write("\r\n");
wr.write(soapXml);
wr.flush();
The requests are successful, meaning I get from the service the responses I expect.
For instance, when I put in my hand written "soapXml" some invalid parameters, I get a 500 error back with XML explaining the problem, If I set everything correctly I get a 200 OK with an xml body following the header.
The problem is that the socket hangs 60 seconds before reading the HTTP body in case of a 200OK.
It basically reads the full header, then waits 60 seconds, then (I think some protocol times out and) finally reads the xml body.
Here's the code with which I read the response:
String line;
BufferedReader rd = new BufferedReader(new java.io.InputStreamReader(sock.getInputStream(),"UTF-8"));
while ((line = rd.readLine()) != null) {
i++;
LOG.debug("cycle: "+i+" -------- "+line);
//after printing the header, it hangs 60 seconds before printing the follwing XML
if ((line.length() >0) && (line.charAt(0) == '<'))
{
responseXML = line;
}
}
Here is a sample 200 OK header:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/xml;charset=utf-8
Content-Length: 1052
Date: Sun, 13 Jan 2013 08:33:45 GMT -- hanging 60 seconds here
-- blank line
<?xml version="1.0" encoding="UTF-8"?>................</SOAP-ENV:Envelope>
Has anybody ever faced this? Please note this is not a server problem apparently, since with soap UI the answer is immediately provided.
Thank you

The server is keeping the connection open for HTTP keep-alive, clearly for a 60 second timeout. Either change to HTTP 1.0 or else take note of the returned Content-length header and stop reading at that many bytes.

The server environment change (a new machine was provided) solved the issue.
Now response time is below 1 sec also for 200OK.
Unluckily I don't have any details regarding the tech stack of the envrionment. Anyways thanks for help.

Http Post not posting data

I'm trying to post some data from a Java client using sockets. It talks to localhost running php code, that simply spits out the post params sent to it.
Here is Java Client:
public static void main(String[] args) throws Exception {
Socket socket = new Socket("localhost", 8888);
String reqStr = "testString";
String urlParameters = URLEncoder.encode("myparam="+reqStr, "UTF-8");
System.out.println("Params: " + urlParameters);
try {
Writer out = new OutputStreamWriter(socket.getOutputStream(), "UTF-8");
out.write("POST /post3.php HTTP/1.1\r\n");
out.write("Host: localhost:8888\r\n");
out.write("Content-Length: " + Integer.toString(urlParameters.getBytes().length) + "\r\n");
out.write("Content-Type: text/html\r\n\n");
out.write(urlParameters);
out.write("\r\n");
out.flush();
InputStream inputstream = socket.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
String string = null;
while ((string = bufferedreader.readLine()) != null) {
System.out.println("Received " + string);
}
} catch(Exception e) {
e.printStackTrace();
} finally {
socket.close();
}
}
This is how post3.php looks like:
<?php
$post = $_REQUEST;
echo print_r($post, true);
?>
I expect to see an array (myparams => "testString") as the response. But its not passing post args to server.
Here is output:
Received HTTP/1.1 200 OK
Received Date: Thu, 25 Aug 2011 20:25:56 GMT
Received Server: Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/0.9.8r DAV/2 PHP/5.3.6
Received X-Powered-By: PHP/5.3.6
Received Content-Length: 10
Received Content-Type: text/html
Received
Received Array
Received (
Received )
Just a FYI, this setup works for GET requests.
Any idea whats going on here?

As Jochen and chesles rightly point out, you are using the wrong Content-Type: header - it should indeed be application/x-www-form-urlencoded. However there are several other issues as well...
The last header should be seperated from the body by a blank line between the headers and the body. This should be a complete CRLF (\r\n), in your code it is just a new line (\n). This is an outright protocol violation and I'm a little surprised you haven't just got a 400 Bad Request back from the server, although Apache can be quite forgiving in this respect.
You should specify Connection: close to ensure that you are not left hanging around with open sockets, the server will close the connection as soon as the request is complete.
The final CRLF sequence is not required. PHP is intelligent enough to sort this out by itself, but other server languages and implementations may not be...
If you are working with any standardised protocol in it's raw state, you should always start by at least scanning over the RFC.
Also, please learn to secure your Apache installs...

It looks like you are trying to send data in application/x-www-form-urlencoded format, but you are setting the Content-Type to text/html.

Use
out.write("Content-Type: application/x-www-form-urlencoded\n\n");
instead. As this page states:
The Content-Length and Content-Type headers are critical because they tell the web server how many bytes of data to expect, and what kind, identified by a MIME type.
For sending form data, i.e. data in the format key=value&key2=value2 use application/x-www-form-urlencoded. It doesn't matter if the value contains HTML, XML, or other data; the server will interpret it for you and you'll be able to retrieve the data as usual in the $_POST or $_REQUEST arrays on the PHP end.
Alternatively, you can send your data as raw HTML, XML, etc. using the appropriate Content-Type header, but you then have to retrieve the data manually in PHP by reading the special file php://input:
<?php
echo file_get_contents("php://input");
?>
As an aside, if you're using this for anything sufficiently complex, I would strongly recommend the use of an HTTP client library like HTTPClient.