Develop Reference

Multipart/form-data request with multiple files: request seems to be garbled due to odd byte sequence

We need to send a multipart/form-data request containing two files. Since we had some issues with the requests, we set up a request bin page to try and analyze the requests themselves, since Postman seemed to work fine.
My second file is generated by a different service as a byte stream and represents a JSON object. This byte stream seems to have a NUL (0) byte terminator at the end.
If I add this second file to the multipart/form-data request, something like the following results:
With the body looking as follows:
--9ac41cec-a0ff-4afd-b687-6b8c2f1bddd6.
Content-Disposition: form-data; name="metadata"; filename="metadata-1584981615.json".
Content-Type: application/json.
Content-Length: 581.
.
{"...":"..."}.
--9ac41cec-a0ff-4afd-b687-6b8c2f1bddd6.
Content-Disposition: form-data; name="file"; filename="file-1584981615.json".
Content-Type: application/json.
Content-Length: 12205.
.
{"...":"..."}..
--9ac41cec-a0ff-4afd-b687-6b8c2f1bddd6--.
Obviously, this is supremely invalid. It seems that the multipart request is adding the . characters at the end of every line.
I was able to reproduce this behaviour with Spring Web 5.1.5.RELEASE, okHttp 3.14.7 and Apache HttpComponents 4.5.7. The behaviour does not seem to change if I remove the NUL (0) byte terminator from the byte stream.
Here's the odd part: the receiving webserver accepts this request as a valid multipart/form-data request.
For the record, here's what a valid request (that is also accepted by the webserver) looks like from request bin's point of view (different file):
--83611e7f-ad57-4893-9d1b-5ba2c4543d2a
Content-Disposition: form-data; name="metadata"; filename="metadata-1584982345.json"
Content-Type: application/json
Content-Length: 895
{"...": "..."}
--83611e7f-ad57-4893-9d1b-5ba2c4543d2a
Content-Disposition: form-data; name="file"; filename="file-1584982345.json"
Content-Type: application/json
Content-Length: 9868
{"...": "..."}
--83611e7f-ad57-4893-9d1b-5ba2c4543d2a--
Aside from having the explicit terminator at the end (which does not seem to make a difference at all), the byte stream does not look broken. new String(bytes) gives a valid JSON object. So I'm thinking it has to be something else.
Has anyone encountered this issue before? I'm unsure how to proceed with debugging this issue, and I'm fairly certain that sending improper requests, even though they might be accepted by the remote server at this time, almost certainly guarantees that my code will break somewhere along the line...

Sending Query Parameters encoded in a URI on a Post which also has json

I have a server that handles a POST request with JSON. It also looks and decodes query parameters from the URI. My Java Client currently uses HTTPPost to send across the Json with ContentType application/json.
I wonder whehther URLEncodeUtil method format would be able to accomplish this. Except the documentation mentions
suitable for use as an application/x-www-form-urlencoded list of
parameters in an HTTP PUT or HTTP POST.
So my question is
1. Would this work with ContenType set to application/json.
2. Is there another way to accomplish what the Server requires, ie: have JSON as well as Query parameters encoded in the URI.

There are two official methods of posting form data via the (HTML spec). The pertinent value is application/x-www-form-urlencoded which adds a ? along with the name/value pairs encoded in the URL. If the form method is POST then it will be the first line after the HTTP POST statement.
Everything we do with HTTP in REST web services is valid HTTP, but not for HTML. So the application/json can have a combination of the application/x-www-form-urlencoded style parameters and the JSON payload.
The HTTP request will look something like this:
POST /blog/posts?myparam=Something%20Good&token=donotdothis
Accept: application/json
Content-Type: application/json
Content-Length: 57
{"title":"Hello World!","body":"This is my first post!"}
Also spelled out here: http://www.jsonrpc.org/historical/json-rpc-over-http.html
It's the ? that marks the beginning of extra parameters. So while that is technically legal, it does beg the question why everything you need to post can't be part of your JSON. The downside of this approach is that the query parameters are all part of your HTTP logs and is very visible. You definitely should not use this approach with passwords or any other personally identifiable information. Depending on privacy laws in your country, you want to minimize unnecessary records to make compliance much easier.

What is the format of a HTTP PUT request?

I am currently working on a simple java program that tests ping, upload speed, and download speed. It is for a class project, and therefore rather than using java's httpURLConnection package, I'm doing it manually using Java Sockets. I found it quite easy to find the format of a GET request, put cannot find the format of a PUT.
Here, for example, is how my GET request looks:
//Create GET request
outToServer.writeBytes("GET / HTTP/1.1\r\n
+ "Host: " + server + "\r\n"
+ "Connection: close\r\n\r\n");
where outToServer is a DataOutputStream using the outputStream of a Socket. I am looking for a similar way to perform a PUT to upload a file to a server and measure the time.
Thank you very much for any help!

In short, format is the same, it has "PUT" instead of "GET" and it is usually used to "add" or "update" entities in the server, so after the header you would get an empty line (\r\n\r\n) and then a body, whose content type would correspond to the header "content-type" (like, say, text/html or application/json) An example of what a PUT request would look like:
PUT /boo/foo.txt HTTP/1.1
Host: www.foo.com
Content-Type: plain/text
This is a testing content for the text file foo.txt
Now, the server must support the method PUT (it is not required for a server to support such method)

Java HttpURLConnection | POST method | Sequence of HTTP header fields

I am trying to "spoof" a Firefox HTTP POST request in Java using java.net.HttpURLConnection.
I use Wireshark to check the HTTP headers being sent, so I have (hopefully) reliable source of information, why the Java result doesn't match the ideal situation (using Firefox).
I have set all header fields exactly to the values that Firefox sends via HTTP and noticed, that the sequence of the header fields is not the same.
The output for Firefox is like:
POST ...
**Host**
User-Agent
Accept
Accept-Language
Accept-Encoding
Referer
Connection
Content-Type
Content-Length
When I let wireshark tap off my implementation in Java, it gives me a slightly different sequence of fields:
POST...
**User-Agent**
Accept
Accept-Language
Accept-Encoding
Referer
Content-Type
Host
Connection
Content-Length
So basically, I have all the fields, just in a different order.
I have also noticed that the Host field is sent with a different value:
www.thewebsite.com (Firefox) <---> thewebsite.com (Java HttpURLConnection), although I pass on the String to httpUrlConnection.setRequestProperty with the "www."
I have not yet analyzed the byte output of Wireshark, but I know that the server is not returning the same Location in the header fields of my response.
My questions are:
(1) Is is possible to control the sequence the header fields in the request, and if yes is it possible to do using HttpURLConnection? If not, is it possible to directly control the bytes in the HTTP header using Java? [I don't own the server, so my only hope to get the POST method working is through my application pretending to be Firefox, the server is not really verbose, my only info are: Apache with PHP]
(2) Is there a way to fix the setRequestProperty() problem ("www") as described above?
(3) What else could matter? (Do I need to concern the underlying layers, TCP....?)
Thanks for any comments.
PS. I am trying to model a situation without cookies being sent, so that I can ignore the effect.

First, the order of the headers is irrelevant.
Second, in order to manually override the host header you need to set sun.net.http.allowRestrictedHeaders=true either in code
System.setProperty("sun.net.http.allowRestrictedHeaders", "true")
or at JVM start
-Dsun.net.http.allowRestrictedHeaders=true
This is a security precaution introduced by Oracle a while ago. That's because according to RFC
The Host request-header field specifies the Internet host and port
number of the resource being requested, as obtained from the original
URI given by the user or referring resource (generally an HTTP URL).

the headers order is not important. the headers got by server are also out-of-order. And you can not control httpUrlConnection header order. But if you write your own TCP client, you can control your header order. like:
clientSocket = new Socket(serverHost, serverPort);
OutputStream os = clientSocket.getOutputStream();
String send = "GET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\nGET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\n";
os.write(send.getBytes());
The Second question is answered by Marcel Stör in the first answer.
a

I got lucky with Apache Http Components, my guess is that the "Host" header's missing "www." made the difference, which can be set exactly as intended using Apache's HttpPost:
httpPost.setHeader("Host", "www.thewebsite.com");
The Wireshark output confirmed my suspicion. Also this time the TCP communication prior to my HTTP post looks different (client ---> server, server ---> client, client ---> server) instead of (client ---> server, server ---> client, client ---> server, client---> server).
Now I get the desired Location header value and the server is also setting the cookies. :)
For the most part, this question is resolved.
Actually I wanted to use the lightweihgt HttpUrlConnection because that's what the Android Developers blog suggesting. The System.setProperty("sun.net.http.allowRestrictedHeaders", "true") might work as well, if it allows to "www." in the Host value.

How can I send an HTTP Response using only standard network libraries?

I'm working on my first homework project in a web programming class, which is to write a simple web server in Java. I'm at the point where I have data being transmitted back and forth, and to the untrained eye, my baby server seems to be working fine. However, I can't find a way to send appropriate responses. (In other words, an invalid page request would show a 404-ish HTML page, but it still returns a 200 OK status when I view response headers).
I'm limited to being able to use standard network libraries for socket management and standard I/O libraries to read and write bytes and strings from an input stream. Here's some pertinent code:
From my main...
ServerSocket servSocket = new ServerSocket(port, 10); // Bind the socket to the port
System.out.println("Opened port " + port + " successfully!");
while(true) {
//Accept the incoming socket, which means that the server process will
//wait until the client connects, then prepare to handle client commands
Socket newDataSocket = servSocket.accept();
System.out.println("Client socket created and connected to server socket...");
handleClient(newDataSocket); //Call handleClient method
}
From the handleClient method...(inside a loop that parses the request method and path)
if(checkURL.compareTo("/status") == 0) { // Check to see if status page has been requested
System.out.println("STATUS PAGE"); // TEMPORARY. JUST TO MAKE SURE WE ARE PROPERLY ACCESSING STATUS PAGE
sendFile("/status.html", dataStream);
}
else {
sendFile(checkURL, dataStream); // If not status, just try the input as a file name
}
From sendFile method...
File f = new File(where); // Create the file object
if(f.exists() == true) { // Test if the file even exists so we can handle a 404 if not.
DataInputStream din;
try {
din = new DataInputStream(new FileInputStream(f));
int len = (int) f.length(); // Gets length of file in bytes
byte[] buf = new byte[len];
din.readFully(buf);
writer.write("HTTP/1.1 200 OK\r\n"); // Return status code for OK (200)
writer.write("Content-Length: " + len + "\r\n"); // WAS WRITING TO THE WRONG STREAM BEFORE!
writer.write("Content-Type: "+type+"\r\n\r\n\r\n"); // TODO VERIFY NEW CONTENT-TYPE CODE
out.write(buf); // Writes the FILE contents to the client
out.flush();
out.close();
} catch (FileNotFoundException e) {
e.printStackTrace(); // Not really handled since that's not part of project spec, strictly for debug.
}
}
else {
writer.write("HTTP/1.1 404 Not Found\r\n"); // Attempting to handle 404 as simple as possible.
writer.write("Content-Type: text/html\r\n\r\n\r\n");
sendFile("/404.html", sock);
}
Can anybody explain how, in the conditional from sendFile, I can change the response in the 404 block (Like I said before, the response headers still show 200 OK)? This is bugging the crap out of me, and I just want to use the HTTPResponse class but I can't. (Also, content length and type aren't displayed if f.exists == true.)
Thanks!

Edit It looks to me like in the 404 situation, you're sending something like this:
HTTP/1.1 404 Not Found
Content-Type: text/html
HTTP/1.1 200 OK
Content-Length: 1234
Content-Type: text/html
...followed by the 404 page. Note the 200 line following the 404. This is because your 404 handling is calling sendFile, which is outputting the 200 response status code. This is probably confusing the receiver.
Old answer that missed that:
An HTTP response starts with a status line followed (optionally) by a series of headers, and then (optionally) includes a response body. The status line and headers are just lines in a defined format, like (to pick a random example):
HTTP/1.0 404 Not Found
To implement your small HTTP server, I'd recommend having a read through the spec and seeing what the responses should look like. It's a bit of a conceptual leap, but they really are just lines of text returned according to an agreed format. (Well, it was a conceptual leap for me some years back, anyway. I was used to environments that over-complicated things.)
It can also be helpful to do things like this from your favorite command line:
telnet www.google.com 80
GET /thispagewontbefound
...and press Enter. You'll get something like this:
HTTP/1.0 404 Not Found
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
Date: Sun, 12 Sep 2010 23:01:14 GMT
Server: sffe
Content-Length: 1361
X-XSS-Protection: 1; mode=block
...followed by some HTML to provide a friendly 404 page. The first line above is the status line, the rest are headers. There's a blank line between the status line/headers and the first line of content (e.g., the page).

The problem you are seeing is most likely related to a missing flush() on your writer. Depending on which type of Writer you use the bytes are first written to a buffer that needs to be flushed to the stream. This would explain why Content-Length and Content-Type are missing in the output. Just flush it before you write additional data to the stream.
Further you call sendFile("/404.html", sock);. You did not post the full method here - but I suppose that you call it recursively inside sendFile and thus send the 200 OK status for your file /404.html.

Based on your reported symptoms, I think the real problem is that you are not actually talking to your server at all! The evidence is that 1) you cannot get a 404 response, and 2) a 200 response does not have the content length and type. Neither of these should be possible ... if you are really talking to the code listed above.
Maybe:
you are talking to an older version of your code; i.e. something is going wrong in your build / deploy cycle,
you are (mistakenly) trying to deploy / run your code in a web container (Jetty, Tomcat, etc), or
your client code / browser is actually talking to a different server due to proxying, an incorrect URL, or something like that.
I suggest that you add some trace printing / logging at appropriate points of your code to confirm that it is actually being invoked.