Develop Reference

Why is my GzipOutputStream starting with 10 bytes and never changing? [duplicate]

I'm trying to send a compressed HTML file through a java socket but the browser displays an empty HTML file.
The thing is, when I try to send the uncompressed HTML everything works find (yes I do modify the HTTP headers accordingly).
private void sendResponse(String headers, String body) throws IOException
{
BufferedOutputStream output = new BufferedOutputStream(
this.SOCKET.getOutputStream());
byte[] byteBody = null;
// GZIP compression
if(body != null && this.encoding.contains("gzip"))
{
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
GZIPOutputStream zipStream = new GZIPOutputStream(byteStream);
zipStream.write(body.getBytes(this.charset));
zipStream.flush();
byteBody = byteStream.toByteArray();
byteStream.flush();
byteStream.close();
zipStream.close();
}
else
byteBody = body.getBytes(this.charset);
// Sending response
byte[] msg1 = (Integer.toHexString(byteBody.length) + "\r\n")
.getBytes(this.charset);
byte[] msg2 = byteBody;
byte[] msg3 = ("\r\n" + "0").getBytes(this.charset);
output.write(headers.getBytes(this.charset));
output.write(msg1);
output.write(msg2);
output.write(msg3);
output.flush();
output.close();
}
Basically, headers contains the HTTP headers, and body the HTML file. The rest seems self explanatory. What could cause such a behavior?
EDIT: The header is generated as such:
headers = "HTTP/1.1 200 OK\r\n";
headers += "Date: " + WebServer.getServerTime(Calendar.getInstance()) + "\r\n";
headers += "Content-Type: text/html; charset=" + this.charset + "\r\n";
headers += "Set-Cookie: sessionID=" + newCookie + "; Max-Age=600\r\n";
headers += "Connection: close \r\n";
if(this.encoding.contains("gzip"))
headers += "Content-Encoding: gzip\r\n";
headers += "Transfer-Encoding: chunked \r\n";
headers += "\r\n";

The problem is that a GZIPOutputStream isn't complete until the finish() method has been called.
It is automatically called when you close() the stream.
Since you are calling byteStream.toByteArray() before that has happened, you're not getting complete data.
Also, you don't need to call flush() since that is also automatically done when you call close(). And closing the GZIPOutputStream automatically closes the underlying stream(s) (i.e. the ByteArrayOutputStream).
So, you code should be:
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
GZIPOutputStream zipStream = new GZIPOutputStream(byteStream);
zipStream.write(body.getBytes(this.charset));
zipStream.close();
byteBody = byteStream.toByteArray();

A simple test class should show you where the problem is:
import java.io.ByteArrayOutputStream;
import java.util.zip.GZIPOutputStream;
public class GZipTest {
public final static void main(String[] args) throws Exception {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream gzos = new GZIPOutputStream(baos);
gzos.write("some data".getBytes());
System.out.println("baos before gzip flush: " + baos.size());
gzos.flush();
System.out.println("baos after gzip flush: " + baos.size());
gzos.close();
System.out.println("baos after gzip close: " + baos.size());
}
}
This leads to the following output:
baos before gzip flush: 10
baos after gzip flush: 10
baos after gzip close: 29
You're closing your GZIPOutputStream after building you body data, so the browser receives incomplete GZIP-data and therefor can't decompress it.

How to write a file to an HTTP Request Java

I am new to using HTTP and I have questions about writing a file and another value to an HTTP Post request in Java. I am using an public API provided by a company called Mojang to write what is known as a "skin" (a png file) to the game Minecraft for player character modles. Here is the documentation of how to use this public API for reference:https://wiki.vg/Mojang_API#Upload_Skin
Here is the code I have written. When ran I get the 415 HTTP Response code (which I assume is "unsupported media type"). Any suggestions on what I am doing wrong and how I can fix this? I found other stack overflow issues for uploading files but I need to also add a value called "variant={classic or slim}". I am a little lost on how to make all of this work. Any help is much appreciated.
(I could not get the code to properally format in the code sample using ' ', it is in a javascript snippet)
public static void uploadSkin(String accessToken, String variant, File file) throws IOException {
URL url = new URL("https://api.minecraftservices.com/minecraft/profile/skins");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setDoOutput(true);
con.setRequestMethod("POST");
con.setRequestProperty("Authorization", "Bearer " + accessToken); // The access token is provided after an
// authentication request has been send, I
// have done this sucessfully in another
// method and am passing it in here
con.addRequestProperty("variant", variant);
OutputStream outputStream = con.getOutputStream();
PrintWriter writer = new PrintWriter(new OutputStreamWriter(con.getOutputStream(), "utf-8"), true);
String boundary = "===" + System.currentTimeMillis() + "===";
String fileName = file.getName();
String LINE_FEED = "\r\n";
String fieldName = "file";
writer.append("--" + boundary).append(LINE_FEED);
writer.append("Content-Disposition: form-data; name=\"" + fieldName + "\"; filename=\"" + fileName + "\"")
.append(LINE_FEED);
writer.append("Content-Type: " + URLConnection.guessContentTypeFromName(fileName)).append(LINE_FEED);
writer.append("Content-Transfer-Encoding: binary").append(LINE_FEED);
writer.append(LINE_FEED);
writer.flush();
FileInputStream inputStream = new FileInputStream(file);
byte[] buffer = new byte[4096];
int bytesRead = -1;
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.flush();
inputStream.close();
writer.append(LINE_FEED);
writer.flush();
}

Alright, found a solution to the problem. Using this maven dependency:
<!-- https://mvnrepository.com/artifact/org.jodd/jodd-http -->
<dependency>
<groupId>org.jodd</groupId>
<artifactId>jodd-http</artifactId>
<version>5.0.2</version>
</dependency>
And then this:
HttpResponse response = HttpRequest.post("https://api.minecraftservices.com/minecraft/profile/skins")
.header("Authorization", "Bearer " + accessToken).header("Content-Type", "multipart/form-data")
.form("variant", variant).form("file", file).send();
I was able to get it to work. Hope this is helpful to anyone that needs to upload a Skin Png file to Minecraft.

gzip/deflate compression with Server Sent Events on tomcat servlet

I am using server sent events for streaming responses (text/event-stream). We want to compress the response using gzip or deflate compression, but the browser shows : ERR_CONTENT_DECODING_FAILED.
Code for the same -
response.setContentType("text/event-stream; charset=UTF-8");
response.addHeader("Connection", "keep-alive");
response.addHeader("Cache-Control", "no-cache, must-revalidate");
response.addHeader("Content-Encoding", "deflate");
PrintWriter writer = response.getWriter();
number = 10;
time = 100;
for (int i = 0; i < number; i++) {
String resp = "data: " + "Some Response" + "\r\n";
Deflater deflater = new Deflater(Deflater.DEFLATED);
byte[] input = resp.getBytes("UTF-8");
deflater.setInput(input);
deflater.finish();
byte[] output = new byte[1024];
deflater.deflate(output);
deflater.end();
writer.write(new String(output, "UTF-8"));
writer.flush();
try {
Thread.sleep(time);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
String resp = "data: close\r\n";
Deflater deflater = new Deflater(Deflater.DEFLATED);
byte[] input = resp.getBytes("UTF-8");
deflater.setInput(input);
deflater.finish();
byte[] output = new byte[1024];
deflater.deflate(output);
deflater.end();
writer.write(new String(output, "UTF-8"));
writer.flush();

Compressed stream is binary data. It must not be printed with response.getWriter(). Use response.getOutputStream() instead.
UPDATE
Shiva Bhalla: Using the response.getOutputStream() with text/event-stream isn't doing proper streaming after the 1st chunk of response is displayed, the request is being failed at the browser.
You have to compress the whole stream using a single Deflater instance.
In your code you compress each fragment separately. This is equivalent to the following:
You take a series of text files (01.txt, 02.txt, ...).
You compress each of them into a gzip archive. (01.txt.gz, 02.txt.gz, ...)
You concatenate the archives into a single file.
The above produces an incorrect archive. Gzip files cannot be concatenated like that. The correct code should do the following:
You take a series of text files (01.txt, 02.txt, ...).
You concatenate the text files into a single file (text.txt).
You compress the file into a gzip archive. (text.txt.gz)
I recommend you to use a java.util.zip.DeflaterOutputStream (1) instead of direct use of a Deflater. E.g.
new DeflaterOutputStream(response.getOutputStream(), true);
Beware of introducing a BREACH vulnerability.

Adding "text/event-stream;charset=UTF-8,ms-stream;charset=UTF-8" as "compressableMimeType" in connector properties did the job!

Socket HTTP request returning invalid GZIP

I am teaching myself more about HTTP requests and such, so I wrote a simple POST request using Java's HttpURLConnection class and it returns compressed data which is easily decompress. I decided to go a lower level and send the HTTP request with sockets (for practice). I figured it out after a series of google searches, but there is one issue. When the server respondes with compressed data it isn't valid. Here is an image of a bit of debugging.
http://i.imgur.com/KfAcero.png
The portion below the "=" separator line is the response when using a HttpURLConnection instance, but the portion above it is the response when using sockets. I'm not too sure what is going on here. The bottom part is valid, while the top is not.
The HttpParameter and header classes simply store a key and value.
public String sendPost(String host, String path, List<HttpParameter> parameters, List<HttpHeader> headers) throws UnknownHostException, IOException {
String data = this.encodeParameters(parameters);
Socket socket = new Socket(host, 80);
PrintWriter writer = new PrintWriter(socket.getOutputStream());
BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
writer.println("POST " + path + " HTTP/1.1");
for(HttpHeader header : headers) {
writer.println(header.getField() + ": " + header.getValue());
}
writer.println();
writer.println(data);
writer.flush();
StringBuilder contentBuilder = new StringBuilder();
for(String line; (line = reader.readLine()) != null;) {
contentBuilder.append(line + "\n");
}
reader.close();
writer.close();
return contentBuilder.toString();
}

Your problem is that you are using Readers and Writers for something that is not text.
InputStream and OutputStream work with bytes; Reader and Writer work with encoded text. If you try to use Reader and Writer with something that is not encoded text, you will mangle it.
Sending the request with a Writer is fine.
You want to do something like this instead:
InputStream in = socket.getInputStream();
// ...
ByteArrayOutputStream contentBuilder = new ByteArrayOutputStream();
byte[] buffer = new byte[32768]; // the size of this doesn't matter too much
int num_read;
while(true) {
num_read = in.read(buffer);
if(num_read < 0)
break;
contentBuilder.write(buffer, 0, num_read);
}
in.close();
writer.close();
return contentBuilder.toByteArray();
and make sendPost return a byte array.

Sending images over http to browser

I would like to implement a simple web server in java.
The problem is that images are not correctly rendered on the web browser; all I can see, if I go to localhost:8888/image.png, is a white square with exact width, height and weight.
Thank you in advance! :)
Here is the code:
public Http(Socket server) throws IOException {
in = new BufferedReader(new InputStreamReader(server.getInputStream()));
parseHeader(in);
String response = new String();
out = new PrintWriter(server.getOutputStream(), true);
Files f = new Files(getHomePath() + httpRequestedPage);
if(!f.exists) {
// 404 ERROR
} else {
response += "HTTP/1.1 200 OK\r\n";
response += "Date: " + nowDate + "\r\n";
response += "Content-Type: image/png\r\n";
response += "Content-Length: " + res.length() + "\r\n";
response += "Connection: keep-alive\r\n";
response += "\r\n";
response += IOUtils.toString(new FileInputStream(getHomePath() + httpRequestedPage));
}
out.println(response);
in.close();
out.close();
}
EDIT:
Unfortunately it returns the same message.
out = new PrintWriter(server.getOutputStream(), true);
OutputStream out2 = server.getOutputStream();
File file = new File(HttpServer.getHomePath() + httpRequestedPage);
InputStream stream = new FileInputStream(file);
String response = new String();
response += "HTTP/1.1 200 OK\r\n";
response += "Date: " + nowDate + "\r\n";
response += "Content-Type: image/png\r\n";
response += "Content-Length: " + file.length() + "\r\n";
response += "Connection: keep-alive\r\n";
response += "\r\n";
out.println(response);
IOUtils.copy(stream, out2);
out.close();
out2.close();

You are using Write class for rendering the image. Use the OutputStream to write the image. Images are bytes and always byte based streams should be used to render them.

If you are converting bytes into String then you must use Base64 encoding. And on the client side you can specify the image src similar to this "data:image/png;base64," + imageData.