Develop Reference

Why is it important to read the response body of an HTTP request?

Learning about HTTP requests in Java. I'd like to know if reading the response body is essential to keeping a connection alive.
Here's an example code block (which posts a message to some URL):
private void writeToConnection(String url, String msg) throws IOException {
try {
HttpURLConnection connection = open(url);
// "Try with resources"
try (BufferedWriter writer = new BufferedWriter(
new OutputStreamWriter(connection.getOutputStream()))) {
writer.write(msg);
}
// Why do I need this line?
IOUtils.readStringFromStream(connection.getInputStream());
int code = connection.getResponseCode();
System.out.println(String.format("Returned response code %d.", code));
} catch (IOException e) {
e.printStackTrace();
}
}
Why is it necessary to read the input stream? The method readStringFromStream is returning a string but the string is not being assigned to anything. Does this ensure the connection stays alive? If so, how does the connection stay alive when the first line in the method opens a new connection? If the next batch of data to be written invokes this method, wouldn't that discard the old connection and open a new one?

I believe the intent of this code is indeed to consume the response body so that the connection can be reused. However, I'm not sure that this approach is correct; it is also likely to depend on the version of Java you are using.
First, it should suffice to get the connection's InputStream and close it; behind the scenes, the body still needs to be read, but closing the stream signals to the connection handler that the application wants to skip the body, and the handler can read and discard the content before putting the connection into a cache for re-use.
However, depending on the status, there could be an error stream instead of an input stream. Even in this case, the body needs to be consumed before the connection can be re-used, but many applications (like this one) don't bother reading the body of an error message. Since Java 7, however, if the error body is small enough, it will be consumed and buffered automatically.
Behind the scenes, a connection cache is used to retain open connections. Although the method names suggest a new connection is opened every time, in fact the cache is first checked for an open connection.

Nothing being sent to server via URLConnection

So I've created a program that communicates with both a FTP and SQL server. I run my program on my Windows machine and everything is working fine. But when I run the program on my Ubuntu VM it is not working. My code is as follows:
try {
URL url = new URL(location);
System.out.println("Created url");
URLConnection urlc = url.openConnection();
System.out.println("Created URLConnection");
urlc.setDoOutput(true);
System.out.println("Set Output");
OutputStreamWriter out = new OutputStreamWriter(urlc.getOutputStream());
System.out.println("Created OutputStreamWriter");
System.out.println("Hello World.");
out.write(Var.pressed);
System.out.println("Wrote String");
out.close();
System.out.println("Closed OutputStreamWriter");
} catch (Exception ex) {
ex.printStackTrace();
}
The program creates a file on my FTP and then writes to it. The file is being created but it is empty. No errors are ever thrown my program prints the following
Created url
Created URLConnection
Set Output
And then does nothing. It doesn't throw an error, it doesn't crash, it doesn't stop. The terminal is just left open. I also tried just running the .jar but nothing happens.
It seems as though the problem is created here:
OutputStreamWriter out = new OutputStreamWriter(urlc.getOutputStream());
Why is nothing being written to the file? And what is wrong with my program if no errors are ever thrown?

It does throw an error,
From everything else, I think you meant to say it doesn't throw an error. My answer is assuming that anyway.
First there are many reasons for a program to hang or wait and not produce an error. Sometimes if you wait long enough a system threshold will be passed and it will timeout even though your program would be happy to wait forever. Other times the system isn't paying attention either. Error conditions are sort of like a courtesy.
I suggest you should try to make the connection with the Ubuntu ftp line command and compare the same with Windows. FTP protocol does some unusual things with ports. The connection opens on one port, but the data transfer connection is switched over to another port. Firewalls can be tripped up with the change of port and prevent the data transfer. This also has created clients that do "passive" versus "active" FTP.

You need to at least get the response code, or the input stream. Otherwise not even a TCP connection is formed. And otherwise you have no idea whether the server accepted your input.
Your title was wildly inaccurate. Obviously you are creating the OutputStreamWriter. Your issue is that nothing is being sent. And the SQL tag was simply irrelevant.

Without an error message it is nearly impossible to figure out why it fails. A few suggestions, though.
First of all, try adding a System.err.println("test") This will prove that the output log will not only catch System.out AKA standard output stream, but also standard error stream.
I asssume, that in after this fix, the ex.printStackTrace() will be visible.
Otherwise: You do not specify how your location String is filled. Are you sure, you have write access to that directory?
If location is /tmp/foo/bar/myFile.txt, make sure the directory /tmp/foo/bar/ exists (mkdir /tmp/foo/bar/) and that the file is accessible (touch /tmp/foo/bar/myFile.txt).
If both report no error AND the first suggestion would bring up no error, the program might really lock up and wait for something from the outside. Difficult to say what at the moment.

Regarding data read from TCP connection

I am making a client socket connection with a hardware device. I am sending a command to this connection to be process by hardware. Now as a acknowledgment or as a reply, the hardware sends a response.
The application sends a command to the connection periodically say in 10 seconds.
Now there exists a problem randomly that the response won't gets synchronized with the sent command from the application. I was thinking of this as hardware specific but to my surprise, when I see the response by connecting putty to the same hardware at same port, I can see that response always gets synchronized. This looks like putty under the hood using some criteria to map the request to response.
Below is the programming steps that I am using to send a command to hardware device:-
Socket clientSocket = new Socket(<IPADDRESS>, 4001);
DataOutputStream outToServer = new DataOutputStream(
clientSocket.getOutputStream());
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(
clientSocket.getInputStream()));
while (true) {
try {
//Get command randomly from array enums for test
Random r = new Random();
Commands[] array = Commands.values();
String command = (String) array[r
.nextInt(Commands.values().length)].getCommand();
outToServer.writeBytes(command);
Thread.sleep(500);
while (!inFromServer.ready()) {
}
System.out.println("COMMAND "+command+", SERVER RESPONSE: "
+ inFromServer.readLine());
Thread.sleep(1500);
} catch (SocketTimeoutException se) {
//Handle Exception
} catch (SocketException se) {
//Handle Exception
}
Can anybody gives a advice how the synchronization of response with request can be achieved as mechanism like putty?

Putty doesn't know any more about your device than you do. The problem is in your code. Get rid of the ready() test and the sleep(). Just call readLine(), if you can be sure that the device sends lines, otherwise just call InputStream.read().

Remove the thread sleep, and rewrite read like this:
String line;
while ((line = inFromServer.readLine()) != null) {
System.out.println("COMMAND "+command+", SERVER RESPONSE: "
+ line);
}
This code can still hang, if the device sends the last message without the newline character \n. Your original code skipped the input.
The main problem is with this line:
while (!inFromServer.ready()) {
InputStreamReader#ready is OK to use only when you have other means to know that all the data has been sent:
Tells whether this stream is ready to be read. An InputStreamReader is ready if its input buffer is not empty, or if bytes are available to be read from the underlying byte stream.
The first message will get read, but that empties the buffer, and when the second message arrives your code isn't reading anymore. You would have to have as many loops as there are messages from device, and that's not practical, at least. And in that case also, it would probably not work all the time.
On the other hand the BufferedReader#readLine:
Returns:
A String containing the contents of the line, not including any line-termination characters, or null if the end of the stream has been reached
will read until all the data that was sent has been read. But if your device send no new line character, then this method will never read the line - the code will hang with all the data in the buffer. In that case you should use InputStreamReader#read as EJP suggested:
Returns:
The character read, or -1 if the end of the stream has been reached
I strongly suggest that you read the IO Streams official tutorial.
Generally speaking, waiting is not done by Thread.sleep and busy waiting (executing empty statements), e.g.:
while (true) {} /*or*/ while(true);
The CPU is executing the empty statement, and it could be doing some other work while waiting on this one to complete. It is a bad practice.
If you want to know more on how to implement waiting I recommend reading the official concurrency tutorial or this one for a broader approach on the matter.

My response seems to hang in response class while working with sockets

I am currently implementing a web proxy but i have run into a problem.I can parse my request from the browser and make a new request quite alright but i seem to have a problem with response.It keeps hanging inside my response loop
serveroutput.write(request.getFullRequest());
// serveroutput.newLine();
serveroutput.flush();
//serveroutput.
//serveroutput.close();
} catch (IOException e) {
System.out.println("Writting tothe server was unsuccesful");
e.printStackTrace();
}
System.out.println("Write was succesful...");
System.out.println("flushed.");
try {
System.out.println("Getting a response...");
response= new HttpResponse(serversocket.getInputStream());
} catch (IOException e) {
System.out.println("tried to read response from server but failed");
e.printStackTrace();
}
System.out.println("Response was succesfull");
//response code
public HttpResponse(InputStream input) {
busy=true;
reader = new BufferedReader(new InputStreamReader(input));
try {
while (!reader.ready());//wait for initialization.
String line;
while ((line = reader.readLine()) != null) {
fullResponse += "\r\n" + line;
}
reader.close();
fullResponse = "\r\n" + fullResponse.trim() + "\r\n\r\n";
} catch (IOException`` e) {
e.printStackTrace();
}
busy = false;
}

You're doing a blocking, synchronous read on a socket. Web servers don't close their connections after sending you a page (if HTTP/1.1 is specified) so it's going to sit there and block until the webserver times out the connection. To do this properly you would need to be looking for the Content-Length header and reading the appropriate amount of data when it gets to the body.
You really shouldn't be trying to re-invent the wheel and instead be using either the core Java provided HttpURLConnection or the Appache HttpClient to make your requests.

while (!reader.ready());
This line goes into an infinite loop, thrashing the CPU until the stream is available for read. Generally not a good idea.

You are making numerous mistakes here.
Using a spin loop calling ready() instead of just blocking in the subsequent read.
Using a Reader when you don't know that the data is text.
Not implementing the HTTP 1.1 protocol even slightly.
Instead of reviewing your code I suggest you review the HTTP 1.1 RFC. All you need to do to implement a naive proxy for HTTP 1.1 is the following:
Read one line from the client. This should be a CONNECT command naming the host you are to connect to. Read this with a DataInputStream, not a BufferedReader, and yes I know it's deprecated.
Connect to the target. If that succeeded, send an HTTP 200 back to the client. If it didn't, send whatever HTTP status is appropriate and close the client.
If you succeeded at (2), start two threads, one to copy all the data from the client to the target, as bytes, and the other to do the opposite.
When you get EOS reading one of those sockes, call shutdownOutput() on the other one.
If shutdownOutput() hasn't already been called on the input socket of this thread, just exit the thread.
If it has been called already, close both sockets and exit the thread.
Note that you don't have to parse anything except the CONNECT command; you don't have to worry about Content-length; you just have to transfer bytes and then EOS correctly.

Interrupting Java DataInputStream readFully()

I have a Java applet that streams video (MJPEG) from a server. I wrote a proxy server in C# (Windows service) to put between the applet and multiple video servers. A HTML/CSS/Js frontend is used along with the Java applet. All functionality works fine (finally!!!), except one thing.
The video server allows you to play back recorded video through a REST interface. When the clip is done, the server leaves the connection open in case you want to send it commands like rewind or seek. The clip is being played fine in the applet until the end. If you try to start a new clip (which entails sending a command from Javscript to the applet), the browser freezes up. However, subsequent commands that would use the same connection work, such as play, pause, and seek. If I stop the windows service, the browser becomes responsive again.
This is what I'm assuming is happening: The clip ends (or is paused); no more data is sent but the connection is still active. The applet is waiting on the proxy for the next frame, but the proxy is waiting on the video server for the next frame, which is not going to send any more data.
This is the code in a while loop that reads each frame
byte[] img = new byte[mContentLength];
inputStream.skipBytes(headerLen);
inputStream.readFully(img);
I need to interrupt this code somehow.
When a new video clip is selected in the HTML frontend, we notify the applet, which calls disconnect() on the CameraStream class. This is that function:
// DataInputStream inputStream
// HttpURLConnection conn
public void disconnect() {
System.out.println("disconnect called.");
if(running) {
running = false;
try {
// close the socket
if(inputStream != null) {
inputStream.close();
}
if(conn != null) {
conn.disconnect();
}
inputStream = null;
System.out.println("closed.");
} catch(Exception ignored) {
System.out.println("exc:" + ignored.getMessage());
main.reportErrorFromThrowable(ignored);
}
}
}
To test this, I let a quick clip play and run to the end. I then select a new clip. In my Java console, I get the output disconnect called. but I don't get the subsequent closed. message, nor does that generic Exception get caught. When I stop the Windows service, I finally get the closed. message, so it seems like inputStream.close(); is blocking.
So I guess my question is how can I stop the blocking? Is the readFully(img) call blocking? Or is it the disconnect function (as suggested by the console output I get)?
edit: just to clarify, I wrote the Java applet, HTML, CSS, Javascript, and C# proxy server, so I have access to all of that code. The only code I can't modify is that of the REST interface on the video server.
edit2: i meant to make bounty for this post https://stackoverflow.com/questions/12219758/proxy-design-pattern

In general, Java I/O methods block. The best solution appears to be to create another thread for reading the data and using NIO buffers. Example of NIO-based read (warning: untested!):
// get the InputStream from somewhere (a queue possibly)
ReadableByteChannel inChannel = Channels.newChannel(inputStream);
ByteBuffer buf = ByteBuffer.allocate(mContentLength + headerLen);
inChannel.read(buf);
byte[] img = new byte[mContentLength];
inChannel.get(img, headerLen, mContentLength);
This code creates a Channel from the InputStream and uses the Channel to read data. The JavaDoc for the ReadableByteChannel.read(ByteBuffer) function says that interrupting the thread that contains the call to inChannel.read(buf) will stop the read.
You will have to adapt this code, I just pulled it out of my head. Good luck!

I finally figured out the answer:
public void disconnect() {
if(running) {
running = false;
try {
try{
// had to add this
conn.getOutputStream().close();
}
catch(Exception exc){
}
// close the socket
if(inputStream != null) {
inputStream.close();
}
if(conn != null) {
conn.disconnect();
}
inputStream = null;
} catch(Exception ignored) {
main.reportErrorFromThrowable(ignored);
}
}
}
Even though I'm using an HttpUrlConnection, which is one way and doesn't have an output stream, trying to close the output stream raised an exception and for some reason made it all work.