Develop Reference

Java HTTP/1.1 GET request BufferedReader readLine never stops

Hello I'm making an HTTP client. I'm trying to fetch google.com's html code. I have a problem the the BufferedReader.readLine() function is blocking endlessly because the remote server apparently doesn't send a blank line? Or could it be that my request is wrong?
Appreciate any help!
public static void main(String[] args) {
String uri = "www.google.com";
int port = 80;
Socket socket = new Socket(uri, port);
PrintWriter toServer = new PrintWriter(socket.getOutputStream(), true);
InputStream inputStream = socket.getInputStream();
get(uri, port, language, socket, toServer, inputStream);
}
public static void get(String uri, int port, String language, Socket socket, PrintWriter toServer, InputStream inputStream) {
try {
toServer.println("GET / HTTP/1.1");
toServer.println("Host: " + uri + ":" + port);
toServer.println();
// Parse header
StringBuilder stringBuilder = new StringBuilder();
BufferedReader fromServer = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = fromServer.readLine()) != null) {
stringBuilder.append(line);
}
System.out.println("done");
} catch (IOException e) {
e.printStackTrace();
}
}

You are sending a HTTP/1.1 request which by default enables HTTP keep-alive. This means that the server might keep the TCP connection open after the response was sent in order to accept more requests from the client. Your code instead assumes that the server will close the connection after the response was finished by explicitly expecting readline to return null. But since the server will not close the connection (or only after some long timeout) the readline will just block.
To fix this either use HTTP/1.0 (which has keep-alive off by default) instead of HTTP/1.1 or explicitly tell the server that no more requests will be send by adding a Connection: close header.
Please note that in general HTTP is way more complex than you might think if you've just seen a few examples. The problem you face in your question is only a glimpse into more problems which you will face when continuing this path. If you really want to implement your own HTTP handling instead of using established libraries please study the actual standard instead of just assuming a specific behavior.

BufferedReader is not reading body of request

I am trying to read the data from an HttpPost, but when I read the data from BufferedReader I only get the header info. Please see the code below.
Here is the server
try {
ServerSocket server = new ServerSocket(8332);
System.out.println("Listening for connection on port 8332 ....");
while (true) {
try (Socket socket = server.accept()) {
BufferedReader buffer = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String request = "";
String line;
while ((line = buffer.readLine()) != null) {
System.out.println(line);
request += line;
}
System.out.print("port 8332 reading: " + request);
} catch (IOException e) {
System.out.print(e.getMessage());
}
}
} catch (IOException e){
System.out.print(e.getMessage());
}
Here is the Client
HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost("http://localhost:8332");
try {
StringEntity params =new StringEntity("details={\"name\":\"myname\",\"age\":\"20\"} ");
post.addHeader("content-type", "application/x-www-form-urlencoded");
post.setEntity(params);
client.execute(post);
} catch (IOException e){
System.out.println(e);
}
When I run this program I just get the following output
Listening for connection on port 8332 ....
POST / HTTP/1.1
content-type: application/x-www-form-urlencoded
Content-Length: 37
Host: localhost:8332
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.6 (Java/1.8.0_131)
Accept-Encoding: gzip,deflate
Upon debugging it seems like the program is not exiting this while loop
while ((line = buffer.readLine()) != null) {
System.out.println(line);
request += line;
}
But I can't figure out why. Please help I have been stuck all day.
Thanks in advance

But I can't figure out why.
The only way that your server will get a null from buffer.readLine() is if the client closes its socket output stream.
The problem is that the client side is trying to keep the connection alive ... which means that it won't close its socket output stream. That means that the server needs to respect the "content-length" header; i.e. count the number of bytes read rather than looking for an end-of-stream.
Fundamentally, your server side is not implementing the HTTP 1.1 specification correctly.
What to do?
Well my advice is to not try to implement HTTP starting from sockets. Use and existing framework ... or the Apache HttpComponents library.
But if you insist on doing it this way1, read the HTTP specification thoroughly before you start trying to implement it. And consult the spec whenever you run into problems with your implementation to check that you are doing the right thing.
1 - Definition: Masochism - the enjoyment of an activity that appears to be painful or tedious.

Use !=0 instead of !=null in your while loop:
while((line = buffer.readLine()).length() !=0) {
output:
port 8332 reading: POST / HTTP/1.1content-type:
application/x-www-form-urlencodedContent-Length: 18Host:
localhost:8332Connection: Keep-AliveUser-Agent:
Apache-HttpClient/4.5.3 (Java/1.8.0_171)Accept-Encoding:
gzip,deflateorg.apache.http.NoHttpResponseException: localhost:8332
failed to respond

Receiving Data On Sinatra Server

I am able to send data to a sinatra server (Strings) but I'm not sure about how to actually receive it in the sinatra code. Could anyone help?
Java client (what i'm using to send data) code:
private static void contactServer() {
try {
String text = "This is a text please work";
Socket sock = new Socket("localhost", 4567);
OutputStream os = sock.getOutputStream();
URL url = new URL("http://localhost:4567/hello");
PrintWriter writer = new PrintWriter(os);
writer.flush();
writer.write(text);
url.openStream();
System.out.println("done");
String strTemp = "";
/*while(null != (strTemp = br.readLine())){
System.out.println(strTemp);
}*/
} catch (Exception e) {
e.printStackTrace();
}
}

That's not going to work. Sinatra's server understands and speaks a language called the http protocol, so sending the string:
"This is a text please work"
through a socket to a Sinatra app is a hopeless prayer.
A protocol is a set of rules that specify how the client and the server will speak to each other--then each party can understand exactly what other party is saying. For the http protocol, clients send something known as a request, and servers reply with something known as a response. The request and the response must be formatted precisely according to the rules specified by the http protocol. The gory details for a request are here:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html
So your java program needs to send a request to Sinatra's server, which is simply a string formatted in a precise manner. Here is what a simple GET request looks like:
GET /path/to/page HTTP/1.1
Host: localhost:4567
If you want to make a GET request for the page:
http://localhost:4567/page1
(i.e. hit the Sinatra route get '/page1')
...then a simple GET request for that page would look like:
GET /page1 HTTP/1.1
Host: localhost:4567
Also, you must end every line in the http request with "\r\n" no matter what OS you are using. Those two characters are part of the http protocol. Furthermore, after the last header there must be a blank line signified by another "\r\n", like this:
GET /page1 HTTP/1.1\r\nHost: localhost:4567\r\n\r\n
Here is the java:
import java.io.*;
import java.net.*;
public class Sinatra {
private static void contactServer() {
try {
Socket sock = new Socket("localhost", 4567);
OutputStream os = sock.getOutputStream();
PrintWriter writer = new PrintWriter(os);
String[] text = {
"GET /page1 HTTP/1.1",
"Host: localhost:4567",
};
String request = "";
for(int i=0; i < text.length; ++i) {
request += text[i] + "\r\n";
}
request += "\r\n";
System.out.println(request);
writer.write(request);
writer.flush();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
Sinatra.contactServer();
}
}
Note: the first few times I ran that java program, the server(which I started with $ ruby myapp.rb) threw the following error:
[2013-08-19 20:10:11] ERROR Errno::ECONNRESET: Connection reset by peer
/Users/7stud/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/webrick/httpserver.rb:80:in `eof?'
/Users/7stud/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/webrick/httpserver.rb:80:in `run'
/Users/7stud/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread'
But when I ran the java program a few more times, the server eventually behaved. I have no idea what is causing that error. When I entered the url in FireFox, the server never threw that error. So I used Firebug to look at the request that Firefox sends; then I used all the same request headers in the java program, but the server still threw that error.
Edit: I can get the server error to go away by making the java program sleep for 1 second before closing the socket. The socket closes when you explicitly close the socket or when the program ends. Without the sleep, I think the socket closes while the server is still processing the request. Because a browser keeps the socket open, a browser never causes the server to throw that error.
The same server error also occurs with a ruby client:
require 'socket'
port = 4567
host = 'localhost'
s = TCPSocket.new host, port
req = [
"GET /page1 HTTP/1.1",
"Host: localhost:4567",
"Accept: */*",
]
req = req.join("\r\n") << ("\r\n" * 2)
print req
s.write req
s.flush
#sleep(1)
s.close
And the fix is the same. The only niggling detail is why the curl unix command doesn't cause the server to throw that error:
$ curl -v http://localhost:4567/page1
* About to connect() to localhost port 4567 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 4567 (#0)
> GET /page1 HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3
> Host: localhost:4567
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html;charset=utf-8
< Content-Length: 0
< X-Xss-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< Server: WEBrick/1.3.1 (Ruby/1.9.3/2012-04-20)
< Date: Tue, 20 Aug 2013 04:59:16 GMT
< Connection: Keep-Alive
<
* Connection #0 to host localhost left intact
* Closing connection #0
With the -v option, curl prints out the request and the response. Using curl to make the request, I never saw the sever throw that error. I wonder if curl does a sleep too?

Cannot output the stream to the browser

I just written a small program to test something, as below:
public class Main {
public static void main(String[] args){
ServerSocket serverSocket = null;
Socket clientSocket = null;
DataInputStream dataInputStream= null;
BufferedWriter bufferedWriter = null;
String line ;
try {
serverSocket = new ServerSocket(80);
clientSocket = serverSocket.accept();
dataInputStream = new DataInputStream(clientSocket.getInputStream());
while((line = dataInputStream.readLine()) != null){
System.out.println(line);
}
bufferedWriter = new BufferedWriter(new OutputStreamWriter(clientSocket.getOutputStream()));
bufferedWriter.write("HTTP/1.0 200 OK \n Date: Fri, 31 Dec 1999 23:59:59 GMT \n Content-Type: text/html \n Content-Length: 1354 \n <html>abcde<html/>");
bufferedWriter.flush();
} catch (IOException e) {
System.out.println("socket port cannot be opened.");
e.printStackTrace();
} finally{
try {
serverSocket.close();
bufferedWriter.close();
} catch (IOException e) {
System.out.println("socket port cannot be closed.");
e.printStackTrace();
}
}
}
}
I found the http response format from the internet, it should be correct. The problem is my browser keep on waiting for the response data (determine from the spinning logo), but the data is not returned successful. What mistake I have made?
I connect to the Java program by typing "localhost" in the browser, I can print out the request string in Java program but only fail to send back the response.

First things first, there are a few issues with the HTTP message being sent: Each line of an HTTP message is separated/ended by CR LF, rather than just a line feed (although I doubt this could be the problem, you should replace "\n" with "\r\n"). In addition, the Content-Length isn't equal to the real size of the message's body, which should be replaced. Before the actual body, all HTTP messages should have an empty line, which you have not. Finally, the forward slash at <html/> should also come before html, like this: </html>
To sum up:
bufferedWriter.write("HTTP/1.0 200 OK\r\nDate: Fri, 31 Dec 1999 23:59:59 GMT\r\nContent-Type: text/html\r\nContent-Length:18\r\n\r\n<html>abcde</html>");
Now for the real issue: the reading loop was constantly waiting for more data. Comparing the result of readLine() to null actually doesn't do what you were looking for. TCP connections make streams of data, and you never know if the client just stopped sending data in a particular moment. Instead, you can read until you find an empty line, which marks the end of the HTTP message's head. Web browsers usually do not send additional content in GET messages, therefore you will not be missing any data.
BufferedReader reader = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
while(!(line = reader.readLine()).isEmpty()){
System.out.println(line);
}

Could it be the Content-Length header field? Try replacing the response with this:
bufferedWriter.write("HTTP/1.0 200 OK \n Date: Fri, 31 Dec 1999 23:59:59 GMT \n Content-Type: text/html \n Content-Length: 18\n <html>abcde<html/>");
(Notice the Content-Length: 18 vs. the original Content-Length: 1354)
I suspect the browser is waiting for your application to send more data.
Edit: This works for me:
import java.util.*;
import java.io.*;
import java.net.*;
public class Main {
public static void main(String[] args) throws Exception{
ServerSocket serverSocket = null;
Socket clientSocket = null;
DataInputStream dataInputStream= null;
BufferedWriter bufferedWriter = null;
try {
serverSocket = new ServerSocket(8080);
clientSocket = serverSocket.accept();
BufferedReader reader = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
while (clientSocket.getInputStream().available() > 0) {
String line = reader.readLine();
if (line == null) {
break;
}
System.out.println(line);
}
bufferedWriter = new BufferedWriter(new OutputStreamWriter(clientSocket.getOutputStream()));
bufferedWriter.write("HTTP/1.0 200 OK \n Date: Fri, 31 Dec 1999 23:59:59 GMT \n Content-Type: text/html \n\n <html>abcde<html/>");
bufferedWriter.flush();
clientSocket.close();
} catch (IOException e) {
System.out.println("socket port cannot be opened.");
e.printStackTrace();
} finally{
try {
serverSocket.close();
bufferedWriter.close();
} catch (IOException e) {
System.out.println("socket port cannot be closed.");
e.printStackTrace();
}
}
}
}
The culprit was the while loop. The method you were calling to fetch more data was blocking until that data was available (which would never happen). I'm not sure whether or not what I did was completely necessary, but it works.

I see two problems:
The content-length header says 1354 bytes will follow. You only are writing 20 or so. Content-length is optional; don't include it if you're not using the correct value.
You need a blank line at the end of the headers and before the content: ".... Content-Length: xx\n\n<html>..."

fetching a webpage using java socket class

Good evening to all of you
I want to fetch a webpage using the socket class in java and i have done this as
import java.net.*;
import java.io.*;
class htmlPageFetch{
public static void main(String[] args){
try{
Socket s = new Socket("127.0.0.1", 80);
DataInputStream dIn = new DataInputStream(s.getInputStream());
DataOutputStream dOut = new DataOutputStream(s.getOutputStream());
dOut.write("GET /index.php HTTP/1.0\n\n".getBytes());
boolean more_data = true;
String str;
while(more_data){
str = dIn.readLine();
if(str==null)
more_data = false;
System.out.println(str);
}
}catch(IOException e){
}
}
}
But it is just giving the null's.
Output
HTTP/1.1 302 Found
Date: Wed, 01 Dec 2010 13:49:02 GMT
Server: Apache/2.2.11 (Unix) DAV/2 mod_ssl/2.2.11 OpenSSL/0.9.8k PHP/5.2.9 mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0
X-Powered-By: PHP/5.2.9
Location: http://localhost/xampp/
Content-Length: 0
Content-Type: text/html
null

I'm not sure if this is causing your problem, but HTTP expects carriage return and line feed for a newline:
dOut.write("GET /index.php HTTP/1.0\r\n\r\n".getBytes());
Also, it wouldn't hurt to flush and close your DataOutputStream:
dOut.flush();
dOut.close();
If you plan on doing anything more with this code than just connecting to simple test cases, I'd recommending using HttpURLConnection for this instead of implenting HTTP in a socket yourself. Otherwise, the result will contain more than just the web page. It will also contain the HTTP response including status codes and headers. Your code would need to parse that.
Update:
Looking at the response you added, that 302 response along with the Location: header indicate that the page you are looking for moved to http://localhost/xampp/ (see HTTP 302) and there is no longer any content at the original URL. This is something that is can be set to be handled automatically by HttpURLConnection or another library like Apache HttpClient. You will need to parse the status code, parse the headers, open a new socket to the response Location and get the page. Depending upon the exact requirements of your assignment, you will probably want to familiarize yourself with the HTTP 1.0 Specification, and the HTTP 1.1 Specification as well.

I think the code is working, except maybe that you don't see the output because it's swamped by all the nulls you print. You should stop the while after the first null.
More generally, DataInputStream and DataOutputStream are not the right classes for this job. Try this code.
public static void main(String[] args) throws IOException {
Socket s = new Socket("127.0.0.1", 80);
BufferedReader dIn = new BufferedReader(new InputStreamReader(s.getInputStream()));
PrintStream dOut = new PrintStream(s.getOutputStream());
dOut.println("GET /index.php HTTP/1.0");
dOut.println();
String str = null;
do {
str = dIn.readLine();
System.out.println(str);
} while (str != null);
}

Why are you using socket directly to perform HTTP connection? This is fine exercise but it requires deep knowledge of internals of HTTP protocol. Why not just to use classes URL, and URLConnection?
BufferedReader dIn = new BufferedReader(new URL("http://127.0.0.1:80").openConnection().getInputStream());
do {
str = dIn.readLine();
System.out.println(str);
} while (str != null);
}