Hello I'm making an HTTP client. I'm trying to fetch google.com's html code. I have a problem the the BufferedReader.readLine() function is blocking endlessly because the remote server apparently doesn't send a blank line? Or could it be that my request is wrong?
Appreciate any help!
public static void main(String[] args) {
String uri = "www.google.com";
int port = 80;
Socket socket = new Socket(uri, port);
PrintWriter toServer = new PrintWriter(socket.getOutputStream(), true);
InputStream inputStream = socket.getInputStream();
get(uri, port, language, socket, toServer, inputStream);
}
public static void get(String uri, int port, String language, Socket socket, PrintWriter toServer, InputStream inputStream) {
try {
toServer.println("GET / HTTP/1.1");
toServer.println("Host: " + uri + ":" + port);
toServer.println();
// Parse header
StringBuilder stringBuilder = new StringBuilder();
BufferedReader fromServer = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = fromServer.readLine()) != null) {
stringBuilder.append(line);
}
System.out.println("done");
} catch (IOException e) {
e.printStackTrace();
}
}
You are sending a HTTP/1.1 request which by default enables HTTP keep-alive. This means that the server might keep the TCP connection open after the response was sent in order to accept more requests from the client. Your code instead assumes that the server will close the connection after the response was finished by explicitly expecting readline to return null. But since the server will not close the connection (or only after some long timeout) the readline will just block.
To fix this either use HTTP/1.0 (which has keep-alive off by default) instead of HTTP/1.1 or explicitly tell the server that no more requests will be send by adding a Connection: close header.
Please note that in general HTTP is way more complex than you might think if you've just seen a few examples. The problem you face in your question is only a glimpse into more problems which you will face when continuing this path. If you really want to implement your own HTTP handling instead of using established libraries please study the actual standard instead of just assuming a specific behavior.
Related
This is a a Java method that tries to crawl a designated web page. I am using writeUTF and readUTF for socket communications to a server.
static void get_html(String host, String page, int port) throws IOException {
Socket sock = new Socket(host, port);
String msg = MessageFormat.format("GET {0} HTTP/1.1\r\nHost: {1}\r\n\r\n", page, host);
DataOutputStream outToServer = new DataOutputStream(sock.getOutputStream());
DataInputStream inFromServer = new DataInputStream(sock.getInputStream());
InputStream stream = new ByteArrayInputStream(msg.getBytes(StandardCharsets.UTF_8));
BufferedReader buf = new BufferedReader(new InputStreamReader(stream));
String outMsg;
while ((outMsg = buf.readLine()) != null) {
System.out.println("Sending message: " + outMsg);
outToServer.writeUTF(outMsg);
String inMsg;
try {
inMsg = inFromServer.readUTF();
} catch (EOFException eof) {
break;
}
System.out.println(inMsg);
}
sock.close();
}
The reason I am writing it this way was to mimic the c code, where you have a while loop of send() making all deliveries from a buffer, and another while loop of recv() from a buffer untill it hits 'null'. When execute my code, it just hangs there, I suspect that is due to a call of readUTF before I finished sending all my messages. If this is the case, is there any way to fix it?
You can't do this. HTTP is defined as text lines. writeUTF() does not write text, it writes a special format starting with a 16-bit binary length word. Similarly the HTTP server won't reply with that format into your readUTF() call. See the Javadoc.
You have to use binary streams and the write() method, with \r\n as the line terminator. Depending on the output format you may or may not be able to use readLine(). Best not, then you don't have to write two pieces of code: use binary streams again.
In fact you should throw it all away and use HttpURLConnection. Implementing HTTP is not as simple as may hastily be supposed.
I'm trying to write a httpserver using sockets and I meet this problem.
As everyone knows , a Http request is like this.
GET /index.html HTTP/1.1
Cache-Control: max-age=0
Host: 127.0.0.1
Accept:xxxxx
User-Agent: xxxx
Connection: keep-alive
CRLF
This is message body！
The question is how can I get full Http request including message body.
I tried to write like this.
ServerSocket serverSocket = new ServerSocket(8000);
while (true) {
Socket socket = serverSocket.accept();
new Thread() {
{
InputStream is = socket.getInputStream();
BufferedReader input = new BufferedReader(new InputStreamReader(is));
String line = null;
while ((line = input.readLine()) != null) {
System.out.println(line);
}
System.out.print("finish");
}
}.start();
}
And the console would never print "finish".Then I changed like this
ServerSocket serverSocket = new ServerSocket(8000);
while (true) {
Socket socket = serverSocket.accept();
new Thread() {
{
InputStream is = socket.getInputStream();
BufferedReader input = new BufferedReader(new InputStreamReader(is));
String line = null;
while (input.ready()) {
line = input.readLine();
System.out.println(line);
}
System.out.println("finish");
}
}.start();
}
Things go to be better, We can see "finish"! But if I refresh the page a little bit faster.The bufferdreader will not be ready and don't get in the while{} !
I want to print all the rerquest and "finish"
Please help me.
Thanks a lot!!
Both your approaches are incorrect.
In the first one, input.readLine() will return null only when the end of the stream has been reached, not when the request ended. That means that you'll loop there as long as the browser maintains the TCP connection open. That might take a while. Plus, multiple requests might be sent on the same connection, so you might end up printing all of them (I don't know if that's what you want to do).
In the second one, you have timing problem. input.ready() checks whether the receive buffer has any content to read, instead of checking whether the request ended. So you might end up printing only a part of the request instead of waiting for the whole thing. With this approach and the right timings, you might end up printing a part of a request, multiple requests, or anything in between (like a request and a half).
Also note that HTTP GET messages almost never carry any payalod, and no browser will requests like the one in your example.
I am wondering how to properly manage the following HTTP response considering the file I am sending to the client contains other linked files representing future HTTP requests.
I know that I can close the PrintWriter which will indicate to the client that the body is finished, but if I do that I don't see how I can receive subsequent requests for the linked pages within "first.html". I tried to include the content-length header but it seems I may have calculated the length incorrectly as any attempt to read from the input stream after sending "first.html" block/stall. Which tells me the client doesn't realize that the first.html file has finished sending. I've read over RFC 2616 but frankly have trouble understanding it without a proper example. I'm a real child when it comes to protocols, so any help would be appreciated!
public static void main(String[] args) throws Exception{
ServerSocket serverSocket = new ServerSocket(80);
Socket clientSocket = serverSocket.accept();
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
PrintWriter out = new PrintWriter(clientSocket.getOutputStream(),true);
String s;
while (!(s = in.readLine()).isEmpty()) {
System.out.println(s);
}
out.write("HTTP/1.0 200 OK\r\n");
out.write("Content-Type: text/html\r\n");
out.write("Content-Length: 1792\r\n");
out.write("\r\n");
File html = new File("/Users/tru/Documents/JavaScript/first.html");
BufferedReader htmlreader = new BufferedReader(new InputStreamReader(new FileInputStream(html)));
int c;
while((c = htmlreader.read()) > 0){
out.write(c);
}
}
I have created a server via sockets with the help of quickserver.org. The server runs solidly.
Now I had to write the client that sends a request (just a string value) to the server for an instruction and waits for its response (xml as string). This works fine when the triggered process by the request on server is not very time consuming. Unfortunately the client connection breaks as far as the server needs a long time for the process and that leads for a connection break and the client doesn't get anything back.
Here is the client code:
public String sendAndReceive(String message) throws IOException {
PrintWriter printWriter = new PrintWriter(new OutputStreamWriter(this.socket.getOutputStream()));
BufferedReader reader = new BufferedReader(new InputStreamReader(this.socket.getInputStream()));
printWriter.print(message);
printWriter.flush();
this.socket.shutdownOutput();
String line = null;
StringBuilder xmlResponse = new StringBuilder();
while ((line = reader.readLine()) != null)
{
xmlResponse.append(line);
}
printWriter.close();
reader.close();
this.socket.close();
return xmlResponse.toString();
}
This method sends the request and waits for the response afterwards. I am not sure about the while loop but all examples I have found on web are praising this construction. On my point of view reader.readline() can be null because the server needs more time for the response and therefore the method ends without getting the response.
How is the best practice for socket clients waiting for the response patiently? What I am doing wrong?
Kind regards,
Hilderich
You are probably getting timeout.
You can use Socket.setSoTimeout(int timeout) to change timeout (in milliseconds).
Okay this is a revised question from earlier today, I have included code to help explain the problem. I am sending two messages from the client to the server. The server then picks the messages up and processes them. The server finally attempts to send a message back to the client(please note in the server code "testmessage"), it is here I am having problems. Either I am not recieving the message at the client side or sending it incorrectly from the server side.
public class ClientConnection {
String address, language, message;
int portNumber;
Socket clientSocket = null;
public ClientConnection(String lan, String mes, String add, int pn) throws IOException{
address = add;
portNumber = pn;
language = lan;
message = mes;
}
public String createAndSend() throws IOException{
// Create and connect the socket
Socket clientSocket = null;
clientSocket = new Socket(address, portNumber);
PrintWriter pw = new PrintWriter(clientSocket.getOutputStream(),true);
BufferedReader br = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
// Send first message - Message is being correctly received
pw.write(language+"\n");
pw.flush();
// Send off the data
// Send the second message - Message is being correctly received
pw.write(message);
pw.flush();
pw.close();
// Send off the data
// NOTE: Either I am not receiving the message correctly or I am not sending it from the server properly.
String translatedMessage = br.readLine();
br.close();
//Log.d("application_name",translatedMessage); Trying to check the contents begin returned from the server.
return translatedMessage;
}
Server Code:
public class ServerConnection {
public static void main(String[] args) throws Exception {
// Delete - Using while loop to keep connection open permanently.
boolean status = false;
while( !status){
ServerSocket serverSocket = null;
try {
serverSocket = new ServerSocket(4444);
} catch (IOException e) {
System.err.println("Could not listen on port: 4444.");
System.exit(1);
}
Socket clientSocket = null;
try {
clientSocket = serverSocket.accept();
} catch (IOException e) {
System.err.println("Accept failed.");
System.exit(1);
}
// Delete - Working as of here, connection is established and program runs awaiting connection on 4444
BufferedReader br = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String language = br.readLine();
String message = br.readLine();
// Test - Works
System.out.println(language);
// Test - Works
System.out.println(message);
// Delete - Working as of here, both messages are passed and applied. Messages are received as sent from client.
TranslateMessage tm = new TranslateMessage();
String translatedMessage = tm.translateMessage(language, message);
// NOTE: This seems to be where I am going wrong, either I am not sending the message correctly or I am not receiving it correctly..
// PrintWriter writer = new PrintWriter(new BufferedOutputStream(clientSocket.getOutputStream()));
PrintWriter pw = new PrintWriter(clientSocket.getOutputStream(),true);
// Send translation back
System.out.println(translatedMessage);
// pw.write(translatedMessage+"\n");
pw.write("Return test"); // Test message!
pw.flush();
// Send off the data
pw.close();
br.close();
clientSocket.close();
serverSocket.close();
}
}
}
The code is a bit of a mess and I can see a few duplicates, I have commented where I feel the problems occour.
Thanks for any help!
You are using BufferedReader.readLine() to read the response from the server, but in the test case you are sending a string that is not terminated with a \n or \r\n, so it will not get the line as far as I can tell from the docs...
public String readLine()
throws IOException
Read a line of text. A line is considered to be terminated by any one of a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed.
Returns:
A String containing the contents of the line, not including any line-termination characters, or null if the end of the stream has been reached
An additional suggestion...
When writing request response protocols like this I would not rely on line endings to terminate the requests or responses. Typically I would use either a fully formatted JSON string, or my preference is for a binary protocol where all requests and response are prepended with a binary count (usually 4 bytes bigendian/network byte order). Then the client and server reads the 4 bytes then reads the number of bytes that follow. This handles the packet fragmentation that typically happens over network connections, also it helps avoid DOS attacks by malicious users sending long strings that never terminate.
In Java you can use ByteBuffer.order() to handle bigendian numbers.