Develop Reference

Java HTTP/1.1 GET request BufferedReader readLine never stops

Hello I'm making an HTTP client. I'm trying to fetch google.com's html code. I have a problem the the BufferedReader.readLine() function is blocking endlessly because the remote server apparently doesn't send a blank line? Or could it be that my request is wrong?
Appreciate any help!
public static void main(String[] args) {
String uri = "www.google.com";
int port = 80;
Socket socket = new Socket(uri, port);
PrintWriter toServer = new PrintWriter(socket.getOutputStream(), true);
InputStream inputStream = socket.getInputStream();
get(uri, port, language, socket, toServer, inputStream);
}
public static void get(String uri, int port, String language, Socket socket, PrintWriter toServer, InputStream inputStream) {
try {
toServer.println("GET / HTTP/1.1");
toServer.println("Host: " + uri + ":" + port);
toServer.println();
// Parse header
StringBuilder stringBuilder = new StringBuilder();
BufferedReader fromServer = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = fromServer.readLine()) != null) {
stringBuilder.append(line);
}
System.out.println("done");
} catch (IOException e) {
e.printStackTrace();
}
}

You are sending a HTTP/1.1 request which by default enables HTTP keep-alive. This means that the server might keep the TCP connection open after the response was sent in order to accept more requests from the client. Your code instead assumes that the server will close the connection after the response was finished by explicitly expecting readline to return null. But since the server will not close the connection (or only after some long timeout) the readline will just block.
To fix this either use HTTP/1.0 (which has keep-alive off by default) instead of HTTP/1.1 or explicitly tell the server that no more requests will be send by adding a Connection: close header.
Please note that in general HTTP is way more complex than you might think if you've just seen a few examples. The problem you face in your question is only a glimpse into more problems which you will face when continuing this path. If you really want to implement your own HTTP handling instead of using established libraries please study the actual standard instead of just assuming a specific behavior.

Reading from a URL in java: when is a request actually sent?

I have an assignment for school that involves writing a simple web crawler that crawls Wikipedia. The assignment stipulates that I can't use any external libraries so I've been playing around with the java.net.URL class. Based on the official tutorial and some code given by my professor I have:
public static void main(String[] args) {
System.setProperty("sun.net.client.defaultConnectTimeout", "500");
System.setProperty("sun.net.client.defaultReadTimeout", "1000");
try {
URL url = new URL(BASE_URL + "/wiki/Physics");
InputStream is = url.openStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
String inputLine;
int lineNum = 0;
while ((inputLine = br.readLine()) != null && lineNum < 10) {
System.out.println(inputLine);
lineNum++;
}
is.close();
}
catch (MalformedURLException e) {
System.out.println(e.getMessage());
}
catch (IOException e) {
System.out.println(e.getMessage());
}
}
In addition, the assignment requires that:
Your program should not continuously send requests to wiki. Your program
must wait for at least 1 second after every 10 requests
So my question is, where exactly in the above code is the "request" being sent? And how does this connection work? Is the entire webpage being loaded in one go? or is it being downloaded line by line?
I honestly don't really understand much about networking at all so apologies if I'm misunderstanding something fundamental. Any help would be much appreciated.

InputStream is = url.openStream();
at the above line you will be sending request
BufferedReader br = new BufferedReader(new InputStreamReader(is));
at this line getting the input stream and reading.
Calling url.openStream() initiates a new TCP connection to the server that the URL resolves to. An HTTP GET request is then sent over the connection. If all goes right (i.e., 200 OK), the server sends back the HTTP response message that carries the data payload that is served up at the specified URL. You then need to read the bytes from the InputStream that the openStream() method returns in order to retrieve the data payload into your program.

Reusing sockets in server-client

I am trying to create a client-server system: my server is a raspberry pi which is running a python webserver on it, and my client is on a different pc and is written is Java. The idea is that the server collects data and when it gets a request from a client, it sends the data to the client.
My client should request the data, wait for 10 seconds and request again etc.
Currently this system is working, but after a day or so, the client starts getting a lot (but not continuously) socket timeouts. I think that this may be the case because for each request I create a new socket for communication and I think that after a day the sockets run out or something like that. This is the code the client executes every 10 seconds:
public static String getData() throws Exception {
TreeSet<Integer> primes = MathUtils.primesSieve(10000);
try {
String data = "";
Socket socket = new Socket(SERVER_ADDRESS, SERVER_PORT);
socket.setReuseAddress(true);
PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
int msg = ColUtils.drawRandomlyWithReplacement(primes, 1, ArrayList::new).get(0);
out.write(msg+"");
out.flush();
String input;
while ((input = in.readLine()) != null) {
data += input;
if (!data.endsWith("#" + prod(msg))) {
throw new Exception("WRONG ECHO");
}
}
socket.close();
return data;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
I tried fixing it by having a socket which is a member of the encapsulating class, but after a singe request the inputstream stopped working. Is there any way where I can keep using a single socket for ALL communications with the server? Or is this the recommended way of doing this sort of communication?

Try first closing the socket and input, output streams. As in your code there is no quarantee that you are releasing the acquired objects.
PrintWriter out = null;
BufferedReader in = null;
Socket socket = null;
try {
...//your statements
} catch (Exception ex) {
//catch or whatever
} finally {
if (out != null) out.close();
if (in != null) in.close();
if (socket != null) socket.close();
}

try to make the Socket object static If possible that would created only once and read the data every 10 sec
Otherwise u can instantiate it before calling the getData method and then read it.
Doing so will make only 1 copy of Socket.
And I don't think u are running out of ports.
The reason might be quit simple that your Program is not receiving the data before the time out. and it is a normal case in a bad network
Socket generally waits indefinitely until it receives data if the timeout is not set Programmatically

Java HTTP request/response from socket, server closing output stream before client can recieve

It seems that using curl and most web browsers my server code is closing the connection before the client is able to read the response. Here is my code
public void run() {
try {
InputStream input = clientSocket.getInputStream();
OutputStream output = clientSocket.getOutputStream();
System.out.println(input);
// getRequestObject(input);
long time = System.currentTimeMillis();
output.write(("HTTP/1.1 200 OK\n\nWorkerRunnable: " + this.serverText + " - " + time + "").getBytes());
output.flush();
output.close();
input.close();
System.out.println("Request processed: " + time);
} catch (IOException e) {
// report exception somewhere.
e.printStackTrace();
}
}
protected String readInputStream(InputStream input) throws IOException {
String inputLine;
BufferedReader in = new BufferedReader(new InputStreamReader(input));
StringBuilder sb = new StringBuilder();
while (!(inputLine = in.readLine()).equals("")) {
sb.append(inputLine);
}
return sb.toString();
}
Any thoughts?

Perhaps the problem could be caused by the fact that you're not reading the client's data. The client is trying to send you HTTP headers, but you immediately start sending the response. Try reading from the InputStream until you receive an empty line (which signals the end of the request HTTP headers) and then start sending the output.
If you need to embed an HTTP server in your application, I strongly recommend you to use an existing library. Implementing your own HTTP compliant server is going to be tedious work. See
Create a simple HTTP server with Java?
NanoHTTPD

Prevent IllegalStateException when reading from request

I have an interceptor for catching Exceptions and sending emails of this exceptions.
All my struts actions extend CoreController which implements SerlvetRequestAware.
In mail service class then I have:
CoreController cc = (CoreController)invocation.getAction();
HttpServletRequest request = cc.getRequest();
I want to insert request body to email, if exists. Like so:
StringWriter msg = new StringWriter();
msg.write("Requested URI: " + request.getRequestURI()+NEW_LINE);
msg.write("Requested Query String: " + request.getQueryString()+NEW_LINE);
msg.write("Request method: "+request.getMethod()+NEW_LINE);
try {
if (request.getReader() != null) {
msg.write("Request body: "+ request.getReader().readLine()+NEW_LINE);
request.getReader().close();
}
} catch (IOException e) {
e.printStrackTrace();
} catch(IllegalStateException e) {
e.printStrackTrace();
}
Now it always throws an IllegalStateException, when reader is not null. How could I "revert" reader or how any other way to read the request body?
EDIT
Exception: getInputStream() has already been called for this request

Perhaps you should try using the request's InputStream rather than its Reader if you receive an IllegalStateException:
BufferedReader bufferedReader;
try {
bufferedReader = request.getReader();
} catch (IllegalStateException e) {
InputStream inputStream = request.getInputStream();
// As per BalusC's comment:
String charsetName = request.getCharacterEncoding();
if (charsetName == null) {
charsetName = "UTF-8";
}
InputStreamReader inputStreamReader = new InputStreamReader(inputStream, charsetName);
bufferedReader = new BufferedReader(inputStreamReader);
}
bufferedReader.readLine();

You will get that exception if someone else has already read the request body or if the other kind of reader (in your case the InputStream) has been opened by someone.
My guess is that this happens in the code which parses the request. So at this stage, you can't read the request body anymore. Instead, you should check the field which contains the exception. Your struts config must contain this code somewhere:
<exception
key="exception"
path="/UserExists.jsp"
type="java.lang.Exception"/>
This means you can find the exception in the request attribute exception.

If you want to read the request body more than once, you could add a Filter that wraps the original HttpServletRequest inside a custom implementation that supports multiple reads (e.g. by storing the request body into a byte array and creating a new ByteArrayInputStream for each call).
Note: Wrapping HttpServletRequest and HttpServletResponse isn't uncommon (see for instance Jetty's GzipFilter, especially its GZIPResponseWrapper class). You should do this carefully though as changing (the behavior of) the request object might affect other filters or servlets.