Develop Reference

URLConnection doesn't read whole page

In my app I need to download some web page. I do it in a way like this
URL url = new URL(myUrl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(5000000);//5 seconds to download
conn.setConnectTimeout(5000000);//5 seconds to connect
conn.setRequestMethod("GET");
conn.setDoInput(true);
conn.connect();
int response = conn.getResponseCode();
is = conn.getInputStream();
String s = readIt(is, len);
System.out.println("got: " + s);
My readIt function is:
public String readIt(InputStream stream) throws IOException {
int len = 10000;
Reader reader;
reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[len];
reader.read(buffer);
return new String(buffer);
}
The problem is that It doesn't dowload the whole page. For example, if myUrl is "https://wikipedia.org", then the output is
How can I download the whole page?
Update
Second answer from here Read/convert an InputStream to a String solved my problem. The problem is in readIt function. You should read response from InputStream like this:
static String convertStreamToString(java.io.InputStream is) {
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
return s.hasNext() ? s.next() : "";
}

There are a number of mistakes your code:
You are reading into a character buffer with a fixed size.
You are ignoring the result of the read(char[]) method. It returns the number of characters actually read ... and you need to use that.
You are assuming that read(char[]) will read all of the data. In fact, it is only guaranteed to return at least one character ... or zero to indicate that you have reached the end of stream. When you reach from a network connection, you are liable to only get the data that has already been sent by the other end and buffered locally.
When you create the String from the char[] you are assuming that every position in the character array contains a character from your stream.
There are multiple ways to do it correctly, and this is one way:
public String readIt(InputStream stream) throws IOException {
Reader reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[4096];
StringBuilder builder = new StringBuilder();
int len;
while ((len = reader.read(buffer) > 0) {
builder.append(buffer, 0, len);
}
return builder.toString();
}
Another way to do it is to look for an existing 3rd-party library method with a readFully(Reader) method.

You need to read in a loop till there are no more bytes left in the InputStream.
while (-1 != (len = in.read(buffer))) { //do stuff here}

You are reading only 10000 bytes from the input stream.
Use a BufferedReader to make your life easier.
public String readIt(InputStream stream) throws IOException {
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
StringBuilder out = new StringBuilder();
String newLine = System.getProperty("line.separator");
String line;
while ((line = reader.readLine()) != null) {
out.append(line);
out.append(newLine);
}
return out.toString();
}

Sending Message using TCP Socket not returning any data

I am using following code
OutputStream out = socket.getOutputStream();
PrintWriter output = new PrintWriter(out);
#SuppressWarnings("unused")
byte[] b = message.getBytes("US-ASCII");
String convertedString = Normalizer.normalize(message, Normalizer.Form.NFD).replaceAll("[^\\p{ASCII}]", "");
Log.d("ASCII", convertedString);
output.println(convertedString);
Log.e("TCP Message sent", convertedString);
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
//read line(s)
String st = input.readLine();
Log.e("TCP Message received", st);
But I am stuck at String st = input.readLine(); It just hangs there and never proceed to next line. Is this correct way of sending message using TCLP And then get response?

You are getting stuck at readLine() because the connection is still open - android is still waiting for more data to come in, and so can't move on.
This is what I did, but whether this is viable may depend on the protocol you are adhering to.
StringBuilder builder = new StringBuilder();
String line;
while ((line = input.readLine()) != ""){
builder.append(line);
}
Log.i("data", builder.toString());

What you need is flushing your output stream after sending message by:
output.flush();
or use auto flush by passing true while creating Output Stream like this:
PrintWriter output = new PrintWriter(out, true);
Hope this helps

BufferedReader readLine detect closed socket and use .ready()

Server snippet:
reader = new BufferedReader(new InputStreamReader(new BufferedInputStream(connection.getInputStream())));
if (reader.ready()) {
result = reader.readLine();
if (result == null) {
quit(); // handle closing the socket
} else {
// process result
}
}
Client snippet:
BufferedOutputStream os = new BufferedOutputStream(connection.getOutputStream());
OutputStreamWriter osw = new OutputStreamWriter(os, "US-ASCII");
osw.write("hello\n");
osw.flush();
The problem:
When the client writes something to the server it gets read correctly but when the client closes it's socket (For example force quitting the program), the reader.readLine() should output null, because of if (reader.ready()) { the null result will never reach quit().
if i leave away the if (reader.ready()) {, the server can't read data.
How do i fix this?
EDIT: i can't use a while ((result = reader.readLine()) != null) { because i need to run more code after that like this:
while (running) {
try {
reader = new BufferedReader(new InputStreamReader(new BufferedInputStream(connection.getInputStream())));
if (reader.ready()) {
result = reader.readLine();
if (result == null) quit();
else // process result
}
catch (Exception e) { ... }
// more code
}
}

Just remove the ready() test. It is as simple as that. ready() is not a test for end of stream. readLine() will block while there is no data.
And you must create the BufferedReader once, outside the read loop. Otherwise you will lose data.

I have solved it myself by running this piece of code every 5 seconds:
try {
BufferedOutputStream os = new BufferedOutputStream(thread_cnn.getOutputStream());
OutputStreamWriter osw = new OutputStreamWriter(os, "US-ASCII");
osw.write("areyoustillthere\n");
osw.flush();
} catch (Exception e) {
quit(); // handle quit if server can't reach client
}

Writing a simple HTTP server to accept GET requests

I'm trying to create a simple server that accepts a request, and then writes the content of a file to the browser that sent the request. The server connects and writes to the socket. However my browser says
no data received
and doesn't display anything.
public class Main {
/**
* #param args
*/
public static void main(String[] args) throws IOException{
while(true){
ServerSocket serverSock = new ServerSocket(6789);
Socket sock = serverSock.accept();
System.out.println("connected");
InputStream sis = sock.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(sis));
String request = br.readLine(); // Now you get GET index.html HTTP/1.1`
String[] requestParam = request.split(" ");
String path = requestParam[1];
System.out.println(path);
PrintWriter out = new PrintWriter(sock.getOutputStream(), true);
File file = new File(path);
BufferedReader bfr = null;
String s = "Hi";
if (!file.exists() || !file.isFile()) {
System.out.println("writing not found...");
out.write("HTTP/1.0 200 OK\r\n");
out.write(new Date() + "\r\n");
out.write("Content-Type: text/html");
out.write("Content length: " + s.length() + "\r\n");
out.write(s);
}else{
FileReader fr = new FileReader(file);
bfr = new BufferedReader(fr);
String line;
while ((line = bfr.readLine()) != null) {
out.write(line);
}
}
if(bfr != null){
bfr.close();
}
br.close();
out.close();
serverSock.close();
}
}
}

Your code works for me (data shows up in the browser), if I use
http://localhost:6789/etc/hosts
and there is a file /etc/hosts (Linux filesystem notation).
If the file does not exist, this snippet
out.write("HTTP/1.0 200 OK\r\n");
out.write(new Date() + "\r\n");
out.write("Content-Type: text/html\r\n");
out.write("\r\n");
out.write("File " + file + " not found\r\n");
out.flush();
will return data that shows up in the browser: Note that I have explicitly added a call to flush() here. Make sure that out is flushed in the other case as well.
The other possibility is to reorder your close statements.
A quote from EJP's answer on How to close a socket:
You should close the outermost output stream you have created from the socket. That will flush it.
This is especially the case if the outermost output stream is (another quote from the same source):
a buffered output stream, or a stream wrapped around one. If you don't close that, it won't be flushed.
So out.close() should be called before br.close().

How to send EOF to a process in Java?

I want to run groff in a Java program. The input comes from a string. In real command line, we will terminate the input by ^D in Linux/Mac. So how to send this terminator in Java program?
String usage +=
".Dd \\[year]\n"+
".Dt test 1\n"+
".Os\n"+
".Sh test\n"+
"^D\n"; // <--- EOF here?
Process groff = Runtime.getRuntime().exec("groff -mandoc -T ascii -");
groff.getOutputStream().write(usage.getBytes());
byte[] buffer = new byte[1024];
groff.getInputStream().read(buffer);
String s = new String(buffer);
System.out.println(s);
Or any other idea?

^D isn't a character; it's a command interpreted by your shell telling it to close the stream to the process (thus the process receives EOF on stdin).
You need to do the same in your code; flush and close the OutputStream:
String usage =
".Dd \\[year]\n" +
".Dt test 1\n" +
".Os\n" +
".Sh test\n";
...
OutputStream out = groff.getOutputStream();
out.write(usage.getBytes());
out.close();
...

I wrote this utility method:
public static String pipe(String str, String command2) throws IOException, InterruptedException {
Process p2 = Runtime.getRuntime().exec(command2);
OutputStream out = p2.getOutputStream();
out.write(str.getBytes());
out.close();
p2.waitFor();
BufferedReader reader
= new BufferedReader(new InputStreamReader(p2.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
return sb.toString();
}