Develop Reference

URLConnection doesn't read whole page

In my app I need to download some web page. I do it in a way like this
URL url = new URL(myUrl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(5000000);//5 seconds to download
conn.setConnectTimeout(5000000);//5 seconds to connect
conn.setRequestMethod("GET");
conn.setDoInput(true);
conn.connect();
int response = conn.getResponseCode();
is = conn.getInputStream();
String s = readIt(is, len);
System.out.println("got: " + s);
My readIt function is:
public String readIt(InputStream stream) throws IOException {
int len = 10000;
Reader reader;
reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[len];
reader.read(buffer);
return new String(buffer);
}
The problem is that It doesn't dowload the whole page. For example, if myUrl is "https://wikipedia.org", then the output is
How can I download the whole page?
Update
Second answer from here Read/convert an InputStream to a String solved my problem. The problem is in readIt function. You should read response from InputStream like this:
static String convertStreamToString(java.io.InputStream is) {
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
return s.hasNext() ? s.next() : "";
}

There are a number of mistakes your code:
You are reading into a character buffer with a fixed size.
You are ignoring the result of the read(char[]) method. It returns the number of characters actually read ... and you need to use that.
You are assuming that read(char[]) will read all of the data. In fact, it is only guaranteed to return at least one character ... or zero to indicate that you have reached the end of stream. When you reach from a network connection, you are liable to only get the data that has already been sent by the other end and buffered locally.
When you create the String from the char[] you are assuming that every position in the character array contains a character from your stream.
There are multiple ways to do it correctly, and this is one way:
public String readIt(InputStream stream) throws IOException {
Reader reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[4096];
StringBuilder builder = new StringBuilder();
int len;
while ((len = reader.read(buffer) > 0) {
builder.append(buffer, 0, len);
}
return builder.toString();
}
Another way to do it is to look for an existing 3rd-party library method with a readFully(Reader) method.

You need to read in a loop till there are no more bytes left in the InputStream.
while (-1 != (len = in.read(buffer))) { //do stuff here}

You are reading only 10000 bytes from the input stream.
Use a BufferedReader to make your life easier.
public String readIt(InputStream stream) throws IOException {
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
StringBuilder out = new StringBuilder();
String newLine = System.getProperty("line.separator");
String line;
while ((line = reader.readLine()) != null) {
out.append(line);
out.append(newLine);
}
return out.toString();
}

Eclipse warns about a potential resource leak although I have a finally block which closes the outermost stream, what am I missing?

is there a reason Eclipse gives me the following resource leak warning: Resource leak: 'br' is never closed" ? The code I am talking about is at the bottom of this post.
I thought my finally block had it all covered, my reasoning:
res will only be null if the FileInputStream constructor threw and therefore nothing has to be closed
res will be the inputstream if the InputStreamReader constructor throws (malformed encoding string for example) and then only the InputStream must be closed so ok
etc...
So what am I missing? Or could this be an eclipse bug?
Kind regards!
S.
public static String fileToString(String fileName, String encoding) throws IOException {
InputStream is;
InputStreamReader isr;
BufferedReader br;
Closeable res = null;
try {
is = new FileInputStream(fileName);
res = is;
isr = new InputStreamReader(is, encoding);
res = isr;
br = new BufferedReader(isr);
res = br;
StringBuilder builder = new StringBuilder();
String line = null;
while ((line = br.readLine()) != null) {
builder.append(line);
builder.append(LS);
}
return builder.toString();
} finally {
if (res != null) {
res.close();
}
}
}

Eclipse probably just isn't understanding the shuffling you're doing with the res variable.
I recommend using the try-with-resources statement (available in Java 7 and up, so three and a half years now), it dramatically simplifies these sorts of chains:
public static String fileToString(String fileName, String encoding) throws IOException {
try (
InputStream is = new FileInputStream(fileName);
InputStreamReader isr = new InputStreamReader(is, encoding);
BufferedReader br = new BufferedReader(isr)
) {
StringBuilder builder = new StringBuilder();
String line = null;
while ((line = br.readLine()) != null) {
builder.append(line);
builder.append(LS);
}
return builder.toString();
}
}
If you can't use try-with-resources, you probably want something like the Apache Commons IOUtils class's closeQuietly methods (either literally that one, or your own) rather than shuffling res around, which is awkward to read and I daresay prone to maintenance issues.
Using IOUtils might look like this:
public static String fileToString(String fileName, String encoding) throws IOException {
InputStream is = null;
InputStreamReader isr = null;
BufferedReader br = null;
try {
is = new FileInputStream(fileName);
isr = new InputStreamReader(is, encoding);
br = new BufferedReader(isr)
StringBuilder builder = new StringBuilder();
String line = null;
while ((line = br.readLine()) != null) {
builder.append(line);
builder.append(LS);
}
br.close();
return builder.toString();
}
finally {
IOUtils.closeQuietly(br, isr, is);
}
}
Note how I use a normal close in the try, but then ensure cleanup in the finally.
But try-with-resources is the better answer, as it's more concise and hooks into the new(ish) "suppressed exceptions" stuff.
Side note: There's no reason for the = null initialization of line, you assign it on the next line.
Side note 2: If the file is likely to be of any size, consider finding out how big it is in advance and setting the capacity of the StringBuilder in the constructor. StringBuilder's default capacity is 16, so a file of even a few hundred bytes involves several reallocations of StringBuilder's internal buffer.

Getting the content of an xml file and storing the content to a string in order to parse the content

I tried putting a toast before the return statement but the String variable returns an empty string.
public String getXmlFile(String pathFile, Context context){
String xmlFileString = "";
AssetManager am = context.getAssets();
try {
InputStream str = am.open(pathFile);
int length = str.available();
byte[] data = new byte[length];
xmlFileString = new String(data);
} catch (IOException e1) {
e1.printStackTrace();
}
return xmlFileString;
}

use this to read byte[] from InputStream:
public byte[] convertStreamToString(InputStream is) throws Exception {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line);
}
is.close();
return sb.toString().getBytes("UTF-8");
}

Use this to read the XML. Without passing UTF-8 to the InputStreamReader you might get a broken XML string.
BufferedReader reader = new BufferedReader(new InputStreamReader(
context.getAssets().open(pathFile), HTTP.UTF_8));
StringBuilder sb = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
sb.append(line);
}
reader.close();
Now when parsing the string to XML in my case there was also the problem that each line break was interpreted as an own XML node. Spaces also were an issue. Use this on the string read above to fix that:
String oneLineXml = sb.toString().replace("\n", "").replaceAll("> +<", "><");
Only then you should parse the string, like this:
Document xml = DocumentBuilderFactory.newInstance().newDocumentBuilder()
.parse(new InputSource(new ByteArrayInputStream(
oneLineXml.getBytes(HTTP.UTF_8))));

String to byte convertion in java

i m new in java....i m trying to read a text file using file input stream. i m reading text line by line and set as a string.. now i want to convert string into byte. but i m getting a number format exception.. please help me to solve this problem.
FileInputStream fstream = new FileInputStream("C:/Users/data.txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
byte[] bytes = null;
String str;
int i=0;
while ((str = br.readLine()) != null)
{
bytes[i] = Byte.parseByte(str,16);
i++;
}
in.close();

Try
byte[] bytes = str.getBytes();
instead of
bytes[i] = Byte.parseByte(str,16);
Also I recommend to specify encoding for InputStreamReader:
BufferedReader br = new BufferedReader(new InputStreamReader(in, "UTF-8"));
Keep in mind that Java String length and internal representation would not be same to C.

You can simply use the getBytes() method from the String class :
str.getBytes()
Or if you don't use the default character set :
str.getBytes(myCharSet);

you can use,
str.getBytes() which will convert the string into the byte array.

you can try this code.
fstream = new FileInputStream("C:/Users/s.hussain/Desktop/test3.txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
byte[] bytes = null;
String str;
int i=0;
while ((str = br.readLine()) != null)
{
bytes = str.getBytes();
i++;
System.out.println(bytes.length);
}
in.close();

Convert InputStream to String with encoding given in stream data

My input is a InputStream which contains an XML document. Encoding used in XML is unknown and it is defined in the first line of XML document.
From this InputStream, I want to have all document in a String.
To do this, I use a BufferedInputStream to mark the beginning of the file and start reading first line. I read this first line to get encoding and then I use an InputStreamReader to generate a String with the correct encoding.
It seems that it is not the best way to achieve this goal because it produces an OutOfMemory error.
Any idea, how to do it?
public static String streamToString(final InputStream is) {
String result = null;
if (is != null) {
BufferedInputStream bis = new BufferedInputStream(is);
bis.mark(Integer.MAX_VALUE);
final StringBuilder stringBuilder = new StringBuilder();
try {
// stream reader that handle encoding
final InputStreamReader readerForEncoding = new InputStreamReader(bis, "UTF-8");
final BufferedReader bufferedReaderForEncoding = new BufferedReader(readerForEncoding);
String encoding = extractEncodingFromStream(bufferedReaderForEncoding);
if (encoding == null) {
encoding = DEFAULT_ENCODING;
}
// stream reader that handle encoding
bis.reset();
final InputStreamReader readerForContent = new InputStreamReader(bis, encoding);
final BufferedReader bufferedReaderForContent = new BufferedReader(readerForContent);
String line = bufferedReaderForContent.readLine();
while (line != null) {
stringBuilder.append(line);
line = bufferedReaderForContent.readLine();
}
bufferedReaderForContent.close();
bufferedReaderForEncoding.close();
} catch (IOException e) {
// reset string builder
stringBuilder.delete(0, stringBuilder.length());
}
result = stringBuilder.toString();
}else {
result = null;
}
return result;
}

The call to mark(Integer.MAX_VALUE) is causing the OutOfMemoryError, since it's trying to allocate 2GB of memory.
You can solve this by using an iterative approach. Set the mark readLimit to a reasonable value, say 8K. In 99% of cases this will work, but in pathological cases, e.g 16K spaces between the attributes in the declaration, you will need to try again. Thus, have a loop that tries to find the encoding, but if it doesn't find it within the given mark region, it tries again, doubling the requested mark readLimit size.
To be sure you don't advance the input stream past the mark limit, you should read the InputStream yourself, upto the mark limit, into a byte array. You then wrap the byte array in a ByteArrayInputStream and pass that to the constructor of the InputStreamReader assigned to 'readerForEncoding'.

You can use this method to convert inputstream to string. this might help you...
private String convertStreamToString(InputStream input) throws Exception{
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line);
}
input.close();
return sb.toString();
}