In my app I need to download some web page. I do it in a way like this
URL url = new URL(myUrl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(5000000);//5 seconds to download
conn.setConnectTimeout(5000000);//5 seconds to connect
conn.setRequestMethod("GET");
conn.setDoInput(true);
conn.connect();
int response = conn.getResponseCode();
is = conn.getInputStream();
String s = readIt(is, len);
System.out.println("got: " + s);
My readIt function is:
public String readIt(InputStream stream) throws IOException {
int len = 10000;
Reader reader;
reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[len];
reader.read(buffer);
return new String(buffer);
}
The problem is that It doesn't dowload the whole page. For example, if myUrl is "https://wikipedia.org", then the output is
How can I download the whole page?
Update
Second answer from here Read/convert an InputStream to a String solved my problem. The problem is in readIt function. You should read response from InputStream like this:
static String convertStreamToString(java.io.InputStream is) {
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
return s.hasNext() ? s.next() : "";
}
There are a number of mistakes your code:
You are reading into a character buffer with a fixed size.
You are ignoring the result of the read(char[]) method. It returns the number of characters actually read ... and you need to use that.
You are assuming that read(char[]) will read all of the data. In fact, it is only guaranteed to return at least one character ... or zero to indicate that you have reached the end of stream. When you reach from a network connection, you are liable to only get the data that has already been sent by the other end and buffered locally.
When you create the String from the char[] you are assuming that every position in the character array contains a character from your stream.
There are multiple ways to do it correctly, and this is one way:
public String readIt(InputStream stream) throws IOException {
Reader reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[4096];
StringBuilder builder = new StringBuilder();
int len;
while ((len = reader.read(buffer) > 0) {
builder.append(buffer, 0, len);
}
return builder.toString();
}
Another way to do it is to look for an existing 3rd-party library method with a readFully(Reader) method.
You need to read in a loop till there are no more bytes left in the InputStream.
while (-1 != (len = in.read(buffer))) { //do stuff here}
You are reading only 10000 bytes from the input stream.
Use a BufferedReader to make your life easier.
public String readIt(InputStream stream) throws IOException {
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
StringBuilder out = new StringBuilder();
String newLine = System.getProperty("line.separator");
String line;
while ((line = reader.readLine()) != null) {
out.append(line);
out.append(newLine);
}
return out.toString();
}
Related
I am using an HttpUrlConnection to GET a very large JSON array from the web. I am reading the data 500 bytes at a time as so:
public String getJSON(String myurl) throws IOException {
URL url = new URL(myurl);
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
try {
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
String result = readIt(in, 500) ;
return result ;
//Log.d(TAG, result);
}
finally {
urlConnection.disconnect();
}
}
public String readIt(InputStream stream, int len) throws IOException, UnsupportedEncodingException {
StringBuilder result = new StringBuilder();
InputStreamReader reader = null;
reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[len];
while(reader.read(buffer) != -1)
{
System.out.println("!##: " + new String(buffer)) ;
result.append(new String(buffer)) ;
buffer = new char[len];
}
System.out.println(result.length()) ;
return result.toString();
}
This works fine on some phones, but not on newer phones. On newer phones I realized that the result JSON string was starting to contain garbage characters once it got to character 2048.
Some of my garbage return data:
ST AUGUSTINE, FL","2012050��������������������������������������
And the full error is:
Error: org.json.JSONException: Unterminated array at character 40549 of {"COLUMNS":["IMAGELI
Probably you append a wrong buffer to your string. You should count the number of char you get when reading and append them to the string but no more:
String str = new String(); // or use a StringBuilder if you prefer
char[] buffer = new char[len];
while ((count = reader.read(buffer, 0, len)) > 0)
{ str += new String(buffer, 0, count); }
Avoid recreate your buffer each time ! You allocate a new one for each loop... Reuse the buffer, as you have flushed it in str.
Be carreful when debugging: you cannot print a too long string in logcat (it will be cut if too long). But your str should be fine and should not contain any garbage data anymore.
I am creating this method which takes an InputStream as parameter, but the readLine() function is returning null. While debugging, inputstream is not empty.
else if (requestedMessage instanceof BytesMessage) {
BytesMessage bytesMessage = (BytesMessage) requestedMessage;
byte[] sourceBytes = new byte[(int) bytesMessage.getBodyLength()];
bytesMessage.readBytes(sourceBytes);
String strFileContent = new String(sourceBytes);
ByteArrayInputStream byteInputStream = new ByteArrayInputStream(sourceBytes);
InputStream inputStrm = (InputStream) byteInputStream;
processMessage(inputStrm, requestedMessage);
}
public void processMessage(InputStream inputStrm, javax.jms.Message requestedMessage) {
String externalmessage = tradeEntryTrsMessageHandler.convertInputStringToString(inputStrm);
}
public String convertInputStringToString(InputStream inputStream) throws IOException {
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
StringBuilder sb = new StringBuilder();
String line;
while ((line = br.readLine()) != null) {
sb.append(line);
}
br.close();
return sb.toString();
}
Kindly try this,
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"));
i believe that raw data as it is taken is not formatted to follow a character set. so by mentioning UTF-8 (U from Universal Character Set + Transformation Format—8-bit might help
Are you sure you are initializing and passing a valid InputStream to the function?
Also, just FYI maybe you were trying to name your function convertInputStreamToString instead of convertInputStringToString?
Here are two other ways of converting your InputStream to String, try these maybe?
1.
String theString = IOUtils.toString(inputStream, encoding);
2.
public String convertInputStringToString(InputStream is) {
java.util.Scanner s = new java.util.Scanner(is, encoding).useDelimiter("\\A");
return s.hasNext() ? s.next() : "";
}
EDIT:
You needn't explicitly convert ByteArrayInputStream to InputStream. You could do directly:
InputStream inputStrm = new ByteArrayInputStream(sourceBytes);
Aim : To read a Url which containing information in Json.
Question: I got a code of reading Url Which is given Below. I have a complete Understanding what code is doing but I do not have any idea why the size of char array is 1024 not 2048 or something else . How to decide what character size array is good at the time of reading Url ?
private static String readUrl(String urlString) throws Exception {
BufferedReader reader = null;
try {
URL url = new URL(urlString);
reader = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuffer buffer = new StringBuffer();
int read;
char[] chars = new char[1024]; ???
while ((read = reader.read(chars)) != -1)
buffer.append(chars, 0, read);
return buffer.toString();
} finally {
if (reader != null)
reader.close();
}
}
As the BufferedReader already has an internal buffer of 4096 characters, implementation-dependent, and as the socket already has a considerably larger receive buffer, it really doesn't make much difference what value you choose. The returns on buffering diminish geometrically with size.
I am working on learning Java and am going through the examples on the Android website. I am getting remote contents of an XML file. I am able to get the contents of the file, but then I need to convert the InputStream into a String.
public String readIt(InputStream stream, int len) throws IOException, UnsupportedEncodingException {
InputStreamReader reader = null;
reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[len];
reader.read(buffer);
return new String(buffer);
}
The issue I am having is I don't want the string to be limited by the len var. But, I don't know java well enough to know how to change this.
How can I create the char without a length?
Generally speaking it's bad practice to not have a max length on input strings like that due to the possibility of running out of available memory to store it.
That said, you could ignore the len variable and just loop on reader.read(...) and append the buffer to your string until you've read the entire InputStream like so:
public String readIt(InputStream stream, int len) throws IOException, UnsupportedEncodingException {
String result = "";
InputStreamReader reader = null;
reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[len];
while(reader.read(buffer) >= 0)
{
result = result + (new String(buffer));
buffer = new char[len];
}
return result;
}
My input is a InputStream which contains an XML document. Encoding used in XML is unknown and it is defined in the first line of XML document.
From this InputStream, I want to have all document in a String.
To do this, I use a BufferedInputStream to mark the beginning of the file and start reading first line. I read this first line to get encoding and then I use an InputStreamReader to generate a String with the correct encoding.
It seems that it is not the best way to achieve this goal because it produces an OutOfMemory error.
Any idea, how to do it?
public static String streamToString(final InputStream is) {
String result = null;
if (is != null) {
BufferedInputStream bis = new BufferedInputStream(is);
bis.mark(Integer.MAX_VALUE);
final StringBuilder stringBuilder = new StringBuilder();
try {
// stream reader that handle encoding
final InputStreamReader readerForEncoding = new InputStreamReader(bis, "UTF-8");
final BufferedReader bufferedReaderForEncoding = new BufferedReader(readerForEncoding);
String encoding = extractEncodingFromStream(bufferedReaderForEncoding);
if (encoding == null) {
encoding = DEFAULT_ENCODING;
}
// stream reader that handle encoding
bis.reset();
final InputStreamReader readerForContent = new InputStreamReader(bis, encoding);
final BufferedReader bufferedReaderForContent = new BufferedReader(readerForContent);
String line = bufferedReaderForContent.readLine();
while (line != null) {
stringBuilder.append(line);
line = bufferedReaderForContent.readLine();
}
bufferedReaderForContent.close();
bufferedReaderForEncoding.close();
} catch (IOException e) {
// reset string builder
stringBuilder.delete(0, stringBuilder.length());
}
result = stringBuilder.toString();
}else {
result = null;
}
return result;
}
The call to mark(Integer.MAX_VALUE) is causing the OutOfMemoryError, since it's trying to allocate 2GB of memory.
You can solve this by using an iterative approach. Set the mark readLimit to a reasonable value, say 8K. In 99% of cases this will work, but in pathological cases, e.g 16K spaces between the attributes in the declaration, you will need to try again. Thus, have a loop that tries to find the encoding, but if it doesn't find it within the given mark region, it tries again, doubling the requested mark readLimit size.
To be sure you don't advance the input stream past the mark limit, you should read the InputStream yourself, upto the mark limit, into a byte array. You then wrap the byte array in a ByteArrayInputStream and pass that to the constructor of the InputStreamReader assigned to 'readerForEncoding'.
You can use this method to convert inputstream to string. this might help you...
private String convertStreamToString(InputStream input) throws Exception{
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line);
}
input.close();
return sb.toString();
}