In my application, there is a separate thread, ran by ScheduledExecutorService.scheduleAtFixedRate() every minute, which parses rss feeds from multiple websites. I am using Apache HttpClient to receive xml.
Sample code:
InputStream inputStream = HTTPClient.get(url);
String xml = inputStreamToString(inputStream, encoding, websiteName);
public static String inputStreamToString(InputStream inputStream, String encoding, String websiteName)
{
BufferedReader bufferedReader = null;
PrintWriter printWriter = null;
StringBuilder stringBuilder = new StringBuilder();
int letter;
try
{
bufferedReader = new BufferedReader(new InputStreamReader(inputStream, encoding));
printWriter = new PrintWriter(new File("src/doclog/"
+ websiteName + "_"
+ new SimpleDateFormat("MM_dd_yyyy_hh_mm_ss").format(new Date(System.currentTimeMillis()))
+ "_" + encoding + ".txt"), encoding);
while((letter = bufferedReader.read()) != -1)
{
char character = (char) letter;
printWriter.print(character);
stringBuilder.append(character);
}
}
catch(IOException e)
{
throw new RuntimeException(e);
}
finally
{
try
{
if(bufferedReader != null)
{
bufferedReader.close();
}
if(printWriter != null)
{
printWriter.close();
}
}
catch(IOException e)
{
e.printStackTrace();
}
}
System.out.println("String built");
return stringBuilder.toString();
}
And HTTPClient class:
public class HTTPClient
{
private static final HttpClient CLIENT = HttpClientBuilder.create().build();
public static InputStream get(String url)
{
try
{
HttpGet request = new HttpGet(url);
HttpResponse response = CLIENT.execute(request);
System.out.println("Response Code: " + response.getStatusLine().toString());
return response.getEntity().getContent();
}
catch(IOException | IllegalArgumentException e)
{
throw new RuntimeException(e);
}
}
}
As the title says, sometimes there is a chance that bufferedReader.readLine() will hang forever. I've seen another answers on this topic, and they suggest to check if bufferedReader.ready() returns true. The problem is that there are websites, which will always return false in bufferedReader.ready(), while processing them, however they parse just fine.
How can i prevent my thread from hanging on bufferedReader.readLine()?
If it matters, response.getStatusLine().toString() always returns HTTP/1.1 200 OK
EDIT
I just found out that bufferedReader.ready() is actually true when hang happens.
EDIT 2
BufferedReader.read() hangs as well. It is strange that hang happens only when dealing with one single website, and it's occurrence is absolutely random. Application either could be working for 15 hours, receiving hundreds of non-problematic responses, or hang just in 10 minutes after launch. I've started to write all characters of every single update into separate file, and found out that nothing special really happens. Xml reading simply stops forever in the middle of document, the last characters were <p dir="ltr"&g. Updated the code.
Also, it's noteworthy to mention that there can't be any unhandled exceptions, because at the highest level of my ScheduledExecutorService.scheduleAtFixedRate() runnable i catch Throwable, and print it's stackTrace.
The ready() method returns true telling you that there are characters available for reading. The problem is that readLine() blocks until it finds an end-of-line in the input.
public String readLine()
throws IOException
Reads a line of text. A line is considered to be terminated by any one
of a line feed ('\n'), a carriage return ('\r'), or a carriage return
followed immediately by a linefeed.
As you are reading from a stream there is no guarantee that the data will come in at line boundaries so the readLine() call blocks.
You can use the read method which will not block, but you will have to check for EOL yourself.
public int read(char[] cbuf, int off, int len) throws IOException
Reads characters into a portion of an array.
This method implements the general contract of the corresponding read
method of the Reader class. As an additional convenience, it attempts
to read as many characters as possible by repeatedly invoking the read
method of the underlying stream. This iterated read continues until
one of the following conditions becomes true:
The specified number of characters have been read,
The read method of the underlying stream returns -1, indicating end-of-file, or
The ready method of the underlying stream returns false, indicating that further input requests would block.
If the first read on the underlying stream returns -1 to indicate
end-of-file then this method returns -1. Otherwise this method returns
the number of characters actually read.
Also you will have to reconstruct the line from the characters read. It is not ss convenient as reading the entire line at once but it is the way it must be done.
Related
I don't understand how to clear a BufferedReader. When I push button in Activity, the variable is set to 1 or 2. Depend on number change file in BufferedReader. When I push second time in buffer will be two files. How to tell BR to clear buffer before second will be upload. And vice versa, of course.
public List<String> getQuestionLinesList() {
String line;
List<String> lines = new ArrayList<String>();
Log.d(TAG, " Trying to get resourses");
Resources res = context.getResources();
try {
if (selectedBox == 1) {
bufferedQuestions = new BufferedReader(new InputStreamReader(res.openRawResource(R.raw.questions_list)));
} else if (selectedBox == 2) {
bufferedQuestions = new BufferedReader(new InputStreamReader(res.openRawResource(R.raw.questions_list_art)));
}
Log.d(TAG, "number i = " + Integer.toString(selectedBox));
while ((line = bufferedQuestions.readLine()) != null) {
lines.add(line);
Log.d(LINETAG, line);
}
} catch (IOException e) {
e.printStackTrace();
}
return lines;
}
Maybe you can try using a local variable in your method for BufferedReader not "bufferedQuestions" that you probably declared like a field in the class where that method : "getQuestionLinesList" belongs.
Add a carriage/ \n at the end of your files, then readLine() should be able to clear your BufferedReader.
JavaDocs for BufferedRead which states
public String readLine()
throws IOException
Reads a line of text. A line is considered to be terminated by any one of a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed.
Returns:
A String containing the contents of the line, not including any line-termination characters, or null if the end of the stream has been reached
I have the following code for compressing and decompressing string.
public static byte[] compress(String str)
{
try
{
ByteArrayOutputStream obj = new ByteArrayOutputStream();
GZIPOutputStream gzip = new GZIPOutputStream(obj);
gzip.write(str.getBytes("UTF-8"));
gzip.close();
return obj.toByteArray();
}
catch (IOException e)
{
e.printStackTrace();
}
return null;
}
public static String decompress(byte[] bytes)
{
try
{
GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(bytes));
BufferedReader bf = new BufferedReader(new InputStreamReader(gis, "UTF-8"));
StringBuilder outStr = new StringBuilder();
String line;
while ((line = bf.readLine()) != null)
{
outStr.append(line);
}
return outStr.toString();
}
catch (IOException e)
{
return e.getMessage();
}
}
I compress into byte array on windows, and then send the byte array through socket to the linux and uncompress it there. However upon uncompression it seem that all my newline characters are gone.
So I thought that the problem was linux to windows relationship. However I have tried writing a simple program on windows that uses it, and found that the newlines are still gone.
Can anyone shed any light as to what causes it? I can't figure out any explanation.
I think the problem is here:
while ((line = bf.readLine()) != null)
{
outStr.append(line);
}
The readLine see's the newline char but doesn't include it in the returned value for line
The problem is worse than you think, perhaps.
readLine() gets all the characters up to, but not including, a newline (or some variety of returns and linefeed characters) OR the end of file. So you don't know if the last line you get had a newline on the end or not.
This might not matter, and if so, you can just add this following the other append:
outStr.append('\n');
Some files might end up with an extra line ending at the end of file.
If it does matter, you will need to use read() and then output all the characters you receive. In that case, you might end up with the infamous "What's at the end of the line?" problem you allude to between Windows, Linux and the MacOS and the way they use different combinations of return and new-line characters to end lines.
It is not GZIP that is "eating" newlines.
It is this code:
while ((line = bf.readLine()) != null)
{
outStr.append(line);
}
The readLine() method reads a line (up to and including a line termination sequence) and then returns it without a newline. You then append it to outStr ... without replacing the line termination that was stripped.
But even if you replaced the line termination, you can't guarantee to preserve the actual line termination sequence that was used ... if you do it that way.
I recommend that you replace the readLine() calls with read() calls; i.e. read and then buffer the data one character at a time. It solves two problems at once. It may even be faster, because you are avoiding the unnecessary overhead of assembling line Strings.
I am trying to read text from a web document using a BufferedReader over an InputStreamReader on an URL (to the file on some Apache server).
String result = "";
URL url = new URL("http://someserver.domain/somefile");
BufferedReader in = null;
in = new BufferedReader(new InputStreamReader(url.openStream(), "iso-8859-1"));
result += in.readLine();
Now this works just fine. But Obviously I'd like the reader not to just read one line, but as many as there are in the file.
Looking at the BufferedReader API the following code should do just that:
while (in.ready()) {
result += in.readLine();
}
I.e. read all lines while there are more lines, stop when no more lines are there. This code does not work however - the reader just never reports ready() = true!
I can even print the ready() value right before reading a line (which reads the correct string from the file) but the reader will report 'false'.
Am I doing something wrong? Why does the BufferedReader return 'false' on ready when there is actually stuff to read?
ready() != has more
ready() does not indicate that there is more data to be read. It only shows if a read will could block the thread. It is likely that it will return false before you read all data.
To find out if there is no more data check if readLine() returns null.
String line = in.readLine();
while(line != null){
...
line = in.readLine();
}
Another way you can do this that bypasses the in.ready() is something like:
while ((nextLine = in.readLine()) != null) {
result += nextLine;
}
You will just continue reading until you are done. This way you do not need to worry about the problem with in.ready().
I think the standard way to write this is to just attempt to read the line and verify that it returned sometime. Something like this:
while ((String nextLine = in.readLine()) != null) {
//System.out.println(nextLine);
result += nextLine;
}
So you just continue to go until you get null returned from the stream. See here for extra information:
http://download.oracle.com/javase/1.5.0/docs/api/java/io/BufferedReader.html#readLine()
The BufferedReader.ready() method is behaving as specified:
The Reader.ready() javadoc says the following:
[Returns] true if the next read() is guaranteed not to block for input, false otherwise. Note that returning false does not guarantee that the next read will block.
Then the BufferedReader.ready() javadoc says the following:
Tells whether this stream is ready to be read. A buffered character stream is ready if the buffer is not empty, or if the underlying character stream is ready.
If you put these two together, it is clear that BufferedReader.ready() can return false in situations where are characters available. In short, you shouldn't rely on ready() to test for logical end-of-file or end-of-stream.
This is what we have been using consistently for years - not sure if it is the "standard" method. I'd like to hear comments about the pros and cons of using URL.openURLStream() directly, and if that is causing the OP's problems. This code works for both HTTP and HTTPS connections.
URL getURL = new URL (servletURL.toString() + identifier+"?"+key+"="+value);
URLConnection uConn = getURL.openConnection();
BufferedReader br = new BufferedReader (new
InputStreamReader (uConn.getInputStream()));
for (String s = br.readLine() ; s != null ; s = br.readLine()) {
System.out.println ("[ServletOut] " + s);
// do stuff with s
}
br.close();
Basically the BufferedReader.ready() method can be used for checking whether the underlying stream is ready for providing data to the method caller.... else we can wait the thread for some time till it becomes ready.
But the real problem is that after we completely read the data stream, it will throw false..
so we didn't know whether the stream is fully read OR underlying stream is busy....
If you want to use in.ready(), the following worked for me well:
for (int i = 0; i < 10; i++) {
System.out.println("is InputStreamReader ready: " + in.ready());
if (!in.ready()) {
Thread.sleep(1000);
} else {
break;
}
}
I am building a simple client-server program , I have in main :
FTPClient ftp = new FTPClient("www.kernel.org");
ftp.getReply();
ftp.sendCommand("USER " + "anonymous");
ftp.getReply();
ftp.sendCommand("PASS " + "anonymous");
ftp.getReply();
String com="";
while (!com.equalsIgnoreCase("quit")){
System.out.println("Enter your Commands . or Enter quit");
BufferedReader Keyboard = new BufferedReader(new InputStreamReader(System.in));
com = Keyboard.readLine();
ftp.sendCommand((com));
ftp.getReply();
System.out.println("===============");
}
ftp.close();
the problem is in the getReply() function, this function is :
public void getReply() throws IOException {
String line="";
while (br.ready())
{
line = br.readline();
System.out.println(line);
System.out.flush();
}
}
br is a BufferedReader.Now all the problem is that when the program starts it doesn't show the welcome message from the Server until I press Enter or any command, when I Debug the program Step by Step every thing is working perfectly.So is the problem in the readline and I should use something else or what?
The problem is likely that the end of the server response does not contain a newline character. The BufferedReader's readLine method will block until a line of data is received, where "a line" consists of some characters followed by a newline character (or the end of the stream). Consequently, the readLine call will not return if no newline is received.
In this situation then, the BufferedReader isn't doing you any good. You'd be better off using the underlying Reader yourself, reading into an array and emitting the output as soon as it comes in, such as the following:
final char[] buffer = new char[256]; // or whatever size you want
int nRead;
while ((nRead = reader.read(buffer)) != -1)
{
System.out.println(new String(buffer, 0, nRead));
System.out.flush();
}
The condition in the while loop there might look confusing if you're not used to it before, but it combines the read operation (which reads into the buffer) with the check that the end of the stream has not been reached. Likewise, the construction of the String within the while loop takes into account the fact that the buffer may not have been filled entirely, so only as many characters as were supplied are used.
Note that this particular snippet keeps looping until the stream is empty; you may wish to add another exit condition in your particular case.
I have an InputStreamReader object. I want to read multiple lines into a buffer/array using one function call (without crating a mass of string objects). Is there a simple way to do so?
First of all mind that InputStreamReader is not so efficient, you should wrap it around a BufferedReader object for maximum performance.
Taken into account this you can do something like this:
public String readLines(InputStreamReader in)
{
BufferedReader br = new BufferedReader(in);
// you should estimate buffer size
StringBuffer sb = new StringBuffer(5000);
try
{
int linesPerRead = 100;
for (int i = 0; i < linesPerRead; ++i)
{
sb.append(br.readLine());
// placing newlines back because readLine() removes them
sb.append('\n');
}
}
catch (Exception e)
{
e.printStackTrace();
}
return sb.toString();
}
Mind that readLine() returns null is EOF is reached, so you should check and take care of it.
If you have some delimiter for multiple lines you can read that many characters using read method with length and offset. Otherwise using a StringBuilder for appending each line read by BufferedReader should work well for you without eating up too much temp memory