Develop Reference

Reading n bytes atomically without blocking

I just asked a question about why my thread shut down wasn't working. It ended up being due to readLine() blocking my thread before the shutdown flag could be recognised. This was easy to fix by checking ready() before calling readLine().
However, I'm now using a DataInputStream to do the following in series:
int x = reader.readInt();
int y = reader.readInt();
byte[] z = new byte[y]
reader.readFully(z);
I know I could implement my own buffering which would check the running file flag while loading up the buffer. But I know this would be tedious. Instead, I could let the data be buffered within the InputStream class, and wait until I have my n bytes read, before executing a non-blocking read - as I know how much I need to read.
4 bytes for the first integer
4 bytes for the second integer y
and y bytes for the z byte array.
Instead of using ready() to check if there is a line in the buffer, is there some equivalent ready(int bytesNeeded)?

The available() method returns the amount of bytes in the InputStreams internal buffer.
So, one can do something like:
while (reader.available() < 4) checkIfShutdown();
reader.readInt();

You can use InputStream.available() to get an estimate of the amount of bytes that can be read. Quoting the Javadoc:
Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking, which may be 0, or 0 when end of stream is detected. The read might be on the same thread or another thread. A single read or skip of this many bytes will not block, but may read or skip fewer bytes.
In other words, if available() returns n, you know you can safely call read(n) without blocking. Note that, as the Javadoc states, the value returned is an estimate. For example, InflaterInputStream.available() will always return 1 if EOF isn't reached. Check the documentation of the InputStream subclass you will be using to ensure it meets your needs.

You are going to need to implement your own equivalent of BufferedInputStream. Either as a sole owner of an InputStream and a thread (possibly borrowed from a pool) to block in. Alternatively, implement with NIO.

Channel.read returns -2

I'm using Java NIO to do socket operations. In the past when working with streams, the read call (where you are reading into a byte array, or in this case a ByteBuffer) returns the number of bytes read from the stream, or -1 if the stream was closed. So you basically can do
while(channel.read(buffer) != -1){
//do stuff
}
However, I noticed that I was killing my servers. When I added some logging statements, I noticed that the read() call was returning -2 at the end of the stream. According to the documentation:
Returns: The number of bytes read, possibly zero, or -1 if the channel
has reached end-of-stream
Has anybody experienced this before? I changed my code to loop on a value >0, but I wanted to make sure I understand what's going on.

java inputstream

What is an InputStream's available() method is supposed to return when the end of the stream is reached?
The documentation doesn't specify the behavior.

..end of the stream is reached
Don't use available() for detecting end of stream! Instead look to the int returned by InputStream.read(), which:
If no byte is available because the end of the stream has been reached, the value -1 is returned.

The JavaDoc does tell you in the Returns section -
an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking or 0 when it reaches the end of the input stream.
(from InputStream JavaDoc)

Theoretically if end of stream is reached there are not bytes to read and available returns 0. But be careful with it. Not all streams provide real implementation of this method. InputStream itself always returns 0.
If you need non-blocking functionality, i.e. reading from stream without being blocked on read use NIO instead.

From the Java 7 documentation:
"an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking or 0 when it reaches the end of the input stream."
So, I would say it should return 0 in this case. That also seems the most intuitive behaviour to me.

Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream. The next invocation might be the same thread or another thread. A single read or skip of this many bytes will not block, but may read or skip fewer bytes.
The available method for class InputStream always returns 0.
http://download.oracle.com/javase/6/docs/api/java/io/InputStream.html#available%28%29

May the FileInputStream.available foolish me?

This FileInputStream.available() javadoc says:
Returns an estimate of the number of
remaining bytes that can be read (or
skipped over) from this input stream
without blocking by the next
invocation of a method for this input
stream. The next invocation might be
the same thread or another thread. A
single read or skip of this many bytes
will not block, but may read or skip
fewer bytes.
In some cases, a non-blocking read (or
skip) may appear to be blocked when it
is merely slow, for example when
reading large files over slow
networks.
I'm not sure if in this check:
if (new FileInputStream(xmlFile).available() == 0)
can I rely that empty files will always return zero?
--
Thanks #SB, who does not exactly answered the question, but was the first to give the best alternative:
If xmlFile is a java.io.File object,
you can use the length() method to get
its size.

You can rely on new FileInputStream(fileName).available() returning zero if the named file is empty.
You cannot rely on new FileInputStream(fileName).available() == 0 as a definitive test that the file is empty. If fileName is a regular file on a local file system it will probably work. But if fileName is a device file or if it is a file on a remote file system, available() may return zero to report that a read() will have to block for a period. (Or in the case of a remote file system, it may not.)
A more reliable way to test the length of a regular file is to use new File(fileName).length() == 0. However for a device file or pipe, a length() call may return zero, irrespective of the number of bytes that can ultimately be read. And bear in mind that new File(fileName).length() also returns zero if the file does not exist.
EDIT If you want a reliable test to see if a file is empty, you have to make a number of calls:
public static isEmptyFile(String fileName) {
File file = new File(fileName);
if (!file.exists()) {
return false;
} else if (file.length() != 0L) {
return false;
} else if (file.isFile()) {
return true;
} else if (file.isDirectory()) {
return false;
} else {
// It may be impossible to tell that a device file / named pipe is
// "empty" without actually reading it. This is not a failing of
// Java: it is a logical consequence of the way that certain
// devices, etc work.
throw new CannotAnswerException(...);
}
}
But you would be well advised to test this carefully with a variety of "file" types on all platforms that you run your application on. The behavior of some of the file predicates is documented as being platform specific; see the javadoc.

I strongly advise against using available() - it can return 0 because the stream is blocked, even though there are still enough bytes to read. It probably won't occur with Files but the API does not guarantee it won't.
The same approach can be used with read() though:
if (new FileInputStream(xmlFile).read() == -1)
System.out.println("!!File empty!!");

If xmlFile is a java.io.File object, you can use the length() method to get its size.

My logical answer to the question "can I rely that empty files will always return zero?" is "Yes, for empty files, available() will return 0".
But you probably do also want to know "can I rely that only empty files will return zero?", and there the answer is "No, not by specification: available() might return 0 even if the file is not empty".
Additionally, you are opening a stream on a file and do not close it again. This may lead to unexpected and undesired behaviour, e.g. you may not be able to move or delete the file as long as your Java program is running. This is especially annoying if your program runs in an application server which usually runs for a very long time, making the file effectively immutable.

Extract first valid line of string from byte array

I am writing a utility in Java that reads a stream which may contain both text and binary data. I want to avoid having I/O wait. To do that I create a thread to keep reading the data (and wait for it) putting it into a buffer, so the clients can check avialability and terminate the waiting whenever they want (by closing the input stream which will generate IOException and stop waiting). This works every well as far as reading bytes out of it; as binary is concerned.
Now, I also want to make it easy for the client to read line out of it like '.hasNextLine()' and '.readLine()'. Without using an I/O-wait stream like buffered stream, (Q1) How can I check if a binary (byte[]) contain a valid unicode line (in the form of the length of the first line)? I look around the String/CharSet API but could not find it (or I miss it?). (NOTE: If possible I don't want to use non-build-in library).
Since I could not find one, I try to create one. Without being so complicated, here is my algorithm.
1). I look from the start of the byte array until I find '\n' or '\r' without '\n'.
2). Then, I cut the byte array from the start to that point and using it to create a string (with CharSet if specified) using 'new String(byte[])' or 'new String(byte[], CharSet)'.
3). If that success without exception, we found the first valid line and return it.
4). Otherwise, these bytes may not be a string, so I look further to another '\n' or '\r' w/o '\n'. and this process repeat.
5. If the search ends at the end of available bytes I stop and return null (no valid line found).
My question is (Q2)Is the following algorithm adequate?
Just when I was about to implement it, I searched on Google and found that there are many other codes for new line, for example U+2424, U+0085, U+000C, U+2028 and U+2029.
So my last question is (Q3), Do I really need to detect these code? If I do, Will it increase the chance of false alarm?
I am well aware that recognize something from binary is not absolute. I am just trying to find the best balance.
To sum up, I have an array of byte and I want to extract a first valid string line from it with/without specific CharSet. This must be done in Java and avoid using any non-build-in library.
Thanks you all in advance.

I am afraid your problem is not well-defined. You write that you want to extract the "first valid string line" from your data. But whether somet byte sequence is a "valid string" depends on the encoding. So you must decide which encoding(s) you want to use in testing.
Sensible choices would be:
the platform default encoding (Java property "file.encoding")
UTF-8 (as it is most common)
a list of encodings you know your clients will use (such as several Russian or Chinese encodings)
What makes sense will depend on the data, there's no general answer.
Once you have your encodings, the problem of line termination should follow, as most encodings have rules on what terminates a line. In ASCII or Latin-1, LF,CR-LF and LF-CR would suffice. On Unicode, you need all the ones you listed above.
But again, there's no general answer, as new line codes are not strictly regulated. Again, it would depend on your data.

First of all let me ask you a question, is the data you are trying to process a legacy data? In other words, are you responsible for the input stream format that you are trying to consume here?
If you are indeed controlling the input format, then you probably want to take a decision Binary vs. Text out of the Q1 algorithm. For me this algorithm has one troubling part.
`4). Otherwise, these bytes may not be a string, so I look further to
another '\n' or '\r' w/o '\n'. and this process repeat.`
Are you dismissing input prior to line terminator and take the bytes that start immediately after, or try to reevaluate the string with now 2 line terminators? If former, you may have broken binary data interface, if latter you may still not parse the text correctly.
I think having well defined markers for binary data and text data in your stream will simplify your algorithm a lot.
Couple of words on String constructor. new String(byte[], CharSet) will not generate any exception if the byte array is not in particular CharSet, instead it will create a string full of question marks ( probably not what you want ). If you want to generate an exception you should use CharsetDecoder.
Also note that in Java 6 there are 2 constructors that take charset
String(byte[] bytes, String charsetName) and String(byte[] bytes, Charset charset). I did some simple performance test a while ago, and constructor with String charsetName is magnitudes faster than the one that takes Charset object ( Question to Sun: bug, feature? ).

I would try this:
make the IO reader put strings/lines into a thread safe collection (for example some implementation of BlockingQueue)
the main code has only reference to the synced collection and checks for new data when needed, like queue.peek(). It doesn't need to know about the io thread nor the stream.
Some pseudo java code (missing exception & io handling, generics, imports++) :
class IORunner extends Thread {
IORunner(InputStream in, BlockingQueue outputQueue) {
this.reader = new BufferedReader(new InputStreamReader(in, "utf-8"));
this.outputQueue = outputQueue;
}
public void run() {
String line;
while((line=reader.readLine())!=null)
this.outputQueue.put(line);
}
}
class Main {
public static void main(String args[]) {
...
BlockingQueue dataQueue = new LinkedBlockingQueue();
new IORunner(myStreamFromSomewhere, dataQueue).start();
while(true) {
if(!dataQueue.isEmpty()) { // can also use .peek() != null
System.out.println(dataQueue.take());
}
Thread.sleep(1000);
}
}
}
The collection decouples the input(stream) more from the main code. You can also limit the number of lines stored/mem used by creating the queue with a limited capacity (see blockingqueue doc).
The BufferedReader handles the checking of new lines for you :) The InputStreamReader handles the charset (recommend setting one yourself since the default one changes depending on OS etc.).

The java.text namespace is designed for this sort of natural language operation. The BreakIterator.getLineInstance() static method returns an iterator that detects line breaks. You do need to know the locale and encoding for best results, though.

Q2: The method you use seems reasonable enough to work.
Q1: Can't think of something better than the algorithm that you are using
Q3: I believe it will be enough to test for \r and \n. The others are too exotic for usual text files.

I just solved this to get test stubb working for Datagram - I did byte[] varName= String.getBytes(); then final int len = varName.length; then send the int as DataOutputStream and then the byte array and just do readInt() on the rcv then read bytes(count) using the readInt.
Not a lib, not hard to do either. Just read up on readUTF and do what they did for the bytes.
The string should construct from the byte array recovered that way, if not you have other problems. If the string can be reconstructed, it can be buffered ... no?
May be able to just use read / write UTF() in DataStream - why not?
{ edit: per OP's request }
//Sending end
String data = new String("fdsfjal;sajssaafe8e88e88aa");// fingers pounding keyboard
DataOutputStream dataOutputStream = new DataOutputStream();//
final Integer length = new Integer(data.length());
dataOutputStream.writeInt(length.intValue());//
dataOutputStream.write(data.getBytes());//
dataOutputStream.flush();//
dataOutputStream.close();//
// rcv end
DataInputStream dataInputStream = new DataInputStream(source);
final int sizeToRead = dataInputStream.readInt();
byte[] datasink = new byte[sizeToRead.intValue()];
dataInputStream.read(datasink,sizeToRead);
dataInputStream.close;
try
{
// constructor
// String(byte[] bytes, int offset, int length)
final String result = new String(datasink,0x00000000,sizeToRead);//
// continue coding here
Do me a favor, keep the heat off of me. This is very fast right in the posting tool - code probably contains substantial errors - it's faster for me just to explain it writing Java ~ there will be others who can translate it to other code language ( s ) which you can too if you wish it in another codebase. You will need exception trapping an so on, just do a compile and start fixing errors. When you get a clean compile, start over from the beginnning and look for blunders. ( that's what a blunder is called in engineering - a blunder )