In my current design I have a named pipe which can be sequentially written to by an unspecified number of writer processes. There is only one reader implemented in Scala, but for the sake of simplicity we can assume it's implemented in Java. The operating system is Linux >= 2.6.
The reader needs to:
re-open the pipe after each writer sends its input
read all the input from pipe until EOF which indicates that the writer closed its end of the named pipe
The main difficulty here is that the reader needs to be able to cancel both open and read calls after a given timeout. Reaching the timeout indicates that all writers have done their job and the reader can safely exit.
In C, I would:
first call open(file_name, O_NONBLOCK) and poll for the pipe to be open for writing
poll for reading in non-blocking mode or change file descriptor to blocking mode and use select()
What is the most straightforward way to complete this in Java? I've looked at the classic IO and NIO but there more I try, the more complex the design becomes and it still doesn't do exactly what I want.
Related
I have a multi-threaded program, in which I open a BufferedReader to read content from FIFO(named Pipe) file. As I want to implement stream type of solution to read text from FIFO file continuously, I have created a BufferedReader outside of thread task run, and want to keep that open forever as long as application is running.(No close() on bufferedReader)
With the limited(let say 10) threads in ThreadPool will keep look for text in FIFO file and process that text for further. As I am using FIFO it will never reach END OF FILE.
By doing this, For a smaller input file it reads successfully, for a large input file it throws Stream closed IOexception(sporadically). It close automatically, I do not have close() statement.
I have a code in place to acquire and close the semaphore lock at the place where i use br.readLine() to handle race condition issue
java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:122) ~[?:1.8.0_152]
at java.io.BufferedReader.readLine(BufferedReader.java:317) ~[?:1.8.0_152]
at java.io.BufferedReader.readLine(BufferedReader.java:389) ~[?:1.8.0_152]
Question:
For this solution I do not want to close the BufferedReader. What are the consequences?
Can I have a bufferedReader which never be closed? if so what steps I should consider in code.
BufferedReader is not a thread-safe class, so, we can get an uncountable number of various error on attempt to use the same object of that class from different threads.
I am writing to and reading from a Linux file in java, which in reality is a communication port to a hardware device. To do this I use RandomAccessFile (I'll explain why later) and it works well in most cases. But sometimes a byte is lost and then my routine blocks indefinitely since there is no timeout on the read method.
To give some more details on the file: it is a USB receipt printer that creates a file called /dev/usb/lp0 and though I can use a cups driver to print, I still need the low level communication through this file to query the status of the printer.
The reason I use RandomAccessFile is that I can have the same object for both reading and writing.
I tried to make a version with InputStream and OutputStream instead (since that would allow me to use the available() method to implement my timeout). But when I first open the InputStream and then the OutputStream I get an exception when opening the OutputStream since the file is occupied.
I tried writing with the OutputStream and then closing it before opening the InputStream to read, but then I lose some or all of the reply before it has opened the InputStream.
I tried switching to channels instead (Files.newByteChannel()). This also allows me to have just one object, and the documentation says it only reads the bytes available and returns the count (which also allows me to implement a timeout). But it blocks in the read method anyway when there is nothing to read, despite what the documentation says.
I also tried a number of ways to implement timeouts on the RandomAccessFile using threads.
The first approach was to start a separate thread at the same time as starting to read, and if the timeout elapsed in the thread I closed the file from the thread, hoping that this would unlock the read() operation with an exception, but it didn't (it stayed blocked).
I also tried to do the read in a separate thread and brutally kill it with the deprecated Thread.stop() once the time had elapsed. This worked one time, but it was not possible to reopen the file again after that.
The only solution I have made work is to have a separate thread that continuously calls read, and whenever it gets a byte it puts it in a LinkedBlockingQueue, which I can read from with a timeout. This approach works, but the drawback is that I can never close the file (again for the same reasons explained above, I can't unblock a blocked read). And my application requires that I sometimes close this connection to the hardware.
Anyone who can think of a way to read from a file with timeout that would work in my case (that allows me to have both a read and a write access open to the file at the same time)?
I am using Java8 by the way.
I have a binary linux executable which prints some bytes in stdout. I consume these bytes from Java application as this:
String[] cmd;
Process p = Runtime.getRuntime().exec(cmd);
InputStream is = p.getInputStream();
int r = is.read();
while(r != -1){
System.out.println(r);
r = is.read(); //1
}
But after some time of working the //1 is blocked for I/O forever (dead-lock). I created thread dump and noticed that
"pool-2-thread-1#627" prio=5 tid=0xd nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(FileInputStream.java:-1)
at java.io.FileInputStream.read(FileInputStream.java:255)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x2c0> (a java.lang.UNIXProcess$ProcessPipeInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
It's locked on a native method
private native int readBytes(byte[] var1, int var2, int var3) throws IOException;
What's a possible reason this method is blocked forever? Maybe this is platform-specific. I'm using Ubuntu 16.04.
You don't need to look at the internal methods of the standard library for an answer. The docs of InputStream.read() specify, in part, that
If no byte is available because the end of the stream has been
reached, the value -1 is returned. This method blocks until input data
is available, the end of the stream is detected, or an exception is
thrown.
Perhaps the key point of confusion is the meaning of "the end of the stream is detected." This does not mean that no more bytes are available to read right now -- that wouldn't be consistent with specification that the method blocks until data is available, and in practice it would work out poorly in many situations. Rather, the end of the stream is detected when the system receives some sort of signal that no more bytes will ever be available from it. Generally, that means that the stream has been closed on the source end, and all bytes drained from it at the destination end.
In your particular case, if the external process remains running, does not write anything further to its standard output, yet does not close its standard output, then Java (or a native consumer) will never see end-of-stream on that process's output.
Your InputStream is waiting for the process to either supply another byte, or close the stream (by finishing).
One reason the process might be doing neither of those things, is that it's written to stderr, and is blocking on that write.
Possibly there is a difference in environment between your shell and the environment used by Java, that means the command is resulting in output to stderr (e.g. the program is not in $PATH; you don't have adequate permissions; your CWD isn't what you expect, etc.)
A quick and dirty way to find out if this is the problem, is to redirect stderr to stdout -- either with ProcessBuilder.redirectErrorStream() or by using 2>&1 on the UNIX side of things.
If this is the case, then it is a form of deadlock -- the Java program is waiting for the native program to write to one pipe. The native program is waiting for the Java program to read from another.
The clean way to handle it, which you might choose to do in the longer term, is to use getErrorStream() and handle its output. This isn't exactly trivial, because in a single-threaded program using blocking reads, you can never know which stream is going to have data. You can either do blocking reads in separate threads, or use NIO to handle both inputs in a non-blocking manner.
By the way - note that the Java docs advise ProcessBuilder.start() over Runtime.exec(), since Java 1.5.
There are five different type IO model under Linux , they are respectively
Blocking IO
NonBlocking IO
IO Multiplexing
Signal-Driven
Asynchornours IO
java.io.FileInputStream.readBytes() use first type, Blocking IO. Application waits until kernel returns data.
consider the following scenario:
Process 1 (Writer) continuously appends a line to a file ( sharedFile.txt )
Process 2 (Reader) continuously reads a line from sharedFile.txt
my questions are:
In java is it possible that :
Reader process somehow crashes Writer process (i.e. breaks the process of Writer)?
Reader some how knows when to stop reading the file purely based on the file stats (Reader doesn't know if others are writing to the file)?
to demonsterate
Process one (Writer):
...
while(!done){
String nextLine;//process the line
writeLine(nextLine);
...
}
...
Process Two (Reader):
...
while(hasNextLine()){
String nextLine= readLine();
...
}
...
NOTE:
Writer Process has priority. so nothing must interfere with it.
Since you are talking about processes, not threads, the answer depends on how the underlying OS manages open file handles:
On every OS I'm familiar with, Reader will never crash a writer process, as Reader's file handle only allows reading. On Linux, system calls a Reader can potentially invoke on the underlying OS are open(2) with O_RDONLY flag, lseek(2) and read(2) -- are known not to interfere with the syscalls that the Writer is invoking, such as write(2).
Reader most likely won't know when to stop reading on most OS. More precisely, on some read attempt it will receive zero as the number of read bytes and will treat this as an EOF (end of file). At this very moment, there can be Writer preparing to append some data to a file, but Reader have no way of knowing it.
If you need a way for two processes to communicate via file, you can do it using some extra files that pass meta-information between Readers and Writers, such as whether there are Writer currently running. Introducing some structure into a file can be useful too (for example, every Writer appends a byte to a file indicating that the write process is happening).
For very fast non-blocking I/O you may want consider memory mapped files via Java's MappedByteBuffer.
The code will not crash. However, the reader will terminate when the end is reached, even if the writer may still be writing. You will have to synchronize somehow!
Concern:
Your reader thread can read a stale value even when you think another writer thread has updated the variable value
Even if you write to a file if synchronization is not there you will see a different value while reading
Java File IO and plain files were not designed for simultaneous writes and reads. Either your reader will overtake your writer, or your reader will never finish.
JB Nizet provided the answer in his comment. You use a BlockingQueue to hold the writer data while you're reading it. Either the queue will empty, or the reader will never finish. You have the means through the BlockingQueue methods to detect either situation.
Ok, So i am trying to read the output of a c binary from java code and I am unable to figure out whether the communication channel is blocking or non blocking.
The setup is such:
A java class (A.java) is run
A.java runs a c binary (B.o) using Runtime.getRuntime().exec("B.o"). At this point I have the Process object (returned by Runtime.exec)
A.java reads from the input stream of the Process object using a bufferedreader
A.java outputs the data read from the input stream to a file (output.txt)
The B.o binary simply prints random lines using printf function call.
Now, if I run the above setup, I receive all the data sent by B.o flawlessly. Then to test (the blocking / nonblocking thing), I changed the A.java to sleep for 5 milliseconds after every read from the inputstream of the Process object of B.o. As it turned out, now I wasn't receiving the complete data in A.java send by B.o. This indicates that the communication channel being used is non-blocking (as per my weak understanding).
Then just to make sure, I started looking at the source code of java to see if I was right. And I have found the following so far:
Every call to Runtime.getRuntime().exec(...) ends up in forkAndExec() method in ProcessImpl_md.c. In ProcessImpl_md.c the command is executed, a process is created, and PIPES are setup for communication (using the pipe function call in c). I can't find anywhere in the source code where the PIPES are being set to nonblocking mode (as indicated by my code). I am assuming the PIPES are blocking by default.
I know this is a very bad way to check what I want to check. I am way out of my depth here and I am just head-banging uselessly, I think.
Can anyone point me in the right direction or tell me:
Are the PIPES of a process created through java runtime API are blocking or non-blocking?
When I make A.java sleep after reading from the input stream, why all data is not received? (Assumption being that the PIPE is blocking)
Any non-programmatic way (i.e. I don't have to change the source code of java and etc!) to figure out if the PIPES of a process are blocking or non-blocking?
Thank you.
EDIT: (added code)
Following is not the actual (or even compilable) code but it shows what i am trying to do.
Source of "B.o":
#include <stdio.h>
void main(int argc, char*argv[]){
int a = 0;
for(; a<9000000; a++){
printf("%s", argv[1]);
}
}
Source of "A.java":
<java imports>
public class A{
public static void main(String[] args) throws Exception{
Process p = Runtime.getRuntime().exec("./B.o");
BufferedReader br = new
BufferedReader(new InputStreamReader(p.getInputStream()));
int a = 0;
while(br.readLine() != null){
a++;
Thread.sleep(5);//data missed if this line not commented out
}
br.close();
System.out.println(a);
}
}
PLEASE CHECK MY ANSWER. USELESS QUESTION BY ME.
Whether the communication channels between Java and the external program (there are three, one from Java to native, and two coming back) are operating in blocking or non-blocking mode is not directly relevant to whether all data will be successfully transmitted across each. Likewise, delays between read requests are not directly relevant to whether all data will be successfully transmitted, regardless of blocking vs. non-blocking I/O in your particular implementation of java.lang.Process.
Really, your efforts to probe blocking vs. non-blocking inter-process I/O are futile, because the I/O interface provided to your Java program is based on InputStream and OutputStream, which provide only for blocking I/O. Even if non-blocking I/O were involved at some low level of the implementation, I can't think of any way for your program to detect that.
With respect to your specific questions, however:
Are the PIPES of a process created through java runtime API are blocking or non-blocking?
They could be either, but they are more likely blocking because that better matches the interface presented to the Process user.
When I make A.java sleep after reading from the input stream, why all data is not received? (Assumption being that the PIPE is blocking)
I can only speculate, but the problem is likely in the external program. Possibly it goes to sleep when its output buffer fills, and nothing happens to wake it up. It might help to invoke myProcess.getOutputStream().close() if your Java program is not sending data to the external program. It's in any case a good idea to close that stream once you've written to it everything you're ever going to write.
Any non-programmatic way (i.e. I don't have to change the source code of java and etc!) to figure out if the PIPES of a process are blocking or non-blocking?
Potentially you could run the VM under strace or connect a native debugger to it, and analyze the VM's behavior that way. If you mean to do this from inside Java then the answer is a resounding "NO". Your Java program will see blocking behavior under all circumstances because the contracts of InputStream and OutputStream demand it.
I was making a big blunder and was completely off base. Posting this answer to clear things up (though I would like to delete the question altogether). I wanted to know if the communication channels between a C binary run from Java code are blocking or non-blocking. And I mentioned that the data was missing when I made my java code sleep after reading from the input stream of the created process (of C code). The data wasn't missing because of that. I had actually put a timer in Java code after which to terminate the process of the C binary. And since the PIPES are blocking, it wasn't able to receive all the data before the timer expired. I was misinterpreting this loss of data to mean that the PIPES were non-blocking. Confirmed this by running STRACE on the created C binary process. There were no EAGAIN errors on the write syscalls. My bad. But thank you very much to all for taking out the time to respond.