Develop Reference

parallel write and read from file

I want to have a central log file in my system, to which a certain application can write and read from.
The writes are for new data, and the reads will be to compare generated data to written data.
I would like this application to run in multiple instances at a time, which means I need to find a way to read diffs from the file, and write.
I have seen this code, but it's good for one go over the file and I don't see it working in multiple instances.
I'm building this app as a command line tool, so I'm thinking about creating a file for each instance and them migrating it to the "general" log file.
I'd like to hear inputs from the forum regarding the different approaches to this question.
What I'm worried about is having a few instances reading and writing from the same file and generating a lock.
This is the code I have found so far:
public class Tp {
public static void main(String[] args) throws IOException{
File f = new File("/path/to/your/file/filename.txt");
BufferedWriter bw = new BufferedWriter(new FileWriter(f));
BufferedReader br = new BufferedReader(new FileReader(f));
bw.write("Some text");
bw.flush();
System.out.println(br.readLine());
bw.write("Some more text");
bw.flush();
bw.close();
br.close();
}
}

You seem to be trying writing and reading the same file not only in one program but even within one thread. I do not believe this would be of benefit as during the program you know when/what you wrote so you can get rid of the whole I/O logic.
In the beginning try to write two different programs that run as separate processes. If need be, you can still try to bring them into the same JVM as separate threads.
Writing for sure is no problem, so the more interesting part is the reading logic. I'd probably implement this algorithm:
Loop until the program is terminated...
open the file, use skip() to jump to the location with new data
consume the existing data
remember how many bytes were read/remember the file size
close the file
wait until file has changed
Waiting for the file to change can be done by monitoring File.lastModified or File.length, or using the WatchService.
But be aware if you have multiple applications writing to the same file in parallel it can break any meaningful structure you have in the data. Log4j ensures parallel writes from within one application/multiple threads will go correctly into the file. If you need multiple processes running synchronized writes, consider logging into a database.

Cannot interrupt named pipe channel in Java on Windows

I have named pipe .\pipe\pipe1 on Windows I want to read from with Java.
From the documentation, FileChannel should be interruptible. Read should throw a ClosedByInterruptException if the reading thread is interrupted from another therad. This probably works for regular files, but I now have a named pipe.
My situation is like this:
RandomAccessFile raf = new RandomAccessFile("\\\\.\\pipe\\pipe1", "r");
FileChannel fileChannel = raf.getChannel();
// later in another thread "readThread"
fileChannel.read(buffer);
// outside the thread reading
readThread.interrupt();
The problem is that the call to interrupt will block and read will remain blocked until something to the named pipe is written so that read will stop blocking.
I need to be able to abort/cancel the read when nothing is written to the pipe while it is not closed yet.
Why does interrupting with the NIO classes not work? Is there a solution to this problem that does not involve busy-waiting or sleep with ready? What would be the best solution for this problem or is there a workaround?

I have not figured out a real solution to the question how to cancel the read. But I needed to adjust anyway and will now explain why. If you have anything to add to the original problem of the blocked read, you can post an additional answer.
A named piped could be treated like a file and opened separately for reading and writing with classic Java IO streams. However, a named piped is often used like a socket and as such, it requires a single file open. So I one could use Java IO streams like this:
RandomAccessFile raf = new RandomAccessFile("\\\\.\\pipe\\pipe1", "rws");
FileChannel fileChannel = raf.getChannel();
InputStream fis = Channels.newInputStream(fileChannel);
OutputStream fos = Channels.newOutputStream(fileChannel);
BufferedReader br = new BufferedReader(new InputStreamReader(fis));
PrintWriter pw = new PrintWriter(fos, true);
Now a problem one will later notice is this: If you write while reading, things will get locket up. It seems concurrent reading/writing is not possible, which is outline here.
To solve this, I used a ReentrantLock set to fair to switch from reading/writing. The reading thread is checking readyness and can be triggered with interrupt if one finishes writing and expects a response after this. If ready, the reading buffer is drained. If it is not ready, an interval can be scheduled or simulated for sporadically expected messages. This last part is not optimal, but actually works very well for my use case.
With this, one can build a solution where all threads can be orchestrated to terminate correctly with no blocking and minimal overhead.

what are the concern regarding simultaneous read and write to a file?

consider the following scenario:
Process 1 (Writer) continuously appends a line to a file ( sharedFile.txt )
Process 2 (Reader) continuously reads a line from sharedFile.txt
my questions are:
In java is it possible that :
Reader process somehow crashes Writer process (i.e. breaks the process of Writer)?
Reader some how knows when to stop reading the file purely based on the file stats (Reader doesn't know if others are writing to the file)?
to demonsterate
Process one (Writer):
...
while(!done){
String nextLine;//process the line
writeLine(nextLine);
...
}
...
Process Two (Reader):
...
while(hasNextLine()){
String nextLine= readLine();
...
}
...
NOTE:
Writer Process has priority. so nothing must interfere with it.

Since you are talking about processes, not threads, the answer depends on how the underlying OS manages open file handles:
On every OS I'm familiar with, Reader will never crash a writer process, as Reader's file handle only allows reading. On Linux, system calls a Reader can potentially invoke on the underlying OS are open(2) with O_RDONLY flag, lseek(2) and read(2) -- are known not to interfere with the syscalls that the Writer is invoking, such as write(2).
Reader most likely won't know when to stop reading on most OS. More precisely, on some read attempt it will receive zero as the number of read bytes and will treat this as an EOF (end of file). At this very moment, there can be Writer preparing to append some data to a file, but Reader have no way of knowing it.
If you need a way for two processes to communicate via file, you can do it using some extra files that pass meta-information between Readers and Writers, such as whether there are Writer currently running. Introducing some structure into a file can be useful too (for example, every Writer appends a byte to a file indicating that the write process is happening).
For very fast non-blocking I/O you may want consider memory mapped files via Java's MappedByteBuffer.

The code will not crash. However, the reader will terminate when the end is reached, even if the writer may still be writing. You will have to synchronize somehow!

Concern:
Your reader thread can read a stale value even when you think another writer thread has updated the variable value
Even if you write to a file if synchronization is not there you will see a different value while reading

Java File IO and plain files were not designed for simultaneous writes and reads. Either your reader will overtake your writer, or your reader will never finish.
JB Nizet provided the answer in his comment. You use a BlockingQueue to hold the writer data while you're reading it. Either the queue will empty, or the reader will never finish. You have the means through the BlockingQueue methods to detect either situation.

Reusing one BufferedWriter instance for one file along application lifetime

I have a multi-threading application where some of my threads should read data from queue and write them to a file. The problem here is that I am confusing should I create new BufferedWriter instance every time when one of my threads reads value from queue and writes it to same file or I can have just one BufferedWriter instance and flush() every time. One problem in second solution is that I should detect when I should close the BufferedWriter without using Java 7 perfect solution for closing resources in try-catch block.
Does the second solution solves some performance issues?
What are best practices on this?

I'd recommend using one BufferedWriter for writing to the file that is shared by all threads. For the sake of performance, the BufferedWriter should be kept open until the application decides that there is no more output. (Opening and closing files is relatively expensive.)
You also need to have the application threads use some kind of locking (e.g. synchronize on the BufferedWriter) to ensure that they don't try to write at the same time. The BufferedWriter class is not thread-safe.
The try/finally or try-with-resources approach to managing file resources is important in cases where you are opening lots of files. If you are only dealing with one file, and it needs to be open for the entire duration of the application, then resource management is not so critical. (But you do need to make sure that you either close or flush before the application exits.)
But I think BufferedWriter is thread-safe, because underlying implementation of write() methods using synchronized blocks
Well in that sense, yes it is. However, that assumes that when your thread writes data, it does it in a single write(...) call.
Note also that BufferedWriter is not specified to be thread-safe, even if it is thread-safe in the current implementation.

A BufferedWriter should ever lead to one file/Writer/OutputStream; if you have many targets you will need many buffers. If you want to write to the same file from multiple threads you will need to synchronize on the earliest bit; you can synchronized on the BufferredWriter if you don't have more high-level constructs that the character stream. If you synchronize on the BufferedWriter, you will not need to call flush() after the end of each access.

Java Best Practices - Need to write to file constantly - BufferedWriter?

So I have a java process that needs to constantly append a new line to a file every 100 milliseconds. I am currently use BufferedWriter for this, but from what I have read, the BufferedWriter object should always be .close()'d when its finished.
If I did this, I would have to create a new BufferedWriter object every few milliseconds, which is not ideal. Are there any issues with creating one static BufferedWriter, and just .flush()'ing it after every write?
Finally, is BufferedWriter the best class to use for this, if performance is a concern? Are there any viable alternatives?
Thanks!

The BufferedWriter should be closed when it's finished. If you're doing something like logging, it's entirely acceptable to hold an open writer in the object responsible for the logging and then close it at the end of the run (or whenever you roll over to a new log file).
What you shouldn't do is simply open the writer and then discard the reference without closing it; this can leak resources, and in the case of something with buffering, might lose the last part of the output.

Are there any issues with creating one static BufferedWriter, and just .flush()'ing it after every write?
There is nothing wrong with a single, long-lived BufferedWriter. In fact it is a good idea. (Whether you use a static or something else is a different issue ... but that design decision does not impact on functionality and performance.)
Calling flush after each write is more questionable from a performance perspective. It will cause your application to make a lot of write syscalls ... which is expensive. I would only do that if you need the logging to be written immediately. The alternatively it to flush on a timer ... or not flush at all, and rely on the (final) close of the BufferedWriter to flush any outstanding data.
But either way, a long-lived BufferedWriter that you flush is likely to be better than creating, writing and closing lots of BufferedWriter objects.