I'm developing disk cache (multithread). Cache can have many users at once. That is why I can write some rules:
1) Cache can't edit (or delete) file when client reads it.
2) When cache is editing file, clients should wait for the end of editing (and after read the file).
I need to organize this lock strategy with the help of java.
I read about synchronizatioan (synchronized block and java.util.concurrent.Locks) and as I understood it can't help here.
I tried to understand the FileLock. But when client reads file, cache lock can abort reading. And if clients will lock files before reading, there will be long sequence of client to read.
I need for advice how to organize it (maybe another ways).
UPDATE
public void write(InputStream is) throws FileNotFoundException, IOException {
File file = new File("path");
try (FileOutputStream fos = new FileOutputStream(file);
FileChannel filechannel = fos.getChannel();
FileLock lock = filechannel.lock(0, Long.MAX_VALUE, false)) {
// writing....
}
}
public void read(OutputStream osToClient) throws IOException {
File file = new File("path");
try (FileInputStream fis = new FileInputStream(file);
FileChannel filechannel = fis.getChannel();
FileLock lock = filechannel.lock(0, Long.MAX_VALUE, true)) {
IOUtils.copy(fis, osToClient);
}
}
You should probably not build this yourself unless you do it for fun or for a school assignment. However, except that there are warnings about portability (so it may not work the way you expect on all platforms) FileLock should do the job. When you are reading, first get a shared lock on the file, read the file and release the lock when done, ideally in a try-with-resources block. When you are writing, get an exclusive lock (shared=false) instead. There can be multiple readers but only one writer, and when there is a writer there can be no readers.
Related
I want to check if the file exist on server on multithread environment and if exists return that file content diractly or download from my s3 service server.
My code like this:
final Object lock = new Object();
File file = new File("/file/path");
if (file.exists()) {
return FileUtils.readFileToByteArray(file);
} else {
byte[] bytes = this.downloadFileFromRemoteServer();
if (!file.exists()) {
synchronized (lock) {
if(!file.exists()) {
FileUtils.writeByteArrayToFile(tempFile, bytes);
}
}
}
tempFile.renameTo(file);
return bytes;
}
The above code similar java double checked locking, is method file.exists() behavior like volatile keyword? And pseudo code correctly?
File.exists() checks the file existence with the file-system, and so it should behave like a volatile, so you are covered there
Some issues though -
1) As soon as a thread sees that the file doesn't exist, it starts downloading the file, which is time consuming, so its likely that other threads will also come and start downloading the same file. So the download part should be moved inside the lock
2) You're renaming the temp file outside the lock. A thread may get to that point without creating/writing-to a temp file. Should move the rename inside the lock as-well
Since IO has much more overhead than locking, I think the above 2 steps would be beneficial
You are overly cautious: since you are writing to a temp file, there is no risk of overwriting an existing file, which carries a possibility of reading a half-written file: your reads are going to be consistent.
The only issue that your code is protecting against is writing the same downloaded content into multiple temporary files, which is not much of a performance problem in comparison to multiple downloads, which would happen anyway.
I would simplify your code as follows:
File file = new File("/file/path");
if (!file.exists()) {
byte[] bytes = this.downloadFileFromRemoteServer();
File tempFile = File.createTempFile(...);
FileUtils.writeByteArrayToFile(tempFile, bytes);
tempFile.renameTo(file);
}
return FileUtils.readFileToByteArray(file);
I am creating a program which needs to read from a file that is still being written.
The main question is this: If the read and write will be performed using InputStream and OutputStream classes running on a separate thread, what are the catches and edge cases that I will need to be aware of in order to prevent data corruption?
In case anyone is wondering if I have considered other, non-InputStream based approach, the answer is yes, I have but unfortunately it's not possible in this project since the program uses libraries that only works with InputStream and OutputStream.
Also, several readers have asked why this complications is necessary. Why not perform reading after the file has been written completely?
The reason is efficiency. The program will perform the following
Download a series of byte chunks of 1.5MB each. The program will receive thousands of such chunks that can total up to 30GB. Also, chunks are downloaded concurrently in order to maximize bandwidth, so they may arrive out of order.
The program will send each chunk for processing as soon as they have arrived. Please note that they will be sent for processing in order. If chunk m arrives before chunk m-1 does, they will be buffered on disk until chunk m-1 arrives and is sent for processing.
perform processing of these chunks starting from chunk 0 up to chunk n until every chunks has been processed
Resend the processed result back.
If we are to wait for the whole file to be transferred, it will introduce a huge delay on what is supposed to be a real-time system.
Use a RandomAccessFile. Via a getChannel or such one could use a ByteBuffer.
You will not be able to "insert" or "delete" middle parts of the file. For such a purpose your original approach would be fine, but using two files.
For concurrency: to keep in synch you could maintain one single object model of the file, do changes there. Only the pending changes need to be kept in memory, other hierarchical data could be reread and reparsed as needed.
So your problem (as you've cleared it up now) is that you can't start processing until chunk#1 has arrived, and you need to buffer every chunk#N (N > 1) until you can process them.
I would write each chunk to their own file and create a custom InputStream that will read every chunk in order. While downloading the chunkfile would be named something like chunk.1.downloading and when the whole chunk is loaded it will be renamed to chunk.1.
The custom InputStream will check to see if file chunk.N exists (where N = 1...X). If not, it will block. Each time a chunk has been downloaded completely, the InputStream is notified, it will check if the downloaded chunk was the next one to be processed. If yes, read as normally, otherwise block again.
You should use PipedInputStream and PipedOutputStream:
static Thread newCopyThread(InputStream is, OutputStream os) {
Thread t = new Thread() {
#Override
public void run() {
byte[] buffer = new byte[2048];
try {
while (true) {
int size = is.read(buffer);
if (size < 0) break;
os.write(buffer, 0, size);
}
is.close();
os.close();
} catch (IOException e) {
e.printStackTrace();
} finally {
}
}
};
return t;
}
public void main(String[] args) throws IOException, InterruptedException {
ByteArrayInputStream bi = new ByteArrayInputStream("abcdefg".getBytes());
PipedInputStream is = new PipedInputStream();
PipedOutputStream os = new PipedOutputStream(is);
Thread p = newCopyThread(bi, os);
Thread c = newCopyThread(is, System.out);
p.start();
c.start();
p.join();
c.join();
}
How can I lock the file in JVM in such way that other non JVM processes can't get access for write access?
I need to read the file and while reading I want to be sure that other processes do not modify the file.
I tried creating FileInputStream and it does lock the file for deleting but it doesn't prohibit the modification of file.
I also tried RandomAccessFile:
RandomAccessFile raf = new RandomAccessFile(file, "rw");
InputStream is = Channels.newInputStream(raf.getChannel());
but it also doesn't prevent modifications.
PS: Further in the code I need InputStream
Unfortunately, this is not something Java can do - perhaps largely because it is supported in different ways on different platforms and Java needs to maintain cross platform compatibility.
I assume, from your question for example, that you are on Windows as under Linux the above code would not even prevent file deletion.
There is some detailed information on file locking at http://en.wikipedia.org/wiki/File_locking which explains the issue.
Have you tried to use FileLock? The usage will be like this snippet:
FileInputStream in = new FileInputStream(file);
try {
java.nio.channels.FileLock lock = in.getChannel().lock();
try {
Reader reader = new InputStreamReader(in, charset);
//Other actions...
} finally {
lock.release();
}
} finally {
in.close();
}
I wrote a java application that accesses a file while other Processes in other VMs try to do the same. Therefore I use the FileLock class:
FileOutputStream fos = new FileOutputStream(filePath,append);
FileChannel f = fos.getChannel();
FileLock lock;
while ((lock = f.tryLock()) == null){
try {
Thread.sleep(100);
} catch (InterruptedException ex) {
Logger.getLogger(Util.class.getName()).log(Level.SEVERE, null, ex);
}
}
OutputStreamWriter out = new OutputStreamWriter( new FileOutputStream(filePath,append));
out.write(textToWrite);
out.close();
lock.release();
All works fine on Mac OSX, but when I run the code on Windows 7 it throws an IOException at the line
out.close();
, when trying to flush.
java.io.IOException: The process cannot access the file because another process has locked a portion of the file
at java.io.FileOutputStream.writeBytes(Native Method)
As far as I understand from How does FileLock work?, the actual obtaining of the lock with
f.tryLock()
forbids me to access it since another process (this one apparently) has exclusive lock.
Now that strikes me as a paradoxon - how am I to obtain an exlusive lock to enable me to write to the file without danger of other processes messing with it at the same time when the actual act of obtaining the lock hinders me to do so?
And consequently why does it work on Mac OS and not on windows? I know from the JavaDocs that there are OS specific differences and difficulties with the FileLock class, but surely not with respect to its designed-for functionality.
Since this can't be the case, I am doing something wrong and this is where I ask for your help.
Thx,
M
There is no file locking on UNIX.: http://www.coderanch.com/t/551144/java/java/File-lock-doesn-prevent-threads. In fact, on UNIX, you can delete a file from under a process and it may not even notice...
So you need to use a lock file that you can check exists.
Paradoxically your code is working on Windows but not on UNIX (i.e. Mac OS), the exception should be the expected result of trying to write to a file that is locked by another process.
So I try to locked the file to read it, but I got IOException, any idea why?
public static void main(String[] args){
File file = new File("C:\\dev\\harry\\data.txt");
FileReader fileReader = null;
BufferedReader bufferedReader = null;
FileChannel channel = null;
FileLock lock = null;
try{
channel = new RandomAccessFile(file, "rw").getChannel();
lock = channel.lock();
fileReader = new FileReader(file);
bufferedReader = new BufferedReader(fileReader);
String data;
while((data = bufferedReader.readLine()) != null){
System.out.println(data);
}
}catch(IOException e){
e.printStackTrace();
}finally{
try {
lock.release();
channel.close();
if(bufferedReader != null) bufferedReader.close();
if(fileReader != null) fileReader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
and I got this error IOException: The process cannot access the file because another process has locked a portion of the file
Might as well add this as an answer instead of a comment.
If you use the FileLock API you need to use the corresponding NIO file apis.
Reproducing my answer from here (in case it gets deleted), and adding Jeff Foster's feedback:
Considering that an instance of the OverlappingFileLockException exception is thrown, it appears that another thread in the same process is attempting to lock on the same file. This is not a conflict between A and B, but rather a conflict within B, if one goes by the API documentation on the lock() method and when the condition under which it throws OverlappingFileLockException:
If a lock that overlaps the requested
region is already held by this Java
virtual machine, or if another thread
is already blocked in this method and
is attempting to lock an overlapping
region of the same file
The only solution to prevent this, is to have any other thread in B prevented from acquiring a lock on the same file, or the same overlapping region in the file.
The IOException being thrown has a bit more interesting message. It probably confirms the above theory, but without looking at the entire source code, I cannot confirm anything. The lock method is expected to block until the exclusive lock is acquired. If it was acquired, then there ought to be no problem in reading from the file. Except for one condition. If the file has already been opened (and locked) by the same JVM in a different thread, using a File object (or in other words, a second/different file descriptor), then the attempted read on the first file descriptor will fail even if the lock was acquired (after all, the lock does not lock out other threads).
An improved design, would be to have a single thread in each process that acquires an exclusive lock on the file (while using a single File object, or a single file descriptor) for only a certain amount of time, perform the required activity in the file, and then release the lock.
As Jeff has pointed out, using the NIO APIs would probably result in resolution of the problem. This is entirely due to the possibility of the FileReader API opening a new file descriptor, which is different from the one that the lock is obtained on.
Maybe what you want is something more like:
FileInputStream fis = new FileInputStream(file);
channel = fis.getChannel();
channel.lock();
bufferedReader = new BufferedReader(new InputStreamReader(fis));