In my Java app, on Linux, I need to periodically read some text files that change often.
(these text files are updated by a separate app).
Do I need to be concerned about the rare case when attempting to read the file at the exact moment it is being updated? If so, how can I guarantee that my reads always return without failing? Does the OS handle this for me, or could I potentially read 1/2 a file?
thanks.
The OS can help you achieve consistent reads, but it requires that both apps are written with this in mind.
In a nutshell, you open the file in your java app with exclusive read/write permission - this ensures that no one else, including your other app is modifying the file while you are reading it. The FileLock class can help you ensure you have exclusive access to a file.
Your other app will then periodically try to write to the file. If it does this at the same time you are reading the file, then access will be denied, and the other app should retry. This is the critical part, since if the app doesn't expect the file to be unavailable and treats this as a fatal error condition, the write will fail, and app doesn't save the data and may fail/exit etc.
If the other app must always be able to write to the file, then you have to avoid using exclusive reads. Instead, you have to try to detect an inconsistent read, such as by checking the last modified timestamp when you start reading, and when you finish reading. If the timestamps are the same, then you are good to go and have a consistent read.
Yes, you need to worry about this.
No, your reads shouldn't "fail" AFAIK, unless the file is momentarily being locked, in which you can catch the exception and try again after a brief pause. You might certainly, though, get more or less data than you expected.
(If you post code we might be able to comment more accurately on what'll happen.)
Related
EDIT : Well, I'm back a bunch of months later, the lock mechanism that I was trying to code doesn't work, because createNewFile isn't reliable on the NFS. Check the answer below.
Here is my situation : I have only 1 application which may access the files, so I don't have any constraint about what other applications may do, but the application is running concurrently on several servers in the production environment for redundancy and performance purposes (a couple of machines are hosting each a couple of JVM with our apps).
Basically, what I need is to put some kind of flag in a folder to tell the other instances to leave this folder alone as another instance is already dealing with it.
Many search results are telling to use FileLock to achieve this, but I checked the Javadoc, and from my understanding it will not help much, since it's using the hosting OS's locking possibilities. So I doubt that it will help much since there are different hosting machines.
This question covers a similar subject : Java file locking on a network , and the accepted answer is recommending to implement your own kind of cooperative locking process (using the File.createNewFile() as asked by the OP).
The Javadoc of File.createNewFile() says that the process is atomically creating the file if it doesn't already exist. Does that work reliably in a network file system ?
I mean, how is it possible with the potential network lag to do both existence check and creation simultaneously ? :
The check for the existence of the file and the creation of the file if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the file.
No, createNewFile doesn't work properly on a network file system.
Even if the system call is atomic, it's only atomic regarding the OS, and not over the network.
Over the time, I got a couple of collisions, like once every 2-3 months (approx. once every 600k files).
The thing that happens is my program is running in 6 separates instances over 2 separate servers, so let's call them A1,A2,A3 and B1,B2,B3.
When A1, A2, and A3 try to create the same file, the OS can properly ensure that only one file is created, since it is working with itself.
When A1 and B1 try to create the same file at the same exact moment, there is some form of network cache and/or network delays happening, and they both get a true return from File.createNewFile().
My code then proceeds by renaming the parent folder to stop the other instances of the program from unnecessarily trying to process the folder and that's where it fails :
On A1, the folder renaming operation is successful, but the lock file can't be removed, so A1 just lets it like that and keeps on processing new incoming folders.
On B1, the folder renaming operation (File.renameTo(), can't do much to fix it) gets stuck in a infinite loop because the folder was already renamed (also causing a huge I/O traffic according to my sysadmin), and B1 is unable to process any new file until the program is rebooted.
The check for the existence of the file and the creation of the file if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the file.
That can be implemented easily via the open() system call or its equivalents in any operating system I have ever used.
I mean, how is it possible with the potential network lag to do both
existence check and creation simultaneously ?
There is a difference between simultaneously and atomically. Java doc is not saying anything about this function being a set of two simultaneous actions but two actions designed to work in atomic way. If this method is built to do two operations atomically than means file will never be created without checking file existence first and if file gets created by current call then it means there were no files present and if file doesn't get created that means there was already a file by that name.
I don't see a reason to doubt function being atomic or working reliably despite call being on network or local disk. Local call is equally unreliable - so many things can go wrong in an IO.
What you have to doubt is when trying to use empty file created by this function as a Lock as explained D-Mac's answer for this question and that is what explicitly mentioned in Java Doc for this function too.
You are looking for a directory lock and empty files working as a directory lock ( to signal other processes and threads to not touch it ) has worked quite well for me provided due care is taken to write logic to check for file existence,lock file clean up and orphaned locks.
If I have multiple threads that use log4j to write to a single log file, and I want another thread to read it back out, is there a way to safely read(line by line) those logs such that I always read a full line?
EDIT:
Reason for this is I need to upload all logs to a central location and it might be logs that are days old or those that are just being written
You should use a read write lock.
Read locks can be held by multiple users if there is no one writing to the file, but a write lock can only be held by 1 thread at a time no matter what.
Just make sure that as your writing thread is done writing, it releases the readwritelock to allow the reading threads to read. Likewise, always release the read lock when the reader are done reading so log4j can continue to write
Check out
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReadWriteLock.html
However, coming to think of it, what is your purpose for this? If you simply want to monitor your logs, you should use a different solution rather than having a monitor thread within the same application. Seems to not make sense. If the data is available within the application / service, why pass it off to a file and read it right back in?
It is going to be a pain if you need to implement what you are doing, especially you have to deal with file rolling.
For your specific requirement, there are better choices:
If the location you are going to be backed up can be directly written (i.e. mounted in your file system), it is better to simply set your file rolling to write to that backup directory; or
Make use of log management tools like Splunk to monitor and manage your log files (so that you don't even need to copy to that backup directory); or
Even you need to do the backup all by yourself, you don't need to (and have no reason to) do it in a separate thread. Trying to write a shell script monitoring your log directory, and make use of tools like rsync or write similar logic by yourself, to do the upload only for files that are not matching in local and remote location.
I am looking to force synchronisation to disk after files are written at certain points in my application. Since it runs on Linux, I could get away with just running
Runtime.getRuntime().exec("sync");
However, I would rather not introduce Linux specific system calls and would rather use
java.io.FileDescriptor#sync();
However, I use Apache VFS to perform operations on the local file system and to my knowledge it does not provide access to the underlying file descriptor. But do I need access to the actual file descriptor that was just written to to force synchronization? Could I not just use any FileDescriptor to call sync for the same effect, for example
FileDescriptor.in.sync();
Would that be a valid approach, and would the results match that of calling sync in Linux?
Just in case anyone knows if / how it is possible to get access to the underlying FileDescriptor in VFS, it would be useful to know as well.
Edit: it appears that
FileDescriptor.in.sync();
does not want to work on Linux (although it works on my Windows machine when run from Eclipse), but
new FileOutputStream(new File("anyfile")).getFD().sync();
definitely works and the results of calling this match the results of calling the Linux sync command directly. However, it involves opening and closing a redundant file output stream, so it's not exactly ideal. Any other reason this might be a bad idea, as it does seem to work? Is there some other way to get a FileDescriptor that can be used to sync?
I investigates such issues some time ago: Question 1, Question 2.
In Linux, a java.io.FileDescriptor#sync call ensures that the modified data of the file associated with the descriptor is sent to the disk. (That cheap disk tend to skip the write and only place the data in an unreliable (aka no NVRAM) write cache is a different/additional problem.)
It does not guarantee that also modified data of other files is written back. This is just not in the contract of sync or of the underlying fsync POSIX function.
However, in certain circumstances (e.g. ext3 in data=ordered mode), an fsync on a file writes back up modified data of the file system. This is really fun because this may create significant latencies just because some other application has created a ton of dirty blocks.
I'm looking for a "trick" or an "hack" to be certain that a file has been persisted on a remote disk, passing through vmware cache, NAS cache, etc.
Flushing and closing a FileOutputStream is not enough. I think Channel.force(true) is neither.
I'm thinking about something like these:
write the file and read back the file
write the file, check timestamp, rename the file, check for a different timestamp
write the file with "wrong content", overwrite with the original content, read it back and check the content
maybe someone had the same problem and found a solution.
My requirement is not to lose data. The java application works in this way:
accept a file from a remote source
add a digital signature and a certified timestamp creating a new file. If this file is lost it cannot be recreated in any way.
write this file to the storage
mark the file as signed on the database
tell the remote side that everything is ok
Tonight we had a crash and three transactions failed after step 5 but before the data was actually flushed to the remote store. So the database says that everything is fine, the remote side was told the same but 15 seconds of signed data was lost. And this is no good.
The correct solution could be to do a "synch mount" of the remote file-system. But this is not going to happen in a short time. Even in this case I do not completely trust this scenario given that the app is running on a VMWare server.
So I'd like to have a "best effort hack" to prevent (mitigate) incidents like this one.
Let's start with one assumption: you cannot guarantee any single write to any single disk. There are just too many layers of software and hardware between your write and the disk platter. And even if you could guarantee the write, you cannot guarantee that the data will be readable. It's possible that the disk will crash between the write and the read.
The only solution is redundancy, either provided by a framework (eg, RDMS) or your app.
When you receive and sign the file, you need to send it to multiple destinations on different physical hosts, and wait for them to reply that they saved the file. One of them might crash. Two of them might crash. How important the data is will determine how many remote hosts you need.
Incidentally, redundancy also applies to your database. The fact that a transaction committed does not mean that you'll be able to recover it after a database crash (although DBMS engineers have a lot more experience than either you or I in ensuring writes, all of it depends on a sysadmin who understands things like "logs and datafiles must reside on separate physical drives). I strongly recommend that you (redundantly) store enough metadata along with the file to be able to reconstruct the database entry.
One of our clients is using some Novel security software that sometimes locks some .class files that our software creates. This causes some nasty problems for them when this occurs and I am trying to research a workaround that we could add to our error handling to address this problem when it comes up. I am wondering are there any calls in the java api that can be used to detect if a file is locked, and if it is, unlock it.
Before attempting to write to a file, you can check if the file is writable by your java application using File.canWrite(). However, you still might run into an issue if the 3rd party app locks the file in between your File.canWrite() check and when your application actually attempts to write. For this reason, I would code your application to simply go ahead and try to write to the file and catch whatever Exception gets thrown when the file is locked. I don't believe there is a native Java way to unlock a file that has been locked by another application. You could exec a shell command as a privileged user to force things along but that seems inelegant.
File.canWrite() has the race condition that Asaph mentioned. You could try FileChannel.lock() and get an exclusive lock on the file. As long as the .class is on your local disk, this should work fine (file locking can be problematic on networked disks).
Alternatively, depending on how the .class name is discovered, you could create a new name for your .class each time; then if the anti-virus software locks your initial class, you can still create the new one.