I'm working on a multithreaded server in Java.
The server monitors a directory of files. Clients can ask the server:
to download a file from the server directory
to upload a new version of an already existing file to the server, overwriting the old version in the server directory.
To do the transfers, I'm planning to use FileChannels and SocketChannels, using the methods transferFrom and transferTo. According to the documentation, these two methods are thread safe.
The thing is that a single call to these two function could not be sufficient to read/write the file entirely.
The problem arises if there are more than one request on the same file at the same time. In this scenario, multiple threads could be doing read/write operations on the same file. Now, the single calls to transferFrom/transferTo are thread safe, according to the Java documentation. But a single call to these two functions could not be sufficient to read/write the file entirely. If thread A is replying to a download request and thread B is replying to an upload request referring to the same file, it could happen that:
Thread A starts reading from the file
In thread A, for some reason the read call returns before the EOF
Thread B overwrites the entire file with a single write call
Thread A continues reading from the file
In this case, the downloading client receives a portion of the old version and a portion of the new version.
To solve this I think I should be using some sort of locking, but I'm not sure how to do it in an efficient way. I could create two synchronized methods for reading and writing, but that creates obviously too much contention.
The best solution I have in mind is to use lock striping. Before doing any read/write operation, an hash based on the filename is calculated. Then, the lock in position lockArr[hash % numOfLocks] is acquired.
I think also that I should be using ReadWriteLocks, since multiple simultaneous reads should be allowed.
Now, this is my analysis of the problem and I could be completely wrong. Is there any better solution to this?
Locking means that somebody has to wait for somebody else -- not the best solution.
When the client uploads a file, you should write it out to a temp file on the same disk (usually in the same directory), and then when the file upload is done:
Rename the old version to a temporary name. Any current readers should be forced to close the old one, re-open the temp version, and seek to the correct position.
Rename the uploaded file to the target file name.
Delete the temp version of the old file when any readers are done with it.
In a typical implementation, you'd need a centralized class (lets call it ConcurrentFileAccessor) to manage the interactions between threads.
Readers would need to register with this class, and synchronize on some object during the actual read operation. When an upload completes, the writer would have to claim all those locks to block reads, close all the read files, rename the old version, reopen, seek, and then release them to allow the readers to continue.
Related
I am using ftpClient.retrieveFile() to download files from an FTP Server while another thread is scanning the directory constantly for files to process. I am wondering if this is dangerous? Could it be that a file is not finished downloading and is processed by the other thread? Should I be using a .temp suffix to save the temporary file and rename it after the transaction is finished?
Files are, in general, visible to other processes or threads as soon as you create them. So your second thread could see and process a file before you have completed writing to it. The correct practice is to use either a temporary extension (like the .temp you have mentioned) or a temporary directory.
In your case, the most appropriate thing to do would be to use some synchronization mechanism so that the second thread blocks when there are no files to process and the first thread notifies the second when a file finishes downloading. Java supports these operations with the wait() and notify() methods of the Object class.
EDIT : Well, I'm back a bunch of months later, the lock mechanism that I was trying to code doesn't work, because createNewFile isn't reliable on the NFS. Check the answer below.
Here is my situation : I have only 1 application which may access the files, so I don't have any constraint about what other applications may do, but the application is running concurrently on several servers in the production environment for redundancy and performance purposes (a couple of machines are hosting each a couple of JVM with our apps).
Basically, what I need is to put some kind of flag in a folder to tell the other instances to leave this folder alone as another instance is already dealing with it.
Many search results are telling to use FileLock to achieve this, but I checked the Javadoc, and from my understanding it will not help much, since it's using the hosting OS's locking possibilities. So I doubt that it will help much since there are different hosting machines.
This question covers a similar subject : Java file locking on a network , and the accepted answer is recommending to implement your own kind of cooperative locking process (using the File.createNewFile() as asked by the OP).
The Javadoc of File.createNewFile() says that the process is atomically creating the file if it doesn't already exist. Does that work reliably in a network file system ?
I mean, how is it possible with the potential network lag to do both existence check and creation simultaneously ? :
The check for the existence of the file and the creation of the file if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the file.
No, createNewFile doesn't work properly on a network file system.
Even if the system call is atomic, it's only atomic regarding the OS, and not over the network.
Over the time, I got a couple of collisions, like once every 2-3 months (approx. once every 600k files).
The thing that happens is my program is running in 6 separates instances over 2 separate servers, so let's call them A1,A2,A3 and B1,B2,B3.
When A1, A2, and A3 try to create the same file, the OS can properly ensure that only one file is created, since it is working with itself.
When A1 and B1 try to create the same file at the same exact moment, there is some form of network cache and/or network delays happening, and they both get a true return from File.createNewFile().
My code then proceeds by renaming the parent folder to stop the other instances of the program from unnecessarily trying to process the folder and that's where it fails :
On A1, the folder renaming operation is successful, but the lock file can't be removed, so A1 just lets it like that and keeps on processing new incoming folders.
On B1, the folder renaming operation (File.renameTo(), can't do much to fix it) gets stuck in a infinite loop because the folder was already renamed (also causing a huge I/O traffic according to my sysadmin), and B1 is unable to process any new file until the program is rebooted.
The check for the existence of the file and the creation of the file if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the file.
That can be implemented easily via the open() system call or its equivalents in any operating system I have ever used.
I mean, how is it possible with the potential network lag to do both
existence check and creation simultaneously ?
There is a difference between simultaneously and atomically. Java doc is not saying anything about this function being a set of two simultaneous actions but two actions designed to work in atomic way. If this method is built to do two operations atomically than means file will never be created without checking file existence first and if file gets created by current call then it means there were no files present and if file doesn't get created that means there was already a file by that name.
I don't see a reason to doubt function being atomic or working reliably despite call being on network or local disk. Local call is equally unreliable - so many things can go wrong in an IO.
What you have to doubt is when trying to use empty file created by this function as a Lock as explained D-Mac's answer for this question and that is what explicitly mentioned in Java Doc for this function too.
You are looking for a directory lock and empty files working as a directory lock ( to signal other processes and threads to not touch it ) has worked quite well for me provided due care is taken to write logic to check for file existence,lock file clean up and orphaned locks.
I am working on Concurrent File Download process, but not sure what approach to take.
About:
An application bundles bunch of files together into a zip file. The files are usually available on the hard drive in a common location (for example /tmp). However there are cases when files are not there and need to be downloaded from a remote http server.
Question:
How can I download multiple files concurrently and ensure that NO other thread (bundling files) downloads the same file at the same time?
More over, how can I ensure that in case of multiple applications running at the same time (remember that the files are all located in a common location), no instance of the application downloads the same file at the same time?
Please describe strategy and perhaps a way to implement it. Perhaps solution the above issue already exists.
Thank you!
You could use a queue or db to download needed files, just keep a
'status' column and a thread will mark the file as 'fetching'. When
done it will set as 'done'. Keep a last change timestamp and if the
file is downloading for a long time, stop or restart download.
Using a database for this file queue might ensure that other apps
don't fetch the same file multiple times (maybe persist download
etc;). Also you can have multiple downloads running and the db could
be used to track download speed, progress, etc;
In the future your question should be formatted with specific code, a specific problem. Your question is very broad and presents a discussion (better suited for chat) vs a single answer someone else might use.
Here is a possible strategy:
In case of a single app: have some sort of dispatcher thread which reads work from a queue (could be some persisted queue too like DB table or other) and spawns new threads for each item that was read from the queue. By read I mean, read and remove from the queue.
Have that queue stored in a shared DB (or any shared storage). In this case there may be a separate single dispatcher app which just reads works or work portions from the DB, and gives work to worker apps. So each worker app asks the dispatcher app for work, this ensures that only the dispatcher app reads from the DB (or the other central storage you decide to use). This on its turn eliminates the need to sync your DB (permanent storage) access.
If I have multiple threads that use log4j to write to a single log file, and I want another thread to read it back out, is there a way to safely read(line by line) those logs such that I always read a full line?
EDIT:
Reason for this is I need to upload all logs to a central location and it might be logs that are days old or those that are just being written
You should use a read write lock.
Read locks can be held by multiple users if there is no one writing to the file, but a write lock can only be held by 1 thread at a time no matter what.
Just make sure that as your writing thread is done writing, it releases the readwritelock to allow the reading threads to read. Likewise, always release the read lock when the reader are done reading so log4j can continue to write
Check out
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReadWriteLock.html
However, coming to think of it, what is your purpose for this? If you simply want to monitor your logs, you should use a different solution rather than having a monitor thread within the same application. Seems to not make sense. If the data is available within the application / service, why pass it off to a file and read it right back in?
It is going to be a pain if you need to implement what you are doing, especially you have to deal with file rolling.
For your specific requirement, there are better choices:
If the location you are going to be backed up can be directly written (i.e. mounted in your file system), it is better to simply set your file rolling to write to that backup directory; or
Make use of log management tools like Splunk to monitor and manage your log files (so that you don't even need to copy to that backup directory); or
Even you need to do the backup all by yourself, you don't need to (and have no reason to) do it in a separate thread. Trying to write a shell script monitoring your log directory, and make use of tools like rsync or write similar logic by yourself, to do the upload only for files that are not matching in local and remote location.
I have multiple applications running in one virtual machine. I have multiple virtual machines running on one server. And I have multiple servers. They all share a file using a shared folder on linux. The file is read and written by all applications. During the write process no application is allowed to read this file. The same for writing: If an application is reading the file no application is allowed to write it.
How do I manage to synchronize the applications so they will wait for the write process to finish before they read, and vice versa? (the applications inside a vm have to be synchronized and also applications across servers)
Curent implementation uses "file semaphores". If the file is about to be written the application tries to "acquire" the semaphore by creating an additional file (lets name it "file.semaphore") in the shared folder. If the "file.semaphore" file already exists this means the semaphore is already locked by a different application. This approach has the problem that I cannot make sure that the "file exists"-test and "create file"- operation are executed atomic. This way it is possible that two applications test for the "file.semaphore" file, see it does not exist and try to create the file at the same time.
You can use NIO locking capabilities. See FileChannel#lock().
However, this will work only if underlying filesystem supports locking over the network. Recent NFS should support it. Probably, Samba supports them too, but can’t say for sure.
See article for example.
Have a look at the Javadocs for the createNewFile() method - it specifically states that creating files is not a reliable method for synchronization, and recommends the FileLock class instead (it's another package in java.nio.channels so is essentially the same as what Ivan Dubrov is suggesting).
This would imply that your identification of the problem is accurate, and no amount of playing around will solve this with traditional file creation. My first thought was to check the return code from createNewFile(), but if the Javadocs say it's not suitable then it's time to move on.
Need to combine file locking for protection between JVM's with synchronization within threads of a given JVM. See the answer by cyber-monk here
I am also trying to determine the best way to solve this problem for a similar situation (less participating processes, but still same underlying problem). If you haven't been able to employ the file locking scheme suggested by Ivan (e.g. system|language|network service does not support it), maybe you could designate one of the participants as a referee. All participants write unique semaphores, call them "participant#.request" when they want the file. The referee polls the file system for these semaphores. When he sees one, he writes back "participant#.lock", and deletes the request. If he happens to see multiple at the "same time" he selects one at random (or first by file modification time) and deletes only their request. Then, the participant issued the lock knows they can access the file safely. When the participant is done with the file, they delete their own lock. While there is a lock in place, no other locks are issued by the referee. Any requests that are present after the user deletes their lock could be served a new lock without issuing a new request, so you could have the other users poll for their lock after sending the request. Probably this is what the locking mechanism is doing anyway, except maybe for the ability to manage the lock as a queue that comes with requests being processed in the order they are received (i.e. if the referee uses modification time). Also, since you're in charge of the referee you could set timeouts to locks, allowing him issue timeout semaphores to the process that is hogging the file and then remove the lock (hoping of course that if that process with the lock died, it did so nicely).