I have two java processes which I want completely decoupled from each other.
I figure that the best way to do this is for one to write out its data to file and the other to read it from that file (the second might also have to write to the file to say its processed the line).
Problems I envisage are do with similtaneous access to the file. Is there a good simple pattern I can use to get around this problem? Is there a library that handles this sort of functionality?
Best way to describe it is as a simple direct message passing mechanism I could implement using files. (Simpler than JMS).
Thanks Dan
If you want a simple solution and you can assume that "rename file" is an atomic operation (this is not completely true), each one of the processes can rename the file when reading it or writing to it and rename back when it finishes. The other one will not find the file and will wait until the file appears.
you mean like a named pipe? it's possible but java doesn't allow pipe creation unless you use non portable processes
You are asking for functionality that is exactly what JMS does. JMS is an API which has many implemententations. Can you you not just use a lightweight implementation? I don't see why you think this is "complicated". By the time you've mananged to reliably implement your solution you'll have found that it's not trivial to deal with all the edge cases.
Correct me if I don't understand your problem...
Why don't you look at file locks ? When a program acquire the lock, the other wait until the lock is released
If you are not locked on a file-based solution, a database can solve your problem.
Each record will be a line written by the writing process. A single column in the record will be untouched and the reading process will use it to indicate that it red the record.
Naturally you will have to deal with cleanup of the table before it becomes to large, or its partitioning so it will be easy for the reading process to find information inside it.
If you must use a file - you can think of another file that just has the ID of the record that the reader process read - that way you don't need to have concurrently writing processes on the same file.
Related
I'm required to create a part of a java program that logs all activity "secretly" that the user is doing with the program. purely to catch people trying to "cheat" the system. The thing is, multiple people will be using the same program on multiple computers
All the information needs to be written using PrintWriter to a single text file that will, later on, be used by an administrative part of the program.
PrintWriter printer = new PrintWriter(new FileWriter(serverFolderLocation + "\\LogUserInfo.txt", true));
It's expected that 50+- computers will be using this program, every few seconds an array of 7+ lines of text is expected to be written to that file every time a specific button is pressed on a computer
I know that writing text to a text file is extremely quick and this is unlikely to happen, but If 2 or more computers happens to write to the text file at the same time, while append is set to true, will data go missing? or will it append normally?
Is this even possible? 2+ devices writing data to a text file at different times?
Do note that it is important that all the data from all 50+ computers arrive at the destinated file.
If problems are likely to occur, what other methods can be used of doing something like this, other than setting up a dedicated database?
Setting append to true in the FileWriter:
append - boolean if true, then data will be written to the end of the file rather than the beginning.
If you've got 50 machines doing this all at the same time, there's going to be conflicts. Your computer complains when you try to modify a file that's being used by another process, don't go throwing in 50 more contenders.
You could try using Sockets.
Whichever machine holds that file, designate that as the 'server' and make the other machines 'clients'. Your clients send messages to the server, your server appends those messages to the file in a synchronised manner.
You could then prevent your ~50 clients from directly changing the logs file with some network & server security.
You are simply re-inventing the wheel here, but in a very wrong way. The other answer is correct: using a simple file, and having multiple distributed users write to the same file just screams for failure.
But I disagree with the idea to use sockets instead. That is really low level, and requires you to implement a lot of (complicated) things yourself. You will have to think about network issues, multi threading, locking, buffering, ...
Sure, if this is for education, then building something like that is a challenge. But if you the goal here is to come to a robust solution that just works, you should rather think about using some 3rd party off-the-shelf solution.
You could start reading here. And then pick a framework such as logback. Alternatively, you could look into messaging services, such as ActiveMQ, or RabbitMQ.
Again: creating and collection logs in distributed environments is A) hard to get right but B) a solved problem.
Is it possible that multiple parallel rename operations of the same file to the same other name on NFS succeeds? This is what I experience.
I implemented file processors that work in parallel. They process files, by first "reserving" the file. The reservation is implemented via rename - adding some fixed prefix (e.g. RESERVED_, same for all processors). Only after successful reservation, they start processing the file. In this implementation I assumed that rename will succeed in only one processor. Unfortunately, what I experience is that rename succeeds in multiple processors...
More details:
Processors are implemented in Java. I use the call File::renameTo.
Processors work on separate servers.
time is synced with NFS via NTP.
I avoid FS-specific implementation (like using fcntl locks), because I wanted the same logic to work also for FTP/SFTP.
Does anyone know what is the expected behavior? Is there something I'm doing wrong or an easy fix (maybe some mounting option)? If this approach is just wrong, do you know of other approaches to generic file-locking on NFS?
Thanks for any help
I am writing an application which will search for for files with special filename extension on computer. (JPG for example). Input data: "D:", ".JPG" Output: txt file with results(file directories); I know one simple reccursive algo, but may be there is smth better. So, may be you tell me an efficient algorithm to traverse the file directory. Also I want to use multithreading for solving this problem to make better performance. But how many threads should I use? If I will use 1 thread for 1 directory - this will be stupid.
The recursive option you name is the only way to go, unless you want to get your hands dirty with the file system. I suspect you don't.
Regarding thread performance, your best choice is to make the number of threads configurable, create some sample directories, and measure performance for each setting.
By the way, most file-finders create an index of files. They scan the disc on a schedule, and update a file which contains the relevant information about the files and directories on disk. The file is in a format designed to facilitate searching. This index file is used to perform actual searches. If you're planning on repeatedly running this search against the same directory, you should do this.
I need to monitor a log file for a pattern. The log file continually gets written by an application.
The application can add new log statements while my program is reading it.
The log gets rolled over when it’s >200 MB or at end of the day, so my program should handle change in filename dynamically.
If my program crashes for any reason, it has to resume from where it left off.
I do not want to re-invent the wheel. I am looking for a Java API. I wrote a program to read file and put in a loop with 30 seconds sleep, but that does not meet all the criteria.
You might consider looking at apache commons io classes, in particular Tailer/TailerListener classes. See http://www.devdaily.com/java/jwarehouse/commons-io-2.0/src/main/java/org/apache/commons/io/input/Tailer.java.shtml.
These two API's can be helpful:
1
JxFileWatcher (Official Site)
Read here what it is capable of
2
Jnotify
JNotify is a java library that allow java application to listen to file system events, such as:
File created
File modified
File renamed
File deleted
If you are using Log4j, or can integrate it, it is possible to append log outputs to a convenient object, such as a StringBuffer, as it has been discussed in this related question: Custom logging to gather messages at runtime
This looks similar: Implementation of Java Tail
Essentially you use a BufferedReader. Tracking where you left off will be something you'll have to add, perhaps capture the last line read?
That same question references JLogTailer which looks interesting and may do most of what you want already.
I have a set of files. The set of files is read-only off a NTFS share, thus can have many readers. Each file is updated occasionally by one writer that has write access.
How do I ensure that:
If the write fails, that the previous file is still readable
Readers cannot hold up the single writer
I am using Java and my current solution is for the writer to write to a temporary file, then swap it out with the existing file using File.renameTo(). The problem is on NTFS, renameTo fails if target file already exists, so you have to delete it yourself. But if the writer deletes the target file and then fails (computer crash), I don't have a readable file.
nio's FileLock only work with the same JVM, so it useless to me.
How do I safely update a file with many readers using Java?
According to the JavaDoc:
This file-locking API is intended to
map directly to the native locking
facility of the underlying operating
system. Thus the locks held on a file
should be visible to all programs that
have access to the file, regardless of
the language in which those programs
are written.
I don't know if this is applicable, but if you are running in a pure Vista/Windows Server 2008 solution, I would use TxF (transactional NTFS) and then make sure you open the file handle and perform the file operations by calling the appropriate file APIs through JNI.
If that is not an option, then I think you need to have some sort of service that all clients access which is responsible to coordinate the reading/writing of the file.
On a Unix system, I'd remove the file and then open it for writing. Anybody who had it open for reading would still see the old one, and once they'd all closed it it would vanish from the file system. I don't know if NTFS has similar semantics, although I've heard that it's losely based on BSD's file system so maybe it does.
Something that should always work, no matter what OS etc, is changing your client software.
If this is an option, then you could have a file "settings1.ini" and if you want to change it, you create a file "settings2.ini.wait", then write your stuff to it and then rename it to "settings2.ini" and then delete "settings1.ini".
Your changed client software would simply always check for settings2.ini if it has read settings1.ini last, and vice versa.
This way you have always a working copy.
There might be no need for locking. I am not too familiar with the FS API on Windows, but as NTFS supports both hard links and soft links, AFAIK, you can try this if your setup allows it:
Use a hard or soft link to point to the actual file, and name the file diferently. Let everyone access the file using the link's name.
Write the new file under a different name, in the same folder.
Once it is finished, have the file point to the new file. Optimally, Windows would allow you to create the new link with replacing the existing link in one atomic operation. Then you'd effectively have the link always identify a valid file, either the old or the new one. At worst, you'd have to delete the old one first, then create the link to the new file. In that case, there'd be a short time span in which a program would not be able to locate the file. (Also, Mac OS X offers a "ExchangeObjects" function that allows you to swap two items atomically - maybe Windows offers something similar).
This way, any program that has the old file already opened will continue to access the old one, and you won't get into its way creating the new one. Only if an app then notices the existence of the new version, it could then close the current and open it again, this way getting access to the new version.
I don't know, however, how to create links in Java. Maybe you have to use some native API for that.
I hope this helps anyways.
I have been dealing with something similar recently. If you are running Java 5, perhaps you could consider using NIO file locks in conjunction with a ReentrantReadWriteLock? Make sure all code referencing the FileChannel object ALSO references the ReentrantReadWriteLock. This way the NIO locks it at a per-VM level while the reentrant lock locks it at a per-thread level.
FileLock fileLock = filechannel.lock(position, size, shared);
reentrantReadWriteLock.lock();
// do stuff
fileLock.release();
reentrantReadWriteLock.unlock();
Of course, some exception handling would be required.