This question already has answers here:
JAVA NIO Watcher: How to detect end of a long lasting (copy) operation?
(2 answers)
Closed 8 years ago.
I am writing a directory monitoring utility in java(1.6) using polling at certain intervals using lastModified long value as the indication of change. I found that when my polling interval is small (seconds) and the copied file is big then the change event is fired before the actual completion of file copying.
I would like to know whether there is a way I can find the status of file like in transit, complete etc.
Environments: Java 1.6; expected to work on windows and linux.
There are two approaches I've used in the past which are platform agnostic.
1/ This was for FTP transfers where I controlled what was put, so it may not be directly relevant.
Basically, whatever is putting a file file.txt will, when it's finished, also put a small (probably zero-byte) dummy file called file.txt.marker (for example).
That way, the monitoring tool just looks for the marker file to appear and, when it does, it knows the real file is complete. It can then process the real file and delete the marker.
2/ An unchanged duration.
Have your monitor program wait until the file is unchanged for N seconds (where N is reasonably guaranteed to be large enough that the file will be finished).
For example, if the file size hasn't changed in 60 seconds, there's a good chance it's finished.
There's a balancing act between not thinking the file is finished just because there's no activity on it, and the wait once it is finished before you can start processing it. This is less of a problem for local copying than FTP.
This solution worked for me:
File ff = new File(fileStr);
if(ff.exists()) {
for(int timeout = 100; timeout>0; timeout--) {
RandomAccessFile ran = null;
try {
ran = new RandomAccessFile(ff, "rw");
break; // no errors, done waiting
} catch (Exception ex) {
System.out.println("timeout: " + timeout + ": " + ex.getMessage());
} finally {
if(ran != null) try {
ran.close();
} catch (IOException ex) {
//do nothing
}
ran = null;
}
try {
Thread.sleep(100); // wait a bit then try again
} catch (InterruptedException ex) {
//do nothing
}
}
System.out.println("File lockable: " + fileStr +
(ff.exists()?" exists":" deleted during process"));
} else {
System.out.println("File does not exist: " + fileStr);
}
This solution relies on the fact that you can't open the file for writing if another process has it open. It will stay in the loop until the timeout value is reached or the file can be opened. The timeout values will need to be adjusted depending on the application's actual needs. I also tried this method with channels and tryLock(), but it didn't seem to be necessary.
Do you mean that you're waiting for the lastModified time to settle? At best that will be a bit hit-and-miss.
How about trying to open the file with write access (appending rather than truncating the file, of course)? That won't succeed if another process is still trying to write to it. It's a bit ugly, particularly as it's likely to be a case of using exceptions for flow control (ick) but I think it'll work.
If I understood the question correctly, you're looking for a way to distinguish whether the copying of a file is complete or still in progress?
How about comparing the size of the source and destination file (i.e. file.length())? If they're equal, then copying is complete. Otherwise, it's still in progress.
I'm not sure it's efficient since it would still require polling. But it "might" work.
You could look into online file upload with progressbar techniques - they use OutputStreamListener and custom writer to notify the listener about bytes written.
http://www.missiondata.com/blog/java/28/file-upload-progress-with-ajax-and-java-and-prototype/
File Upload with Java (with progress bar)
We used to monitor the File Size change for determine whether the File is inComplete or not.
we used Spring integration File endpoint to do the polling for a directory for every 200 ms.
Once the file is detected(regardless of whether it is complete or not), We have a customer File filter, which will have a interface method "accept(File file)" to return a flag indicating whether we can process the file.
If the False is returned by the filter, this FILE instance will be ignored and it will be pick up during the next polling for the same filtering process..
The filter does the following:
First, we get its current file size. and we will wait for 200ms(can be less) and check for the size again. If the size differs, we will retry for 5 times. Only when the file size stops growing, the File will be marked as COMPLETED.(i.e. return true).
Sample code used is as the following:
public class InCompleteFileFilter<F> extends AbstractFileListFilter<F> {
protected Object monitor = new Object();
#Override
protected boolean accept(F file) {
synchronized (monitor){
File currentFile = (File)file;
if(!currentFile.getName().contains("Conv1")){return false;}
long currentSize = currentFile.length();
try { Thread.sleep(200); } catch (InterruptedException e) { e.printStackTrace(); }
int retryCount = 0;
while(retryCount++ < 4 && currentFile.length() > currentSize){
try { Thread.sleep(200); } catch (InterruptedException e) { e.printStackTrace(); }
}
if(retryCount == 5){
return false;
}else{
return true;
}
}
}
}
Related
According to this link, if the source has a problem when opening and throws an exception and it is also in try parentheses, JVM closes it. My question is how to inform the user now that this source is closed and we encountered a problem when opening this resource? In other words, how can this exception be handled?
Seems trivial. Usually, java code is running in some sort of 'no user interaction' environment (servers and the like). The right move is to let the exception bubble up - you want the daily job that is halfway through reading through the database to open the related file to then send the logs to long term storage or whatever it is to completely abort and write a note in the log file. Usually for jobs like that, there's some atomary functionality (in this case, perhaps each such file is independent of the others, and its okay to leave the 'broken' one in place for now until a server admin can look at it whilst continuing to process the remainder - in that case, the 'do the backup rotation thing on THIS file' is the atomary functionality): Catch all exceptions and write code that does what you want when the job fails. For example, my servers can send notifications straight to admin phones (via telegram or pushover, or using slack API, and there are many services that automate this for you too), if it's important, you'd write that in your catch block.
For code that is directly 'triggered' by a user, let's say a 'save file' function, then it's not so much 'the resource is now closed' - resources are not long lived (they cannot be - not if you use try-with-resources). They were either never open in the first place (you attempt to save a file to a dir that doesn't exist - the act of trying to make the new OutputStream already failed, it was never open to begin with), or, perhaps it did open, but it was to a USB stick and the user pulled it out halfway through saving. The resource is just closed, effectively, whether in java you .close() it or not - the entire stick is gone!!
The only thing the 'safe close' aspect of try-with-resources did for you is ensure that your Java Process isn't wasting a file handle.
You handle it the same way you handle pretty much any 'unrecoverable' (you can't write software that hypnotises the user into sticking that USB stick back into the machine, obviously - it is not recoverable as a consequence, like most exceptions) problem: You toss up a dialog box that explains the situation.
try (OutputStream out = Files.newOutputStream(saveGameFile)) {
boardState.save(out);
} catch (IOException e) {
// show dialog here
}
Even when using a try-with-resources, the catch clause still works.
private static void printFile() throws MyCustomException {
try(FileInputStream input = new FileInputStream("file.txt")) {
int data = input.read();
while(data != -1){
System.out.print((char) data);
data = input.read();
}
} catch (IOException e) {
throw new MyCustomException("There was an error while opening the resource", e);
}
}
i have robotframework plugin which may run several processes simultaneously, but the problem is that it create screenshots and one process may rewrite already taken screenshot from another process
i did such thing
File path;
while (true) {
path = new File(logdir, normalizeFilename(filename));
if (path.exists())
continue;
try {
path.createNewFile();
break;
} catch (IOException e) {
e.printStackTrace();
}
}
but i'm not sure it's enough, as long as i have only 2 processes there is no problem, but there may be more, and path.createNewFile(); doesn't give any error when there is already fail available, there is not much chance that after path.exists() return false there will be created same file because of timestamp in the name, but still i think there might be a problem and i don't know hot to synchronize processes.
I'm trying to get this piece of code to work. It's a basic I/O system that copies one file and pastes it into the same directory with the chosen name. It should be simple but for some reason the program runs, it creates the second file but then it gets stuck. The CPU for Java process sits at around 5% and the file is never completed. It only copies over some of the data and then I'd imagine it's stuck in an infinite loop somewhere.
I've already compared my code with the Byte Streams tutorial on the Oracle website.
EXTRA: I just asked it to output what it was reading and it's stuck on an infinite loop reading the value 255. If that helps. Also, I compiled the code directly off the Oracle website and it does the same thing.
It appears consistent from what I can tell. Can anyone tell me what I'm doing wrong? Thank you.
(P.S: I'm Using Eclipse 4.2.0).
This is what I'm doing to copy the file:
package fileIO;
import java.io.*;
import system.Debug;
public class fileUtil {
public static void copyFileTo(String file2Copy, String file2Paste) {
FileInputStream fin = null;
FileOutputStream fout = null;
try {
fin = new FileInputStream(file2Copy);
fout = new FileOutputStream(file2Paste);
int aByte;
while ((aByte = fin.read()) != -1) {
fout.write(aByte);
}
} catch (FileNotFoundException e) {
Debug.out("Error: File Not Found: " + file2Copy);
} catch (IOException e) {
Debug.out("Error: File IO Exception Copying: " + file2Copy);
} catch (Exception e) {
Debug.out("Error: General Exception Closing Streams:" + file2Copy);
} finally {
try {
fin.close();
fout.close();
} catch (IOException e) {
Debug.out("Error: File IO Exception Closing Streams: " + file2Copy);
} catch (Exception e) {
Debug.out("Error: General Exception Closing Streams:" + file2Copy);
}
}
}
}
In my program main class I run this:
fileUtil.copyFileTo("google.bmp", "google(1).bmp");
Try to do fout.flush() before you close the OutputStream.
Okay, so I found out what was happening. Was a really nooby mistake.
I'll put my pride aside encase anyone else has this problem. It's not an infinite loop, it's just that copying using ByteStreams takes AGES. I was expecting a fast result from small image files but even small image files take a long long time to copy. I let it run for 30 seconds and the program terminated properly and I got my image copy just fine.
Thank god it's solved, I was beginning to worry.
... or do not invent bother re-inventing the wheel: use FileUtils.copyFile from the proven Apache commons-io which does it in one line.
(beware: this comment is not as innocent as it seems: File.rename does not work well on Windows shares - commons-io is always the safe bet to do these things)
Edit
Stackoverflow is not a goog place for "homework" - or you must at least say so. It is not that your problem is not real. It is that your objectives differ: you want to learn something, we want to make it work reliably with minimum maintenance.
...which leads to my second point: when you are in professional life, never program this again. As you discovered, even if you make it work, it may be extremely inefficient, handle errors incorrectly, etc.. This is particularly true with IO which is always more tricky than it seems.
Finally, since I gave you a link to a well trusted library under Apache 2.0 license, maybe you could have had a look at the source code ?
This is the first time I've encountered something like below.
Multiple Threads (Inner classes implementing Runnable) sharing a Data Structure (instance variable of the upper class).
Working: took classes from Eclipse project's bin folder, ran on a Unix machine.
NOT WORKING: directly compiled the src on Unix machine and used those class files. Code compiles and then runs with no errors/warnings, but one thread is not able to access shared resource properly.
PROBLEM: One thread adds elements to the above common DS. Second thread does the following...
while(true){
if(myArrayList.size() > 0){
//do stuff
}
}
The Log shows that the size is updated in Thread 1.
For some mystic reason, the workflow is not enetering if() ...
Same exact code runs perfectly if I directly paste the class files from Eclipse's bin folder.
I apologize if I missed anything obvious.
Code:
ArrayList<CSRequest> newCSRequests = new ArrayList<CSRequest>();
//Thread 1
private class ListeningSocketThread implements Runnable {
ServerSocket listeningSocket;
public void run() {
try {
LogUtil.log("Initiating...");
init(); // creates socket
processIncomongMessages();
listeningSocket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
private void processIncomongMessages() throws IOException {
while (true) {
try {
processMessage(listeningSocket.accept());
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
}
private void processMessage(Socket s) throws IOException, ClassNotFoundException {
// read message
ObjectInputStream ois = new ObjectInputStream(s.getInputStream());
Object message = ois.readObject();
LogUtil.log("adding...: before size: " + newCSRequests.size());
synchronized (newCSRequests) {
newCSRequests.add((CSRequest) message);
}
LogUtil.log("adding...: after size: " + newCSRequests.size()); // YES, THE SIZE IS UPDATED TO > 0
//closing....
}
........
}
//Thread 2
private class CSRequestResponder implements Runnable {
public void run() {
LogUtil.log("Initiating..."); // REACHES..
while (true) {
// LogUtil.log("inside while..."); // IF NOT COMMENTED, FLOODS THE CONSOLE WITH THIS MSG...
if (newCSRequests.size() > 0) { // DOES NOT PASS
LogUtil.log("inside if size > 0..."); // NEVER REACHES....
try {
handleNewCSRequests();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
....
}
UPDATE
Solution was to add synchronized(myArrayList) before I check the size in the Thread 2.
To access a shared structure in a multi-threaded environment, you should use implicit or explicit locking to ensure safe publication and access among threads.
Using the code above, it should look like this:
while(true){
synchronized (myArrayList) {
if(myArrayList.size() > 0){
//do stuff
}
}
//sleep(...) // outside the lock!
}
Note: This pattern looks much like a producer-consumer and is better implemented using a queue. LinkedBlockingQueue is a good option for that and provides built-in concurrency control capabilities. It's a good structure for safe publishing of data among threads.
Using a concurrent data structure lets you get rid of the synchronized block:
Queue queue = new LinkedBlockingQueue(...)
...
while(true){
Data data = queue.take(); // this will wait until there's data in the queue
doStuff(data);
}
Every time you modify a given shared variable inside a parallel region (a region with multiple threads running in parallel) you must ensure mutual exclusion. You can guarantee mutual exclusion in Java by using synchronized or locks, normally you use locks when you want a finer grain synchronization.
If the program only performance reads on a given shared variable there is no need for synchronized/lock the accesses to this variable.
Since you are new in this subject I recommend you this tutorial
If I got this right.. There are at least 2 threads that work with the same, shared, datastructure. The array you mentioned.. One thread adds values to the array and the second thread "does stuff" if the size of the array > 0.
There is a chance that the thread scheduler ran the second thread (that checks if the collection is > 0), before the first thread got a chance to run and add a value.
Running the classes from bin or recompiling them has nothing to do. If you were to run the application over again from the bin directory, you might seen the issue again. How many times did you ran the app?
It might not reproduce consistently but at one point you might see the issue again.
You could access the datastruce in a serial fashion, allowing only one thread at a time to access the array. Still that does not guarantee that the first thread will run and only then the second one will check if the size > 0.
Depending on what you need to accomplish, there might be better / other ways to achieve that. Not necessarily using a array to coordinate the threads..
Check the return of
newCSRequests.add((CSRequest) message);
I am guessing its possible that it didn't get added for some reason. If it was a HashSet or similar, it could have been because the hashcode for multiple objects return the same value. What is the equals implementation of the message object?
You could also use
List list = Collections.synchronizedList(new ArrayList(...));
to ensure the arraylist is always synchronised correctly.
HTH
This question already has answers here:
JAVA NIO Watcher: How to detect end of a long lasting (copy) operation?
(2 answers)
Closed 8 years ago.
I am writing a directory monitoring utility in java(1.6) using polling at certain intervals using lastModified long value as the indication of change. I found that when my polling interval is small (seconds) and the copied file is big then the change event is fired before the actual completion of file copying.
I would like to know whether there is a way I can find the status of file like in transit, complete etc.
Environments: Java 1.6; expected to work on windows and linux.
There are two approaches I've used in the past which are platform agnostic.
1/ This was for FTP transfers where I controlled what was put, so it may not be directly relevant.
Basically, whatever is putting a file file.txt will, when it's finished, also put a small (probably zero-byte) dummy file called file.txt.marker (for example).
That way, the monitoring tool just looks for the marker file to appear and, when it does, it knows the real file is complete. It can then process the real file and delete the marker.
2/ An unchanged duration.
Have your monitor program wait until the file is unchanged for N seconds (where N is reasonably guaranteed to be large enough that the file will be finished).
For example, if the file size hasn't changed in 60 seconds, there's a good chance it's finished.
There's a balancing act between not thinking the file is finished just because there's no activity on it, and the wait once it is finished before you can start processing it. This is less of a problem for local copying than FTP.
This solution worked for me:
File ff = new File(fileStr);
if(ff.exists()) {
for(int timeout = 100; timeout>0; timeout--) {
RandomAccessFile ran = null;
try {
ran = new RandomAccessFile(ff, "rw");
break; // no errors, done waiting
} catch (Exception ex) {
System.out.println("timeout: " + timeout + ": " + ex.getMessage());
} finally {
if(ran != null) try {
ran.close();
} catch (IOException ex) {
//do nothing
}
ran = null;
}
try {
Thread.sleep(100); // wait a bit then try again
} catch (InterruptedException ex) {
//do nothing
}
}
System.out.println("File lockable: " + fileStr +
(ff.exists()?" exists":" deleted during process"));
} else {
System.out.println("File does not exist: " + fileStr);
}
This solution relies on the fact that you can't open the file for writing if another process has it open. It will stay in the loop until the timeout value is reached or the file can be opened. The timeout values will need to be adjusted depending on the application's actual needs. I also tried this method with channels and tryLock(), but it didn't seem to be necessary.
Do you mean that you're waiting for the lastModified time to settle? At best that will be a bit hit-and-miss.
How about trying to open the file with write access (appending rather than truncating the file, of course)? That won't succeed if another process is still trying to write to it. It's a bit ugly, particularly as it's likely to be a case of using exceptions for flow control (ick) but I think it'll work.
If I understood the question correctly, you're looking for a way to distinguish whether the copying of a file is complete or still in progress?
How about comparing the size of the source and destination file (i.e. file.length())? If they're equal, then copying is complete. Otherwise, it's still in progress.
I'm not sure it's efficient since it would still require polling. But it "might" work.
You could look into online file upload with progressbar techniques - they use OutputStreamListener and custom writer to notify the listener about bytes written.
http://www.missiondata.com/blog/java/28/file-upload-progress-with-ajax-and-java-and-prototype/
File Upload with Java (with progress bar)
We used to monitor the File Size change for determine whether the File is inComplete or not.
we used Spring integration File endpoint to do the polling for a directory for every 200 ms.
Once the file is detected(regardless of whether it is complete or not), We have a customer File filter, which will have a interface method "accept(File file)" to return a flag indicating whether we can process the file.
If the False is returned by the filter, this FILE instance will be ignored and it will be pick up during the next polling for the same filtering process..
The filter does the following:
First, we get its current file size. and we will wait for 200ms(can be less) and check for the size again. If the size differs, we will retry for 5 times. Only when the file size stops growing, the File will be marked as COMPLETED.(i.e. return true).
Sample code used is as the following:
public class InCompleteFileFilter<F> extends AbstractFileListFilter<F> {
protected Object monitor = new Object();
#Override
protected boolean accept(F file) {
synchronized (monitor){
File currentFile = (File)file;
if(!currentFile.getName().contains("Conv1")){return false;}
long currentSize = currentFile.length();
try { Thread.sleep(200); } catch (InterruptedException e) { e.printStackTrace(); }
int retryCount = 0;
while(retryCount++ < 4 && currentFile.length() > currentSize){
try { Thread.sleep(200); } catch (InterruptedException e) { e.printStackTrace(); }
}
if(retryCount == 5){
return false;
}else{
return true;
}
}
}
}