Develop Reference

Java NIO WatchService

I have Java project that we need to keep listening some path to detect if there is any new XML file and if true, we need to process it through few other rules.
The WatchService is doing very well to discover if I have a new file there to be processed, but I wasnt able to manage it so I could read the file, I just gives me back the file name, from the event.
Is there any way to that with WatchService? If not, whats another sugestion to reach?
Thanks.

Sure, you can easely to that:
Create the watch service:
WatchService watchService = null;
try {
watchService = FileSystems.getDefault().newWatchService();
Path path = get(pathToSee);
path.register(watchService, new WatchEvent.Kind[]{ENTRY_MODIFY, ENTRY_CREATE}, SensitivityWatchEventModifier.HIGH);
watchEvent(watchService, path);
log.info("Watch Service has ben created!");
} catch (IOException e) {
log.error("Exception has ben throw when the service have tried to createWatchService()", e);
}
Note: If you have a large amount of files to be added, you can put:
SensitivityWatchEventModifier.HIGH
To increase sensitivity.
Watch if have changes in you directory:
WatchKey key;
while (true) {
try {
if ((key = watchService.take()) == null) break;
for (WatchEvent<?> event : key.pollEvents()) {
log.info("Event kind:" + event.kind()
+ ". File affected: " + event.context() + ".");
String fileName = event.context().toString();
File directory = path.toFile();
yourService.readContent(directory, fileName);
}
key.reset();
} catch (InterruptedException | IOException e) {
log.error("InterruptedException when try watchEvent()" + e);
}
}
And finally, you can do what you want to do with that infos:
try (BufferedReader br = new BufferedReader(new FileReader(directory + "/" + fileName))) {
String strLine;
while ((strLine = br.readLine()) != null) {
}
}
}
}
Tips:
You can create a separete thread to do that, or use Spring #Async to create a separete thread to process this information and increase concurrency in your application.
You can use Apache Commons you to do that too!
e.g:
public void getAll() throws Exception {
FileAlterationObserver observer = new FileAlterationObserver(pathToSee);
observer.addListener(new FileAlterationListenerAdaptor() {
#SneakyThrows
#Override
public void onFileCreate(File file) {
}
#Override
public void onFileDelete(File file) {
}
});
}

how to watch multiple hard disk partitions in FileWacther in java

I am using FileWatcher to watch a whole hard disk partition in my case its D drive eg: D:/ the code is working fine with one path i provided, all i want is to watch other hard disk partitions as well like C:,D: and E: how can i achieve that here is the code
public class FileWatcher {
private final WatchService watcher;
private final Map<WatchKey, Path> keys;
static Logger log = LoggerFactory.getLogger(GitCloneRepo.class);
/**
* Creates a WatchService and registers the given directory
*/
FileWatcher(Path dir) throws IOException {
this.watcher = FileSystems.getDefault().newWatchService();
this.keys = new HashMap<WatchKey, Path>();
walkAndRegisterDirectories(dir);
}
/**
* Register the given directory with the WatchService; This function will be called by FileVisitor
*/
private void registerDirectory(Path dir) throws IOException
{
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
keys.put(key, dir);
}
/**
* Register the given directory, and all its sub-directories, with the WatchService.
*/
private void walkAndRegisterDirectories(final Path start) throws IOException {
// register directory and sub-directories
Files.walkFileTree(start, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
registerDirectory(dir);
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException {
if (exc instanceof AccessDeniedException) {
return FileVisitResult.SKIP_SUBTREE;
}
return super.visitFileFailed(file, exc);
}
});
}
/**
* Process all events for keys queued to the watcher
*/
void processEvents() {
for (;;) {
// wait for key to be signalled
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException x) {
log.error("InterruptedException ",x);
return;
}
Path dir = keys.get(key);
if (dir == null) {
log.warn("WatchKey not recognized!!");
continue;
}
for (WatchEvent<?> event : key.pollEvents()) {
#SuppressWarnings("rawtypes")
WatchEvent.Kind kind = event.kind();
// Context for directory entry event is the file name of entry
#SuppressWarnings("unchecked")
Path name = ((WatchEvent<Path>)event).context();
Path child = dir.resolve(name);
log.info("watching files");
// print out event
if (kind == ENTRY_MODIFY) {
log.info("event.kind().name() {}: child {}", event.kind().name(), child);
log.info("child {} ends with docx? {} ",child,child.endsWith(".docx"));
String c= child.toString();
log.info("**child {}***c.endsWith(.docx)"
+ ""
+ " {}",c,c.endsWith(".docx"));
}
// if directory is created, and watching recursively, then register it and its sub-directories
if (kind == ENTRY_CREATE) {
try {
if (Files.isDirectory(child)) {
walkAndRegisterDirectories(child);
}
} catch (IOException x) {
// do something useful
}
}
}
// reset key and remove from set if directory no longer accessible
boolean valid = key.reset();
if (!valid) {
keys.remove(key);
// all directories are inaccessible
if (keys.isEmpty()) {
break;
}
}
}
}
//working code
public static void main(String[] args) throws IOException {
try{
Path dir = Paths.get("D:");
FileWatcher fileWatcher=new FileWatcher(dir);
fileWatcher.processEvents();
}
catch (AccessDeniedException xx) {
log.error("AccessDeniedException ",xx);
}
catch (FileSystemException x) {
log.error("exception",x);
}
}
}
i looked at this question but i did not seemed to solve my problem
How to implement file watcher to Watch multiple directories

Your processEvents method enters an infinite loop and doesn't return until either the thread is interrupted or the path is gone, which means any code after processEvents won't execute until it's done. If you want your main method to spin up multiple watchers, you'll need to call processEvents from other threads, e.g. with Java 8:
// Create watchers
List<FileWatcher> watchers = new ArrayList<>();
try{
watchers.add(new FileWatcher(Paths.get("D:")));
watchers.add(new FileWatcher(Paths.get("E:")));
} catch (AccessDeniedException xx) {
log.error("AccessDeniedException ",xx);
} catch (FileSystemException x) {
log.error("exception",x);
}
// Create and start threads
List<Thread> threads = watchers.stream()
.map(w -> new Thread(w::processEvents))
.peek(Thread::start)
.collect(Collectors.toList());
With Java 7:
// Create watchers
List<FileWatcher> watchers = new ArrayList<>();
try{
watchers.add(new FileWatcher(Paths.get("D:")));
watchers.add(new FileWatcher(Paths.get("E:")));
} catch (AccessDeniedException xx) {
log.error("AccessDeniedException ",xx);
} catch (FileSystemException x) {
log.error("exception",x);
}
// Create and start threads
List<Thread> threads = new ArrayList<>();
for (FileWatcher watcher : watchers) {
Thread thread = new Thread(new Runnable() {
#Override
public void run() {
watcher.processEvents();
}
});
thread.start();
threads.add(thread);
}

stop watcher service on application quit

I use gradle and application plugin to run the application that watches the changes in the directory.
My main class looks like this
public static void main(String[] args) throws IOException {
WatcherThread thread = new WatcherThread(EXTENSION_FOLDER);
thread.start();
try(BufferedReader br = new BufferedReader(new InputStreamReader(System.in))) {
String input = null;
ConsoleInputController controller = new ConsoleInputController(br);
while (!QUIT_COMMAND.equals(StringUtils.trim(input))) {
System.out.println(CONSOLE_TEMPLATE);
System.out.println("input (to exit write [quit]):> ");
input = br.readLine();
controller.handleInput(input);
}
} catch (IOException exc) {
LOGGER.error("Failed to process input.", exc);
}
thread.stopThread();
}
WatcherThread is a thread class that uses WatcherService (some wrapper over WatchService of java)
public class WatcherThread extends Thread {
private static final Logger LOGGER = LoggerFactory.getLogger(WatcherThread.class);
private boolean watch = true;
private WatcherService watcherService;
public WatcherThread(String searchingPath) throws IOException {
watcherService = new WatcherService(Paths.get(searchingPath));
}
#Override
public void run() {
LOGGER.info("Artifact watching thread started.");
while(watch) {
if (!watcherService.watch()) {
break;
}
}
LOGGER.info("Artifact watching thread stopped.");
}
public void stopThread() {
watch = false;
}
}
WatcherService looks like this
public class WatcherService {
private static final Logger LOGGER = LoggerFactory.getLogger(WatcherThread.class);
private final WatchService watcher;
private final Map<WatchKey, Path> keys;
private boolean trace;
WatcherService(Path dir) throws IOException {
watcher = FileSystems.getDefault().newWatchService();
keys = new HashMap<>();
register(dir);
trace = true;
}
private void register(Path dir) throws IOException {
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
if (trace) {
Path prev = keys.get(key);
if (null == prev) {
LOGGER.info("Register path: [{}].", dir);
} else {
if (!dir.equals(prev)) {
LOGGER.info("Updated path: [{}] -> [{}].", prev, dir);
}
}
}
keys.put(key, dir);
}
boolean watch() {
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException exc) {
return false;
}
Path dir = keys.get(key);
if (null == dir) {
LOGGER.warn("WatchKey is not recognized!");
return false;
}
// forEach?
for (WatchEvent event: key.pollEvents()) {
LOGGER.info("Polling events");
WatchEvent.Kind kind = event.kind();
if (OVERFLOW == kind) {
continue;
}
WatchEvent<Path> ev = (WatchEvent<Path>) event;
Path name = ev.context();
Path child = dir.resolve(name);
LOGGER.info("Event occurred [{}] in [{}].", event.kind().name(), child);
WatchEventResolver.resolveEvent(ev, child);
}
boolean valid = key.reset();
if (!valid) {
keys.remove(key);
if (keys.isEmpty()) {
return false;
}
}
return true;
}
}
When I do not start my WatcherThread - console input works fine. I can quit for example without problems. But when I run thread and want to quit, it is waiting for couple of seconds and then only ends.
As far as I understand it is something with WatchService that cannot stop watching the directory.
How to stop it on quit application immediately?

It looks like you need an extra method in your WatcherService class, that calls to watcher.close(). Then, in your WatcherThread class you can call to that method inside stopThread().
In the Javadoc for WatchService you can see that take() keeps waiting. You can force it to finish by closing it.

Divide all the files present in the directory/folder to different thread for faster execution

I got a program in which I have to write two thread that will perform same operation on the files present in a particular folder. That operation is to read all the files in that folder and then delete all of them present there. These two thread will have same operation but just to increase the process time I have to divide the files between two thread so that execution time is saved. I tried this way but it is not dividing the no. of files between threads but takes all the files and do the operation and if the file is deleted by one thread then other thread should not pick that file up. The code I wrote is:
public static void main(String[] args) throws InterruptedException {
ExecutorService service = Executors.newFixedThreadPool(2);
service.submit(new thread1());
service.submit(new thread2());
service.shutdown();
service.awaitTermination(1, TimeUnit.DAYS);
System.exit(0);
}
public static class thread1 implements Callable<Object> {
#Override
public Object call() throws Exception {
t1();
return null;
}
}
public static class thread2 implements Callable<Object> {
#Override
public Object call() throws Exception {
t1();
return null;
}
}
public static void t1() {
Path myDir = Paths.get("D:/Astha/");
File file = new File("D:/Astha/");
boolean running = true;
while (running) {
try {
WatchService watcher = myDir.getFileSystem().newWatchService();
myDir.register(watcher, StandardWatchEventKinds.ENTRY_CREATE);
WatchKey watckKey = watcher.take();
List<WatchEvent<?>> events = watckKey.pollEvents();
for (#SuppressWarnings("rawtypes")
WatchEvent event : events) {
if (event.kind() == StandardWatchEventKinds.ENTRY_CREATE) {
System.out.println("Created: " + event.context().toString() + "By "
+ Thread.currentThread().getName());
}
}
if (file.exists()) {
File[] files = file.listFiles();
for (File f : files) {
if (f.delete()) {
System.out.println("Deleting the file: " + f.getName() + "By "
+ Thread.currentThread().getName());
}
}
} else {
System.out.println("No files in the folder");
}
} catch (Exception e) {
System.out.println("Error: " + e.toString());
}
}
}
In this above program, I also need to apply lock on one thread so that no other thread can perform operation on that. How do I implement lock here?

Based on your (updated) code, you have two basic task. You have a "watch" task and you have a "process" task
WatcherService
The WatcherService basically takes a Path and a ExecutorService, it monitors the given path and creates new FileTask tasks
public class WatcherService implements Callable<Object> {
private Path path;
private ExecutorService service;
public WatcherService(Path path, ExecutorService service) {
this.path = path;
this.service = service;
}
#Override
public Object call() throws Exception {
do {
try {
WatchService watcher = path.getFileSystem().newWatchService();
path.register(watcher, StandardWatchEventKinds.ENTRY_CREATE);
WatchKey watckKey = watcher.take();
List<WatchEvent<?>> events = watckKey.pollEvents();
for (#SuppressWarnings("rawtypes") WatchEvent event : events) {
WatchEvent<Path> we = (WatchEvent<Path>)event;
service.submit(new FileTask(we.context()));
}
} catch (IOException | InterruptedException exp) {
exp.printStackTrace();
}
} while (true && !Thread.currentThread().isInterrupted());
return null;
}
}
FileTask
The FileTask takes a Path and performs some operation upon it
public class FileTask implements Callable<Object> {
private Path path;
public FileTask(Path file) {
this.path = file;
}
#Override
public Object call() throws Exception {
File file = path.toFile();
if (file.exists()) {
if (file.delete()) {
//...
}
}
return null;
}
}
Hooking it up
Basically, you create a ExecutorService and submit a WatcherService and let it run...
Path path = Paths.get("D:/Astha/");
ExecutorService service = Executors.newFixedThreadPool(3);
service.submit(new WatcherService(path, service));
This creates a pooled service of three threads, one for the watcher and two for the FileTasks
You may find that this still does not offer you any benefit, as the disk I/O won't allow multiple operations to carried out in parallel and will block until the first operation completes before the next can be carried out

You could put all Files to edit in a HashSet or Map.
private static volatile Set<String> filenames;
If one of your threads is able to get the next File, use a synchronized Method to deliver it.
public synchronized String getNextMessage() {
if(filenames.size()<1) {
return null;
} else {
final String result = filenames.get(0);
filenames.remove(0);
return result;
}
Instead of a String, you could also use File, URI or Path, depending on your needs.

how to restart a thread

I tried to write a file monitor which will check the file if a new line is appended,the monitor in fact is a thread which will read the line by a randomaccessfile all the time.
This is the monitor core codes:
public class Monitor {
public static Logger log = Logger.getLogger(Monitor.class);
public static final Monitor instance = new Monitor();
private static final ArrayList<Listener> registers = new ArrayList<Listener>();
private Runnable task = new MonitorTask();
private Thread monitorThread = new Thread(task);
private boolean beStart = true;
private static RandomAccessFile raf = null;
private File monitoredFile = null;
private long lastPos;
public void register(File f, Listener listener) {
this.monitoredFile = f;
registers.add(listener);
monitorThread.start();
}
public void replaceFile(File newFileToBeMonitored) {
this.monitoredFile = newFileToBeMonitored;
// here,how to restart the monitorThread?
}
private void setRandomFile() {
if (!monitoredFile.exists()) {
log.warn("File [" + monitoredFile.getAbsolutePath()
+ "] not exist,will try again after 30 seconds");
try {
Thread.sleep(30 * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
setRandomFile();
return;
}
try {
if (raf != null) {
raf.close();
lastPos = 0;
}
raf = new RandomAccessFile(monitoredFile, "r");
log.info("monitor file " + monitoredFile.getAbsolutePath());
} catch (FileNotFoundException e) {
// The file must exist now
} catch (IOException e) {}
}
private void startRead() {
beStart = true;
String line;
while (beStart) {
try {
raf.seek(lastPos);
while ((line = raf.readLine()) != null) {
fireEvent(new FileEvent(monitoredFile.getAbsolutePath(),
line));
}
lastPos = raf.getFilePointer();
} catch (IOException e1) {}
}
}
private void stopRead() {
this.beStart = false;
}
private void fireEvent(FileEvent event) {
for (Listener lis : registers) {
lis.lineAppended(event);
}
}
private class MonitorTask implements Runnable {
#Override
public void run() {
stopRead();
//why putting the resetReandomAccessFile in this thread method is that it will sleep if the file not exist.
setRandomFile();
startRead();
}
}
}
This is some help classes:
public interface Listener {
void lineAppended(FileEvent event);
}
public class FileEvent {
private String line;
private String source;
public FileEvent(String filepath, String addedLine) {
this.line = addedLine;
this.source = filepath;
}
//getter and setter
}
And this is a example to call the monitor:
public class Client implements Listener {
private static File f = new File("D:/ab.txt");
public static void main(String[] args) {
Monitor.instance.register(f, new Client());
System.out.println(" I am done in the main method");
try {
Thread.sleep(5000);
Monitor.instance.replaceFile(new File("D:/new.txt"));
} catch (InterruptedException e) {
System.out.println(e.getMessage());
}
}
#Override
public void lineAppended(FileEvent event) {
String line = event.getLine();
if (line.length() <= 0)
return;
System.err.println("found in listener:" + line + ":" + line.length());
}
}
Now,my probelm is the code work well if I just call:
Monitor.instance.register(file,listener);
This will monitor the file for line appending,and will notify the listener.
However it does not work when I call the :
Monitor.instance.replaceFile(anotherfile);
This means I want to monitor another file rather than before.
So in my Monitor I have to restart the thread,how to make it?
I have tried the:
monitorThread.interruppt();
It does not wrok.
Anyone can fix it for me or tell me how to do ?
Thanks.
Before I ask,I have googling the "restart java thread",so I know one can not restart a dead thread,but my thread does not return,so I think it can be restarted.

You don't restart a Thread, instead you create a new one each time you want to start a thread.
A better alternative may be to use Executors.newCachedThreadPool() which gives you a pool of thread which will be started/recycle for you.
BTW: You are using recursion rather than a loop to poll if the file exists. Using recursion can mean if you wait too long it will throw a StackOverflowError. IMHO you shouldn't wait at all, the polling thread should repeatedly attempt to open the file until it is told to stop (or the file appears)
Your current implementation also means if the file is replaced, you will have to reopen the file in the background thread anyway.

Instead of explaining, I just coded up a skeleton example. I did not test it terribly well, but it may be of some use.
In order to monitor a(nother) file, just create a new Monitor, passing it a ScheduledExecutorService. Starting and stopping monitoring is straightforward. You can (should) reuse the same executor for multiple monitors.
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public interface Event
{
}
public interface Listener
{
void handle(Event event);
}
public class Monitor
{
private static final int CHECK_EVERY_SECONDS = 10;
private static final int RECHECK_AFTER_IF_NOT_EXISTS_SECONDS = 30;
private File file;
private ScheduledExecutorService executor;
private boolean active;
private List<Listener> listeners;
public Monitor(File file, ScheduledExecutorService executor)
{
super();
this.file = file;
this.executor = executor;
listeners = new ArrayList<Listener>();
}
public synchronized void start()
{
if (active)
{
return;
}
active = true;
executor.execute(new Runnable()
{
public void run()
{
synchronized (Monitor.this)
{
if (!active)
{
System.out.println("not active");
return;
}
}
if (!file.exists())
{
System.out.println("does not exist, rescheduled");
executor.schedule(this, RECHECK_AFTER_IF_NOT_EXISTS_SECONDS, TimeUnit.SECONDS);
return;
}
Event event = doStuff(file);
System.out.println("generated " + event);
updateListeners(event);
System.out.println("updated listeners and rescheduled");
executor.schedule(this, CHECK_EVERY_SECONDS, TimeUnit.SECONDS);
}
});
}
private Event doStuff(final File file)
{
return new Event()
{
public String toString()
{
return "event for " + file;
}
};
}
public synchronized void stop()
{
active = false;
}
public void addListener(Listener listener)
{
synchronized (listeners)
{
listeners.add(listener);
}
}
public void removeListener(Listener listener)
{
synchronized (listeners)
{
listeners.remove(listener);
}
}
private void updateListeners(Event event)
{
synchronized (listeners)
{
for (Listener listener : listeners)
{
listener.handle(event);
}
}
}
public static void main(String[] args) throws IOException
{
ScheduledExecutorService executor = Executors.newScheduledThreadPool(4);
File file = new File("test.png");
Monitor monitor = new Monitor(file, executor);
monitor.addListener(new Listener()
{
public void handle(Event event)
{
System.out.println("handling " + event);
}
});
monitor.start();
System.out.println("started...");
System.in.read();
monitor.stop();
System.out.println("done");
executor.shutdown();
}
}

See this post How to start/stop/restart a thread in Java?

I assume you answered your question
one can not restart a dead thread
This link may be helpful to you How to restart thread in java?

A thread in Java cannot be re-started. Every time you need to restart the thread you must make a new one.
That said, you might want to look at:
private void setRandomFile() {
if (!monitoredFile.exists()) {
log.warn("File [" + monitoredFile.getAbsolutePath()
+ "] not exist,will try again after 30 seconds");
try {
Thread.sleep(30 * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
setRandomFile();
return;
}
// ....
}
Here you sleep for 30 seconds if the file does not exist, then recursively call the same function. Now, I don't know what business requirements you have, but if this recursion ran long enough you will run out of stack space. Perhaps you will be better served with a while loop or even better, a little synchronisation like a Semaphore.