Develop Reference

Java WatchService, perform action on event using threads

I can monitor a directory by registering cwith a WatchKey (there are tons of examples on the web) however this watcher catches every single event. E.g. On windows If am monitoring the d:/temp dir and I create a new .txt file and rename it I get the following events.
ENTRY_CREATE: d:\temp\test\New Text Document.txt
ENTRY_MODIFY: d:\temp\test
ENTRY_DELETE: d:\temp\test\New Text Document.txt
ENTRY_CREATE: d:\temp\test\test.txt
ENTRY_MODIFY: d:\temp\test
I want to perform an action when the new file is created or updated. However I don't want the action to run 5 times in the above example.
My 1st Idea: As I only need to run the action (in this case a push to a private Git server) once every now an then (e.g. check every 10 seconds only if there are changes to the monitored directory and only then perform the push) I thought of having an object with a boolean parameter that I can get and set from within separate threads.
Now this works kinda ok (unless the gurus can help educated me as to why this is a terrible idea) The problem is that if a file event is caught during the SendToGit thread's operation and this operation completes it sets the "Found" parameter to false. Immediately thereafter one of the other events are caught (as in the example above) they will set the "Found" parameter to true again. This is not ideal as I will then run the SendToGit operation immediately again which will be unnecessary.
My 2nd Idea Investigate pausing the check for changes in the MonitorFolder thread until the SendToGit operation is complete (I.e. Keep checking if the ChangesFound Found parameter has been set back to false. When this parameter is false start checking for changes again.
Questions
Is this an acceptable way to go or have I gone down a rabbit hole with no hope of return?
If I go down the road of my 2nd idea what happens if I am busy with the SendToGit operation and a change is made in the monitoring folder? I suspect that this will not be identified and I may miss changes.
The Rest of the code
ChangesFound.java
package com.acme;
public class ChangesFound {
private boolean found = false;
public boolean wereFound() {
return this.found;
}
public void setFound(boolean commitToGit) {
this.found = commitToGit;
}
}
In my main app I start 2 threads.
MonitorFolder.java Monitors the directory and when Watcher events are found set the ChangesFound variable "found" to true.
SendToGit.java Every 10 seconds checks if the ChangesFound variable found is true and if it is performs the push. (or in this case just prints a message)
Here is my App that starts the threads:
package com.acme;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;
public class App {
private static ChangesFound chg;
public static void main(String[] args) throws IOException {
String dirToMonitor = "D:/Temp";
boolean recursive = true;
chg = new ChangesFound();
Runnable r = new SendToGit(chg);
new Thread(r).start();
Path dir = Paths.get(dirToMonitor);
Runnable m = new MonitorFolder(chg, dir, recursive);
new Thread(m).start();
}
}
SendToGit.java
package com.acme;
public class SendToGit implements Runnable {
private ChangesFound changes;
public SendToGit(ChangesFound chg) {
changes = chg;
}
public void run() {
while (true) {
try {
Thread.sleep(10000);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
System.out.println(java.time.LocalDateTime.now() + " [SendToGit] waking up.");
if (changes.wereFound()) {
System.out.println("\t***** CHANGES FOUND push to Git.");
changes.setFound(false);
} else {
System.out.println("\t***** Nothing changed.");
}
}
}
}
MonitorFolder.java (Apologies for the long class I only added this here in case it helps someone else.)
package com.acme;
import java.io.IOException;
import java.nio.file.FileSystems;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import static java.nio.file.LinkOption.NOFOLLOW_LINKS;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import static java.nio.file.StandardWatchEventKinds.ENTRY_CREATE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_DELETE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY;
import static java.nio.file.StandardWatchEventKinds.OVERFLOW;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.HashMap;
import java.util.Map;
public class MonitorFolder implements Runnable {
private static WatchService watcher;
private static Map<WatchKey, Path> keys;
private static boolean recursive;
private static boolean trace = false;
private static boolean commitGit = false;
private static ChangesFound changes;
#SuppressWarnings("unchecked")
static <T> WatchEvent<T> cast(WatchEvent<?> event) {
return (WatchEvent<T>) event;
}
/**
* Creates a WatchService and registers the given directory
*/
MonitorFolder(ChangesFound chg, Path dir, boolean rec) throws IOException {
changes = chg;
watcher = FileSystems.getDefault().newWatchService();
keys = new HashMap<WatchKey, Path>();
recursive = rec;
if (recursive) {
System.out.format("[MonitorFolder] Scanning %s ...\n", dir);
registerAll(dir);
System.out.println("Done.");
} else {
register(dir);
}
// enable trace after initial registration
this.trace = true;
}
/**
* Register the given directory with the WatchService
*/
private static void register(Path dir) throws IOException {
WatchKey key = dir.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY);
if (trace) {
Path prev = keys.get(key);
if (prev == null) {
System.out.format("register: %s\n", dir);
} else {
if (!dir.equals(prev)) {
System.out.format("update: %s -> %s\n", prev, dir);
}
}
}
keys.put(key, dir);
}
/**
* Register the given directory, and all its sub-directories, with the
* WatchService.
*/
private static void registerAll(final Path start) throws IOException {
// register directory and sub-directories
Files.walkFileTree(start, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs)
throws IOException {
register(dir);
return FileVisitResult.CONTINUE;
}
});
}
/**
* Process all events for keys queued to the watcher
*/
public void run() {
for (;;) {
// wait for key to be signalled
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException x) {
return;
}
Path dir = keys.get(key);
if (dir == null) {
System.err.println("WatchKey not recognized!!");
continue;
}
for (WatchEvent<?> event : key.pollEvents()) {
WatchEvent.Kind kind = event.kind();
// TBD - provide example of how OVERFLOW event is handled
if (kind == OVERFLOW) {
System.out.println("Something about Overflow");
continue;
}
// Context for directory entry event is the file name of entry
WatchEvent<Path> ev = cast(event);
Path name = ev.context();
Path child = dir.resolve(name);
// print out event and set ChangesFound object Found parameter to True
System.out.format("[MonitorFolder] " + java.time.LocalDateTime.now() + " - %s: %s\n", event.kind().name(), child);
changes.setFound(true);
// if directory is created, and watching recursively, then
// register it and its sub-directories
if (recursive && (kind == ENTRY_CREATE)) {
try {
if (Files.isDirectory(child, NOFOLLOW_LINKS)) {
registerAll(child);
}
} catch (IOException x) {
// ignore to keep sample readbale
}
}
}
// reset key and remove from set if directory no longer accessible
boolean valid = key.reset();
if (!valid) {
keys.remove(key);
// all directories are inaccessible
if (keys.isEmpty()) {
System.out.println("keys.isEmpty");
break;
}
}
}
}
}

Both of your strategies will lead to issues because the Watch Service is very verbose and sends many messages when maybe one or two is actually needed to your downstream handling - so sometimes you may do unnecessary work or miss events.
When using WatchService you could collate multiple notifications together and pass on as ONE event listing a sets of recent deletes, creates and updates:
DELETE followed by CREATE => sent as UPDATE
CREATE followed by MODIFY => sent as CREATE
CREATE or MODIFY followed by DELETE => sent as DELETE
Instead of calling WatchService.take() and acting on each message, use WatchService.poll(timeout) and only when nothing is returned act on the union of preceeding set of events as one - not individually after each successful poll.
It is easier to decouple the problems as two components so that you don't repeat the WatchService code the next time you need it:
A watch manager which handles watch service + dir registrations and collates the duplicates to send to event listeners as ONE group
A Listener class to receive the group of changes and act on the set.
This example may help illustrate - see WatchExample which is the manager which sets up the registrations BUT passes on much fewer events to the callback defined by setListener. You could set up MonitorFolder like WatchExample to reduce the events discovered, and make your code in SendToGit as a Listener which is called on demand with the aggregated set of fileChange(deletes, creates, updates).
public static void main(String[] args) throws IOException, InterruptedException {
final List<Path> dirs = Arrays.stream(args).map(Path::of).map(Path::toAbsolutePath).collect(Collectors.toList());
Kind<?> [] kinds = { StandardWatchEventKinds.ENTRY_CREATE, StandardWatchEventKinds.ENTRY_MODIFY, StandardWatchEventKinds.ENTRY_DELETE};
// Should launch WatchExample PER Filesystem:
WatchExample w = new WatchExample();
w.setListener(WatchExample::fireEvents);
for(Path dir : dirs)
w.register(kinds, dir);
// For 2 or more WatchExample use: new Thread(w[n]::run).start();
w.run();
}
public class WatchExample implements Runnable {
private final Set<Path> created = new LinkedHashSet<>();
private final Set<Path> updated = new LinkedHashSet<>();
private final Set<Path> deleted = new LinkedHashSet<>();
private volatile boolean appIsRunning = true;
// Decide how sensitive the polling is:
private final int pollmillis = 100;
private WatchService ws;
private Listener listener = WatchExample::fireEvents;
#FunctionalInterface
interface Listener
{
public void fileChange(Set<Path> deleted, Set<Path> created, Set<Path> modified);
}
WatchExample() {
}
public void setListener(Listener listener) {
this.listener = listener;
}
public void shutdown() {
System.out.println("shutdown()");
this.appIsRunning = false;
}
public void run() {
System.out.println();
System.out.println("run() START watch");
System.out.println();
try(WatchService autoclose = ws) {
while(appIsRunning) {
boolean hasPending = created.size() + updated.size() + deleted.size() > 0;
System.out.println((hasPending ? "ws.poll("+pollmillis+")" : "ws.take()")+" as hasPending="+hasPending);
// Use poll if last cycle has some events, as take() may block
WatchKey wk = hasPending ? ws.poll(pollmillis,TimeUnit.MILLISECONDS) : ws.take();
if (wk != null) {
for (WatchEvent<?> event : wk.pollEvents()) {
Path parent = (Path) wk.watchable();
Path eventPath = (Path) event.context();
storeEvent(event.kind(), parent.resolve(eventPath));
}
boolean valid = wk.reset();
if (!valid) {
System.out.println("Check the path, dir may be deleted "+wk);
}
}
System.out.println("PENDING: cre="+created.size()+" mod="+updated.size()+" del="+deleted.size());
// This only sends new notifications when there was NO event this cycle:
if (wk == null && hasPending) {
listener.fileChange(deleted, created, updated);
deleted.clear();
created.clear();
updated.clear();
}
}
}
catch (InterruptedException e) {
System.out.println("Watch was interrupted, sending final updates");
fireEvents(deleted, created, updated);
}
catch (IOException e) {
throw new UncheckedIOException(e);
}
System.out.println("run() END watch");
}
public void register(Kind<?> [] kinds, Path dir) throws IOException {
System.out.println("register watch for "+dir);
// If dirs are from different filesystems WatchService will give errors later
if (this.ws == null) {
ws = dir.getFileSystem().newWatchService();
}
dir.register(ws, kinds);
}
/**
* Save event for later processing by event kind EXCEPT for:
* <li>DELETE followed by CREATE => store as MODIFY
* <li>CREATE followed by MODIFY => store as CREATE
* <li>CREATE or MODIFY followed by DELETE => store as DELETE
*/
private void
storeEvent(Kind<?> kind, Path path) {
System.out.println("STORE "+kind+" path:"+path);
boolean cre = false;
boolean mod = false;
boolean del = kind == StandardWatchEventKinds.ENTRY_DELETE;
if (kind == StandardWatchEventKinds.ENTRY_CREATE) {
mod = deleted.contains(path);
cre = !mod;
}
else if (kind == StandardWatchEventKinds.ENTRY_MODIFY) {
cre = created.contains(path);
mod = !cre;
}
addOrRemove(created, cre, path);
addOrRemove(updated, mod, path);
addOrRemove(deleted, del, path);
}
// Add or remove from the set:
private static void addOrRemove(Set<Path> set, boolean add, Path path) {
if (add) set.add(path);
else set.remove(path);
}
public static void fireEvents(Set<Path> deleted, Set<Path> created, Set<Path> modified) {
System.out.println();
System.out.println("fireEvents START");
for (Path path : deleted)
System.out.println(" DELETED: "+path);
for (Path path : created)
System.out.println(" CREATED: "+path);
for (Path path : modified)
System.out.println(" UPDATED: "+path);
System.out.println("fireEvents END");
System.out.println();
}
}

Java - Pausing a thread pool when it is executing with a JButton

I have a pause mechanism for my events(Runnable) but I am not sure how to use/call it when I need it to be called.
In my class, I have a run method that submits an arraylist full of events to an executor. (see run() on GreenhouseControls). When the Events are submitted, Thread.sleep(eventTime); is called and afterwards the action() inside the event is called.
My program requirement needs a mechanism that pauses all the threads with a method(which will be called by a method when clicked) which can be resumed later on with another button.
Does anyone know how to implement this mechanism?
Here are the files:
GreenhouseControls.java
package control;
/**
* In this exercise we take a different design approach to GreenhouseControls
*
* Compiled/Tested using Eclipse Version: Luna Release (4.4.0)
* TME4 Folder is located in C:\COMP 308\
*/
import java.io.*;
import java.util.regex.*;
import java.util.logging.*;
import java.util.*;
import java.util.concurrent.*;
import java.lang.reflect.*;
import tme4.*;
/**
* GreenhouseControls consists of the Greenhouse's status and methods that
* control what actions will be performed inside the Greenhouse.
* #author Ray Masiclat
*
*/
public class GreenhouseControls extends Controller implements Serializable{
/**
* status is a Set of StatusPair objects which contains information about the
* GreenhouseControls' status. The Default States are initialized in the default constructor.
*/
private Set<StatusPair> status = new HashSet<StatusPair>();
private static String file = "src/error.log";
/**
* Default Constructor - initializes each state of the Greenhouse
*
*/
public GreenhouseControls(){
status.add(new StatusPair<String, Boolean>("light", false));
status.add(new StatusPair<String, Boolean>("water", false));
status.add(new StatusPair<String, Boolean>("fans", false));
status.add(new StatusPair<String, Boolean>("windowok", true));
status.add(new StatusPair<String, Boolean>("poweron", true));
status.add(new StatusPair<String, String>("thermostat", "Day"));
status.add(new StatusPair<String, Integer>("errorcode", 0));
status = Collections.synchronizedSet(status);
}
/**
* Prints out in the console how to use the program.
*/
public static void printUsage() {
System.out.println("Correct format: ");
System.out.println(" java GreenhouseControls -f <filename>, or");
System.out.println(" java GreenhouseControls -d dump.out");
}
/**
* Takes in an errorcode and returns the appropriate Fix that will fix the
* error that occured in the past.
* #param errorcode
* #return fix
*/
public Fixable getFixable(int errorcode){
Fixable fix = null;
switch(errorcode){
case 1:
fix = new FixWindow(this);
break;
case 2:
fix = new PowerOn(this);
break;
default:
System.out.println("No Error");
break;
}
return fix;
}
/**
* shutdown - method creates a Logger that creates an error log which consists of information about the
* reason of why the program was shut down. After logging the error information, it serializes
* the current state of the program in order for it to be fixed/restored in the future.
* #throws IOException
*/
public void shutdown() throws IOException{
System.err.println("System Shutting Down");
Logger logger = Logger.getLogger("ControllerException");
FileHandler fh;
try {
fh = new FileHandler("src/error.log");
logger.addHandler(fh);
SimpleFormatter formatter = new SimpleFormatter();
fh.setFormatter(formatter);
} catch (SecurityException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
//Check for errorcode using getError method.
int checkError = this.getError();
if(checkError == 1){
logger.info("Window Malfunction");
} else if (checkError == 2){
logger.info("Power Outage");
} else {
logger.info("No Error");
}
/**
* Serialize the current state and output it onto the src/ directory as dump.out
*/
ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream("src/dump.out"));
out.writeObject(this);
out.writeObject(getEventList());
out.close();
System.exit(0);
}
/**
* getStatus - returns the status Set which consists of information about the current state of the program.
* #return status
*/
public Set<StatusPair> getStatus(){
return status;
}
/**
* getError - Assigns an integer which will be used when logging the error.
* The errorNum is retrieved using the status Set
* #return errorNum
*/
public int getError(){
int errorNum = 0;
//Iterate through the status Set
for(StatusPair sp : status){
//Check for the name "errorcode" and initializes errorNum from its status
if(sp.getName().equals("errorcode")){
errorNum = (int) sp.getStatus();
}
}
return errorNum;
}
/**
* getEventList - returns a List of Events in the program.
* #return eventList
*/
public List<Event> getEventList(){
return eventList;
}
/**
* Create a method in GreenhouseControls called setVariable to
* handle updating to this collection. Use the synchronization feature
* in java to ensure that two Event classes are not trying to add to
* the structure at the same time.
* s - represents the name of the status
* o - represents the status it is going to be replaced with
* #param s
* #param o
*/
public void setVariable(String s, Object o){
for(StatusPair sp : getStatus()){
if(sp.getName().equals(s))
{
sp.setStatus(o);
}
}
}
/**
* addStatus - Adds a new status in the status Set and it is used
* if the program cannot find a name in the status Set.
* #param name
* #param status
*/
public void addStatus(String name, Object status){
getStatus().add(new StatusPair<String, Object>(name,status));
}
/**
* run - creates a thread array which will be used to run Events from
* the text file. Then a for loop is created to fill up the thread
* array with Events on each index. Then the thread is started once
* an index is initialized. After an event is added in to the thread
* array it is removed from the events list.
*/
public void run(){
ExecutorService exec = Executors.newCachedThreadPool();
if(eventList.size() == 1){
exec.submit(eventList.get(0));
eventList.remove(eventList.get(0));
} else {
for(Event e : eventList){
exec.submit(e);
}
}
exec.shutdown();
/*while(eventList.size() > 0){
for(int i = 0; i < eventList.size(); i++){
exec.submit(eventList.get(i));
}
}*/
/*Thread[] threads = new Thread[eventList.size()];
while(eventList.size() > 0)
for(int i = 0; i < eventList.size(); i++){
threads[i] = new Thread(eventList.get(i));
threads[i].start();
eventList.remove(i);
}*/
}
} ///:~
Event.java
/**
* Make Event implements Runnable so that each type of event provides
* its own timing. Each event file should be a class of its own. Change
* the rest of the design to simplify this model.
*
* Assignment: TME4
* #author Ray Masiclat
* #studentid 3231308
* #date July 27, 2015
*
*
* Compiled/Tested using Eclipse Version: Luna Release (4.4.0)
* TME4 Folder is located in C:\COMP 308\
*/
package tme4;
import java.io.*;
import java.util.logging.FileHandler;
import java.util.logging.Logger;
import java.util.logging.SimpleFormatter;
import gui.*;
import control.*;
/**
* Event class that implements Runnable and Serializable
* #author Ray Masiclat
*
*/
public abstract class Event implements Runnable, Serializable{
/**
* eventTime - represents time it takes for the program to wait and then
* call its action method.
*/
protected long eventTime;
protected GreenhouseControls gcontrol;
protected boolean suspended = false;
/**
* Event class constructor which is used to initialize the Event's eventTime and
* GreenhouseControls object which is used for the Event to have access to its status
* variables.
* #param gc
* #param eventTime
*/
public Event(GreenhouseControls gc,long eventTime){
this.eventTime = eventTime;
this.gcontrol = gc;
}
/**
* getTime - returns the event's eventTime initialized from the constructor.
* #return eventTime
*/
public long getTime(){
return eventTime;
}
/**
* setTime - sets the eventTime
*/
public void setTime(long eventTime){
this.eventTime = eventTime;
}
/**
* run - Event class' run method is called when the Event is added in to the Thread
* and then "started". This method puts the Thread to sleep for however long the
* eventTime is. Afterwards, once it is done, it tries to run the Event's action
* method. If the action method throws an error, it is caught by the try-catch
* block which calls the GreenhouseControls object's shutdown which outputs an error.log
* file and serializes the current state of the GreenhouseControls object to a dump.out file.
*/
public void run(){
try {
synchronized(this){
while(suspended){
try {
wait();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Thread.sleep(eventTime);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
this.action();
} catch (ControllerException e) {
//Use shutdown to create error log
try {
gcontrol.shutdown();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
e.printStackTrace();
}
}
public void suspend(){
suspended = true;
}
public synchronized void resume(){
suspended = false;
notify();
}
/**
* abstract method used to call an Event's action method in the run method.
* #throws ControllerException
*/
public abstract void action() throws ControllerException;
} ///:~
(Pastebin: GreenhouseControls.java
and Event.java)

Something like this:
boolean stop = false;
You can use a suspending/stopping object, like so
public class Stopping {
public synchronized void checkStop() {
if (stopped) {
wait(); // wait until stop has been cleared
}
}
public synchronized void setStop(boolean val) {
stop = val;
if (val) {
notifyAll();
}
}
}
Create one of these and share it with all your threads, making sure to call checkStop() frequently. Alternatively, you could make the methods in this class static, and then your threads wouldn't need an object reference - they could just call Stopping.checkStop();

How to implement fair usage policy of threads in API backend?

I am building an REST API in Java which I would be exposing to the outside world. People who would be invoking the API would have to be registered and would be sending their userid in the request.
There would be a maximum of, say, 10 concurrent threads available for executing the API request. I am maintaining a queue which holds all the request ids to be serviced (the primary key of the DB entry).
I need to implement some fair usage policy as follows -
If there are more than 10 jobs in the queue (i.e more than max number of threads), a user is allowed to execute only one request at a time (the other requests submitted by him/her, if any, would remain in the queue and would be taken up only once his previous request has completed execution). In case there are free threads, i.e. even after allotting threads to requests submitted by different users, then the remaining threads in the thread pool can be distributed among the remaining requests (even if the user who has submitted the request is already holding one thread at that moment).
The current implementation is as follows -
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.PriorityBlockingQueue;
import java.util.concurrent.Semaphore;
public class APIJobExecutor implements Runnable{
private static PriorityBlockingQueue<Integer> jobQueue = new PriorityBlockingQueue<Integer>();
private static ExecutorService jobExecutor = Executors.newCachedThreadPool();
private static final int MAX_THREADS = 10;
private static Semaphore sem = new Semaphore(MAX_THREADS, true);
private APIJobExecutor(){
}
public static void addJob(int jobId)
{
if(!jobQueue.contains(jobId)){
jobQueue.add(new Integer(jobId));
}
}
public void run()
{
while (true) {
try {
sem.acquire();
}catch (InterruptedException e1) {
e1.printStackTrace();
//unable to acquire lock. retry.
continue;
}
try {
Integer jobItem = jobQueue.take();
jobExecutor.submit(new APIJobService(jobItem));
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
sem.release();
}
}
}
}
Edit:
Is there any out of the box Java data structure that gives me this functionality. If not, how do I go about implementing the same?

This is a fairly common "quality of service" pattern and can be solved using the bucket idea within a job-queue. I do not know of a standard Java implementation and/or datastructure for this pattern (maybe the PriorityQueue?), but there should be at least a couple of implementations available (let us know if you find a good one).
I did once create my own implementation and I've tried to de-couple it from the project so that you may modify and use it (add unit-tests!). A couple of notes:
a default-queue is used in case QoS is not needed (e.g. if less than 10 jobs are executing).
the basic idea is to store tasks in lists per QoS-key (e.g. the username), and maintain a separate "who is next" list.
it is intended to be used within a job queue (e.g. part of the APIJobExecutor, not a replacement). Part of the job queue's responsibility is to always call remove(taskId) after a task is executed.
there should be no memory leaks: if there are no tasks/jobs in the queue, all internal maps and lists should be empty.
The code:
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.locks.ReentrantLock;
import java.util.*;
import org.slf4j.*;
/** A FIFO task queue. */
public class QosTaskQueue<TASKTYPE, TASKIDTYPE> {
private static final Logger log = LoggerFactory.getLogger(QosTaskQueue.class);
public static final String EMPTY_STRING = "";
/** Work tasks queued which have no (relevant) QoS key. */
private final ConcurrentLinkedQueue<TASKIDTYPE> defaultQ = new ConcurrentLinkedQueue<TASKIDTYPE>();
private final AtomicInteger taskQSize = new AtomicInteger();
private final Map<TASKIDTYPE, TASKTYPE> queuedTasks = new ConcurrentHashMap<TASKIDTYPE, TASKTYPE>();
/** Amount of tasks in queue before "quality of service" distribution kicks in. */
private int qosThreshold = 10;
/** Indicates if "quality of service" distribution is in effect. */
private volatile boolean usingQos;
/**
* Lock for all modifications to Qos-queues.
* <br>Must be "fair" to ensure adding does not block polling threads forever and vice versa.
*/
private final ReentrantLock qosKeyLock = new ReentrantLock(true);
/*
* Since all QoS modifications can be done by multiple threads simultaneously,
* there is never a good time to add or remove a Qos-key with associated queue.
* There is always a chance that a key is added while being removed and vice versa.
* The simplest solution is to make everything synchronized, which is what qosKeyLock is used for.
*/
private final Map<String, Queue<TASKIDTYPE>> qosQueues = new HashMap<String, Queue<TASKIDTYPE>>();
private final Queue<String> qosTurn = new LinkedList<String>();
public boolean add(TASKTYPE wt, TASKIDTYPE taskId, String qosKey) {
if (queuedTasks.containsKey(taskId)) {
throw new IllegalStateException("Task with ID [" + taskId + "] already enqueued.");
}
queuedTasks.put(taskId, wt);
return addToQ(taskId, qosKey);
}
public TASKTYPE poll() {
TASKIDTYPE taskId = pollQos();
return (taskId == null ? null : queuedTasks.get(taskId));
}
/**
* This method must be called after a task is taken from the queue
* using {#link #poll()} and executed.
*/
public TASKTYPE remove(TASKIDTYPE taskId) {
TASKTYPE wt = queuedTasks.remove(taskId);
if (wt != null) {
taskQSize.decrementAndGet();
}
return wt;
}
private boolean addToQ(TASKIDTYPE taskId, String qosKey) {
if (qosKey == null || qosKey.equals(EMPTY_STRING) || size() < getQosThreshold()) {
defaultQ.add(taskId);
} else {
addSynced(taskId, qosKey);
}
taskQSize.incrementAndGet();
return true;
}
private void addSynced(TASKIDTYPE taskId, String qosKey) {
qosKeyLock.lock();
try {
Queue<TASKIDTYPE> qosQ = qosQueues.get(qosKey);
if (qosQ == null) {
if (!isUsingQos()) {
// Setup QoS mechanics
qosTurn.clear();
qosTurn.add(EMPTY_STRING);
usingQos = true;
}
qosQ = new LinkedList<TASKIDTYPE>();
qosQ.add(taskId);
qosQueues.put(qosKey, qosQ);
qosTurn.add(qosKey);
log.trace("Created QoS queue for {}", qosKey);
} else {
qosQ.add(taskId);
if (log.isTraceEnabled()) {
log.trace("Added task to QoS queue {}, size: " + qosQ.size(), qosKey);
}
}
} finally {
qosKeyLock.unlock();
}
}
private TASKIDTYPE pollQos() {
TASKIDTYPE taskId = null;
qosKeyLock.lock();
try {
taskId = pollQosRecursive();
} finally {
qosKeyLock.unlock();
}
return taskId;
}
/**
* Poll the work task queues according to qosTurn.
* Recursive in case empty QoS queues are removed or defaultQ is empty.
* #return
*/
private TASKIDTYPE pollQosRecursive() {
if (!isUsingQos()) {
// QoS might have been disabled before lock was released or by this recursive method.
return defaultQ.poll();
}
String qosKey = qosTurn.poll();
Queue<TASKIDTYPE> qosQ = (qosKey.equals(EMPTY_STRING) ? defaultQ : qosQueues.get(qosKey));
TASKIDTYPE taskId = qosQ.poll();
if (qosQ == defaultQ) {
// DefaultQ should always be checked, even if it was empty
qosTurn.add(EMPTY_STRING);
if (taskId == null) {
taskId = pollQosRecursive();
} else {
log.trace("Removed task from defaultQ.");
}
} else {
if (taskId == null) {
qosQueues.remove(qosKey);
if (qosQueues.isEmpty()) {
usingQos = false;
}
taskId = pollQosRecursive();
} else {
qosTurn.add(qosKey);
if (log.isTraceEnabled()) {
log.trace("Removed task from QoS queue {}, size: " + qosQ.size(), qosKey);
}
}
}
return taskId;
}
#Override
public String toString() {
StringBuilder sb = new StringBuilder(this.getClass().getName());
sb.append(", size: ").append(size());
sb.append(", number of QoS queues: ").append(qosQueues.size());
return sb.toString();
}
public boolean containsTaskId(TASKIDTYPE wid) {
return queuedTasks.containsKey(wid);
}
public int size() {
return taskQSize.get();
}
public void setQosThreshold(int size) {
this.qosThreshold = size;
}
public int getQosThreshold() {
return qosThreshold;
}
public boolean isUsingQos() {
return usingQos;
}
}

Hazelcast queue stops working, where to look for the error?

I use Hazelcast as a non-persistent queue between two applications running in a Tomcat.
Problem: QueueListener stops listening to its queue. This means, until a certain point, the following line appears periodically in the log, then it disappears:
LOGGER.debug("No messages on {}, {}", queueName, QueueListener.this.getClass().getSimpleName());
There is no error in the logs. I have several class that extends the QueueListener, all of them listen to a different named queue. One of them just stops and I have no clue why, except one thing: it happens right after handling an item. The descendant class's handle method logs the item I can see that in the logs. Then the "No messages on {queuename}" loglines just disappear. The executor had 2 threads. Both stopped, not sure if at once.
The descendant class's handle method executes a Http request and logs the response. Note that the response did not appear in the logs for the previous two handle call, before the listener stopped.
The descendant class's handle method does not have any catch block so it will not swallow any Exceptions. No exception was logged in the QueueListener.
My question, how to proceed to find the cause of this? Where to look for it?
The application that send messages into this queue runs in the same Tomcat as the one that listens to this queue. Multicast is enabled (see full HazelCast config bellow). There is an other Tomcat that runs on the same host and some other Tomcats running on different hosts, all connecting to this same Hazelcast instance. They're using the same confing.
Hazelcast version: 2.6
QueueListener.java:
package com.mi6.publishers;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
public abstract class QueueListener<T> {
private static final long TIMEOUT = 10000L;
private static final Logger LOGGER = LoggerFactory.getLogger(QueueListener.class);
/**
* queue which is processed
*/
private IQueue<T> queue;
private final String queueName;
#Autowired
private HazelcastInstance instance;
private ExecutorService svc;
private final int threadCount;
private volatile boolean shutdown = false;
/**
* Constructor
*
* #param queueName
* #param threadCount
*/
public QueueListener(String queueName, int threadCount) {
this.queueName = queueName;
this.threadCount = threadCount;
}
/**
* #PostConstuct Start background threads
*/
#PostConstruct
public void init() {
LOGGER.info("Constructing hazelcast listener for {}", getClass().getSimpleName());
if (instance != null) {
queue = instance.getQueue(queueName);
svc = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < threadCount; i++) {
svc.submit(new Runnable() {
#Override
public void run() {
while (!shutdown) {
try {
T item = queue.poll(TIMEOUT, TimeUnit.MILLISECONDS);
if (item != null) {
handle(item);
} else {
LOGGER.debug("No messages on {}, {}", queueName, QueueListener.this.getClass().getSimpleName());
}
} catch (InterruptedException ex) {
// do nothing if interrupted
} catch (Exception ex) {
LOGGER.error("Error while receiving messages from queue:{}", queueName);
LOGGER.error("Error while receiving messages", ex);
}
}
}
});
}
} else {
throw new IllegalStateException("Hazelcast instance cannot be null");
}
}
/**
* call before stop
*/
#PreDestroy
public void destroy() {
shutdown = true;
if (svc != null) {
svc.shutdown();
}
}
/**
* Event handler
*
* #param item
*/
public abstract void handle(T item);
public String getQueueName() {
return queueName;
}
}
This is how Hazelcast is configured:
#Value("${hazelcast.multicast:True}")
private Boolean hazelcastMulticast;
#Value("${hazelcast.group:groupNameNotSet}")
private String hazelcastGroup;
#Bean(destroyMethod = "shutdown")
public HazelcastInstance hazelcastInstance() {
Config cfg = new Config();
cfg.setInstanceName(hazelcastGroup);
NetworkConfig network = cfg.getNetworkConfig();
network.setPortAutoIncrement(true);
Join join = network.getJoin();
join.getMulticastConfig().setEnabled(hazelcastMulticast);
cfg.getGroupConfig().setName(hazelcastGroup);
cfg.getGroupConfig().setPassword(hazelcastGroup);
QueueConfig sms = new QueueConfig();
sms.setName("some-queue-name1");
cfg.addQueueConfig(sms);
QueueConfig flash = new QueueConfig();
flash.setName("some-queue-name2");
cfg.addQueueConfig(flash);
QueueConfig apns = new QueueConfig();
apns.setName("some-queue-name3");
cfg.addQueueConfig(apns);
QueueConfig gcm = new QueueConfig();
gcm.setName("some-queue-name4");
cfg.addQueueConfig(gcm);
return Hazelcast.newHazelcastInstance(cfg);
}

How do I retrieve the modified date for the file?

In my program, I am trying to get the last modified date for some items in my box folders, but I am unable to access that information.
According to the documentation for the box API and the javadoc for the java library, any of the following methods of BoxTypedObject should return the information I want:
getModifiedAt() should return a String in ISO 8601 format for the date.
getModifiedDate() should return a Date object for the date.
getValue("modified_at") should also return a String in ISO 8601 format for the date.
getExtraData("modified_at") is also a possible way, although I am not as sure of this one.
However, none of those methods has worked for me; they all just returh null.
The (vastly simplified) code that I am using to retrieve the dates is as follows, with one of the above methods substituted for the comment block:
private static void printAll(BoxFolder boxFolder){
for(BoxTypedObject file : boxFolder.getItemCollection().getEntries())
System.out.printf("[\"%1$s\" %2$s]%n",
file.getValue("name"), file./*[???]*/);
}
The other fields all return the correct values, only when I try to get the date does it fail on me.
How do I retrieve the modified date for the BoxTypedObjects?
EDIT: I have figured out one way to get it, but it is somewhat slow.
client.getFilesManager().getFile(file.getId(), null).getModifiedAt()
retrieves the date. I am still interested if there is a better way to do it, though.
Additional information (may or may not be relevant to the problem):
The authentication of the box client is handled by the following class:
import java.awt.Desktop;
import java.io.*;
import java.net.*;
import com.box.boxjavalibv2.BoxClient;
import com.box.boxjavalibv2.dao.BoxOAuthToken;
import com.box.boxjavalibv2.exceptions.*;
import com.box.boxjavalibv2.requests.requestobjects.BoxOAuthRequestObject;
import com.box.restclientv2.exceptions.BoxRestException;
/**
* This class handles the storage and use of authentication keys, to
* simplify the process of obtaining n authenticated client. This class
* will store refresh keys in a file, so that it can authenticate a client
* without needing for user intervention.
* <p>
* Copyright 2013 Mallick Mechanical, Inc.
*
* #author Anson Mansfield
*/
public class Authenticator {
/**
* Constructs an {#code Authenticator} for use obtaining
* authenticated {#Code BoxClient}s
*
* #param key The OAuth client id and application key.
* #param secret The OAuth client secret.
* #param authFile The file to be used for storing authentications
* for later use.
*/
public Authenticator(String key, String secret, File authFile){
this.key = key;
this.secret = secret;
this.authFile = authFile;
}
/**
* Constructs a new {#Code BoxClient} object, authenticates it,
* and returns it.
*/
public BoxClient getAuthenticatedClient(){
BoxClient client = new BoxClient(key,secret);
client.authenticate(getToken(client));
return client;
}
public final String host = "http://localhost";
public final int port = 4000;
public final String key, secret;
public final File authFile;
public final String url = "https://www.box.com/api/oauth2/authorize?response_type=code&client_id=";
/**
* Obtains a token that can be used to authenticate the box client,
* and stores its refresh value in a file, so it can be used later.
* #param client The client to obtain a token for.
* #return A token that can be used to authenticate the client, or
* {#code null} if one could not be obtained.
*/
private BoxOAuthToken getToken(BoxClient client){
BoxOAuthToken token = null;
try{
if((token = getOldToken(client)) != null) return token;
if((token = getNewToken(client)) != null) return token;
return token;
}finally{
writeNewToken(token);
}
}
/**
* Attempts to write a token's refresh token to a file.
* #param token The token whose refresh value is to be written.
*/
private void writeNewToken(BoxOAuthToken token) {
if(token != null)
try(BufferedWriter out = new BufferedWriter(new FileWriter(authFile))){
out.write(token.getRefreshToken());
}catch(IOException ex){
System.out.println("couldn't update new token");
}
}
/**
* Reads the last session's refresh token from a file and attempts
* to get a new authentication token with it.
* #param client The client for which the authentication token is for.
* #return The token obtained from the refresh, or {#code null} if one
* could not be obtained.
*/
private BoxOAuthToken getOldToken(BoxClient client) {
System.out.println("attempting to use old token");
BoxOAuthToken token = null;
try(BufferedReader in = new BufferedReader(new FileReader(authFile))){
token = client.getOAuthManager().refreshOAuth(
BoxOAuthRequestObject.refreshOAuthRequestObject(
in.readLine(), key, secret
));
System.out.println("refreshed old token");
}catch(IOException ex){
System.out.println("couldn't read old token");
} catch(BoxRestException | BoxServerException | AuthFatalFailureException ex){
System.out.println("couldn't refresh old token");
}
return token;
}
/**
* Connects to the OAuth server and gets a new authentication token.
* #param client The client to get a token for.
* #return The new token obtained from the server, or {#code null} if one could not be obtained.
*/
private BoxOAuthToken getNewToken(BoxClient client) {
System.out.println("attempting to get new token");
BoxOAuthToken token = null;
try {
Desktop.getDesktop().browse(java.net.URI.create(url + key));
token = client.getOAuthManager().createOAuth(
BoxOAuthRequestObject.createOAuthRequestObject(getCode(), key, secret, host + port)
);
} catch (BoxRestException | BoxServerException | AuthFatalFailureException | IOException ex) {
ex.printStackTrace();
return null;
}
return token;
}
/**
* This listens on the configured port for the code included in the callback.
* It also deploys a script on the receiving socket to close the browser tab navigating to it.
* #return The authentication code to generate a token with.
*/
private String getCode(){
try (ServerSocket serverSocket = new ServerSocket(port);
Socket socket = serverSocket.accept();
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter out = new PrintWriter(socket.getOutputStream());){
out.println("<script type=\"text/javascript\">");
out.println( "window.open('', '_self', '');");
out.println( "window.close();");
out.println("</script>"); //Close the tab
while (true){
String code = "";
code = in.readLine ();
System.out.println(code);
String match = "code";
int loc = code.indexOf(match);
if( loc >0 ) {
int httpstr = code.indexOf("HTTP")-1;
code = code.substring(code.indexOf(match), httpstr);
String parts[] = code.split("=");
code=parts[1];
return code;
} else {
// It doesn't have a code
}
}
} catch (IOException | NullPointerException e) {
return "";
}
}
}
The actual class that will be getting the modified field (not yet finished, though):
import java.io.File;
import java.util.Scanner;
import com.box.boxjavalibv2.BoxClient;
import com.box.boxjavalibv2.dao.BoxFolder;
import com.box.boxjavalibv2.dao.BoxTypedObject;
/**
* Copyright 2013 Mallick Mechanical, Inc.
*
* #author Anson Mansfield
*/
public class BoxStuff {
static BoxClient client;
public void main(String ... args) throws Exception {
client = new Authenticator(args[0], args[1], new File(args[2]))
.getAuthenticatedClient();
userSelectFolder("Select the project folder");
}
private static BoxFolder userSelectFolder(String prompt) throws Exception{
Scanner kbd;
if(System.console()!=null)
kbd = new Scanner(System.console().reader());
else
kbd = new Scanner(System.in);
String line = "";
System.out.println();
System.out.println(prompt);
System.out.println("(leave prompt blank to select folder)");
BoxFolder current = client.getFoldersManager().getFolder("0", null);
printAll(current);
System.out.print("select>");
while(!(line = kbd.nextLine()).isEmpty()){
BoxFolder next = select(current, Integer.parseInt(line));
if(next != null) current = next;
printAll(current);
System.out.print("select>");
}
return current;
}
private static void printAll(BoxFolder boxFolder){
int idx=0;
System.out.println(" 0:[parent folder]");
for(BoxTypedObject file : boxFolder
.getItemCollection()
.getEntries()){
if(file.getType().equals("folder")){
System.out.printf("%1$3d:[%2$-32s %3$-6s %4$-9s]%n",
++idx, format((String) file.getValue("name"),30), file.getType(), file.getId());
} else {
System.out.printf(" [%1$-32s %2$-6s %3$-9s Edit:%4$s]%n",
format((String) file.getValue("name"),32), file.getType(), file.getId(), file.getExtraData("modified_at"));
}
}
}
private static String format(CharSequence source, int length){
StringBuilder b = new StringBuilder(length);
b.append('"');
if(source.length() > 30)
b.append(source.subSequence(0, 29)).append('~');
else
b.append(String.format("%1$-30s",source));
b.append('"');
return b.toString();
}
private static BoxFolder select(BoxFolder boxFolder, int i) throws Exception{
int idx=0;
for(BoxTypedObject file : boxFolder.getItemCollection().getEntries()){
if(file.getType().equals("folder") && ++idx == i){
return client.getFoldersManager().getFolder(file.getId(), null);
}
}
if(idx==0){
if(boxFolder.getParent() == null)
return client.getFoldersManager().getFolder("0", null);
else
return client.getFoldersManager().getFolder(boxFolder.getParent().getId(), null);
}else{
System.out.println("Selection is out of range!");
return boxFolder;
}
}
}
If anyone else wants to use these classes for something, just ask me. It is probably OK (they are a mechanical contractor, not a software company), I just need to clear it with my boss (This code does still belong to the company).

This is actually a little tricky. The api call to get folder items by default only return children items with some default fields, they don't include fields like modified_at. However if you supply extra fields parameters you should be able to get them.
Here is what you can do when using the getFolderItems method(this is also in the readme in github):
BoxFolderRequestObject requestObj =
BoxFolderRequestObject.getFolderItemsRequestObject(30, 20)
.addField(BoxFolder.FIELD_NAME)
.addField(BoxFolder.FIELD_MODIFIED_AT);
BoxCollection collection =
boxClient.getFoldersManager().getFolderItems(folderId, requestObj);
There is another tricky thing here though, after you supply these fields, the result children items will only contain the supplied fields(plus some basic fields), so make sure you add all the fields you want.

Here is one way I figured out to get it (that actually works):
client.getFilesManager().getFile(file.getId(), null).getModifiedAt()
This, however, is somewhat slow, so I would greatly appreciate it if someone else knows a faster solution.