I am building an REST API in Java which I would be exposing to the outside world. People who would be invoking the API would have to be registered and would be sending their userid in the request.
There would be a maximum of, say, 10 concurrent threads available for executing the API request. I am maintaining a queue which holds all the request ids to be serviced (the primary key of the DB entry).
I need to implement some fair usage policy as follows -
If there are more than 10 jobs in the queue (i.e more than max number of threads), a user is allowed to execute only one request at a time (the other requests submitted by him/her, if any, would remain in the queue and would be taken up only once his previous request has completed execution). In case there are free threads, i.e. even after allotting threads to requests submitted by different users, then the remaining threads in the thread pool can be distributed among the remaining requests (even if the user who has submitted the request is already holding one thread at that moment).
The current implementation is as follows -
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.PriorityBlockingQueue;
import java.util.concurrent.Semaphore;
public class APIJobExecutor implements Runnable{
private static PriorityBlockingQueue<Integer> jobQueue = new PriorityBlockingQueue<Integer>();
private static ExecutorService jobExecutor = Executors.newCachedThreadPool();
private static final int MAX_THREADS = 10;
private static Semaphore sem = new Semaphore(MAX_THREADS, true);
private APIJobExecutor(){
}
public static void addJob(int jobId)
{
if(!jobQueue.contains(jobId)){
jobQueue.add(new Integer(jobId));
}
}
public void run()
{
while (true) {
try {
sem.acquire();
}catch (InterruptedException e1) {
e1.printStackTrace();
//unable to acquire lock. retry.
continue;
}
try {
Integer jobItem = jobQueue.take();
jobExecutor.submit(new APIJobService(jobItem));
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
sem.release();
}
}
}
}
Edit:
Is there any out of the box Java data structure that gives me this functionality. If not, how do I go about implementing the same?
This is a fairly common "quality of service" pattern and can be solved using the bucket idea within a job-queue. I do not know of a standard Java implementation and/or datastructure for this pattern (maybe the PriorityQueue?), but there should be at least a couple of implementations available (let us know if you find a good one).
I did once create my own implementation and I've tried to de-couple it from the project so that you may modify and use it (add unit-tests!). A couple of notes:
a default-queue is used in case QoS is not needed (e.g. if less than 10 jobs are executing).
the basic idea is to store tasks in lists per QoS-key (e.g. the username), and maintain a separate "who is next" list.
it is intended to be used within a job queue (e.g. part of the APIJobExecutor, not a replacement). Part of the job queue's responsibility is to always call remove(taskId) after a task is executed.
there should be no memory leaks: if there are no tasks/jobs in the queue, all internal maps and lists should be empty.
The code:
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.locks.ReentrantLock;
import java.util.*;
import org.slf4j.*;
/** A FIFO task queue. */
public class QosTaskQueue<TASKTYPE, TASKIDTYPE> {
private static final Logger log = LoggerFactory.getLogger(QosTaskQueue.class);
public static final String EMPTY_STRING = "";
/** Work tasks queued which have no (relevant) QoS key. */
private final ConcurrentLinkedQueue<TASKIDTYPE> defaultQ = new ConcurrentLinkedQueue<TASKIDTYPE>();
private final AtomicInteger taskQSize = new AtomicInteger();
private final Map<TASKIDTYPE, TASKTYPE> queuedTasks = new ConcurrentHashMap<TASKIDTYPE, TASKTYPE>();
/** Amount of tasks in queue before "quality of service" distribution kicks in. */
private int qosThreshold = 10;
/** Indicates if "quality of service" distribution is in effect. */
private volatile boolean usingQos;
/**
* Lock for all modifications to Qos-queues.
* <br>Must be "fair" to ensure adding does not block polling threads forever and vice versa.
*/
private final ReentrantLock qosKeyLock = new ReentrantLock(true);
/*
* Since all QoS modifications can be done by multiple threads simultaneously,
* there is never a good time to add or remove a Qos-key with associated queue.
* There is always a chance that a key is added while being removed and vice versa.
* The simplest solution is to make everything synchronized, which is what qosKeyLock is used for.
*/
private final Map<String, Queue<TASKIDTYPE>> qosQueues = new HashMap<String, Queue<TASKIDTYPE>>();
private final Queue<String> qosTurn = new LinkedList<String>();
public boolean add(TASKTYPE wt, TASKIDTYPE taskId, String qosKey) {
if (queuedTasks.containsKey(taskId)) {
throw new IllegalStateException("Task with ID [" + taskId + "] already enqueued.");
}
queuedTasks.put(taskId, wt);
return addToQ(taskId, qosKey);
}
public TASKTYPE poll() {
TASKIDTYPE taskId = pollQos();
return (taskId == null ? null : queuedTasks.get(taskId));
}
/**
* This method must be called after a task is taken from the queue
* using {#link #poll()} and executed.
*/
public TASKTYPE remove(TASKIDTYPE taskId) {
TASKTYPE wt = queuedTasks.remove(taskId);
if (wt != null) {
taskQSize.decrementAndGet();
}
return wt;
}
private boolean addToQ(TASKIDTYPE taskId, String qosKey) {
if (qosKey == null || qosKey.equals(EMPTY_STRING) || size() < getQosThreshold()) {
defaultQ.add(taskId);
} else {
addSynced(taskId, qosKey);
}
taskQSize.incrementAndGet();
return true;
}
private void addSynced(TASKIDTYPE taskId, String qosKey) {
qosKeyLock.lock();
try {
Queue<TASKIDTYPE> qosQ = qosQueues.get(qosKey);
if (qosQ == null) {
if (!isUsingQos()) {
// Setup QoS mechanics
qosTurn.clear();
qosTurn.add(EMPTY_STRING);
usingQos = true;
}
qosQ = new LinkedList<TASKIDTYPE>();
qosQ.add(taskId);
qosQueues.put(qosKey, qosQ);
qosTurn.add(qosKey);
log.trace("Created QoS queue for {}", qosKey);
} else {
qosQ.add(taskId);
if (log.isTraceEnabled()) {
log.trace("Added task to QoS queue {}, size: " + qosQ.size(), qosKey);
}
}
} finally {
qosKeyLock.unlock();
}
}
private TASKIDTYPE pollQos() {
TASKIDTYPE taskId = null;
qosKeyLock.lock();
try {
taskId = pollQosRecursive();
} finally {
qosKeyLock.unlock();
}
return taskId;
}
/**
* Poll the work task queues according to qosTurn.
* Recursive in case empty QoS queues are removed or defaultQ is empty.
* #return
*/
private TASKIDTYPE pollQosRecursive() {
if (!isUsingQos()) {
// QoS might have been disabled before lock was released or by this recursive method.
return defaultQ.poll();
}
String qosKey = qosTurn.poll();
Queue<TASKIDTYPE> qosQ = (qosKey.equals(EMPTY_STRING) ? defaultQ : qosQueues.get(qosKey));
TASKIDTYPE taskId = qosQ.poll();
if (qosQ == defaultQ) {
// DefaultQ should always be checked, even if it was empty
qosTurn.add(EMPTY_STRING);
if (taskId == null) {
taskId = pollQosRecursive();
} else {
log.trace("Removed task from defaultQ.");
}
} else {
if (taskId == null) {
qosQueues.remove(qosKey);
if (qosQueues.isEmpty()) {
usingQos = false;
}
taskId = pollQosRecursive();
} else {
qosTurn.add(qosKey);
if (log.isTraceEnabled()) {
log.trace("Removed task from QoS queue {}, size: " + qosQ.size(), qosKey);
}
}
}
return taskId;
}
#Override
public String toString() {
StringBuilder sb = new StringBuilder(this.getClass().getName());
sb.append(", size: ").append(size());
sb.append(", number of QoS queues: ").append(qosQueues.size());
return sb.toString();
}
public boolean containsTaskId(TASKIDTYPE wid) {
return queuedTasks.containsKey(wid);
}
public int size() {
return taskQSize.get();
}
public void setQosThreshold(int size) {
this.qosThreshold = size;
}
public int getQosThreshold() {
return qosThreshold;
}
public boolean isUsingQos() {
return usingQos;
}
}
Related
I have a daemon thread running which calls a function (prepareOrder) whenever the cook is not busy and there are orders to be delivered. The prepareOrder calls the orderComplete function after a certain interval of time depending upon the time required to complete the order. Now the problem i am facing is only the last call to the prepareOrder gets displayed on sout.
The daemon
package ui;
import Model.takeOrderModel;
public class daemonThread extends Thread{
//call this method in the main method of driving fucntion
private takeOrderModel orderModel;
daemonThread(takeOrderModel orderModel){
this.orderModel = orderModel;
}
public void assignCook(){
while(true){
int toComplete = orderModel.toCompleteOrders.size();
if ( !orderModel.cookBusy && toComplete>0 ) orderModel.prepareOrder();
}
}
}
The prepare order function.
public void prepareOrder(){
// pick the last element from list
if (toCompleteOrders.size() > 0){
String nextPrepare = toCompleteOrders.get(toCompleteOrders.size()-1);
order orderToComplete = allOrdersPlaced.get(nextPrepare);
completeOrder(orderToComplete);
toCompleteOrders.remove(nextPrepare);
}
}
//Helper function to prepareOrder moves an order from toComplete to prepared order
private void completeOrder(order orderToComplete){
changeCookState();
new java.util.Timer().schedule(
new java.util.TimerTask(){
#Override
public void run() {
changeCookState();
preparedOrders.add(orderToComplete.id);
deliverOrder(orderToComplete.id);
}
}, (long) (orderToComplete.timeToComplete*60)
);
}
public void changeCookState(){
this.cookBusy = !cookBusy;
}
// MODIFIES removes a order from the prepared list and puts it in delivered list
public String deliverOrder(String completedOrder){
preparedOrders.remove(completedOrder);
deliveredOrders.add(completedOrder);
System.out.println(String.format("The order of %s is here", allOrdersPlaced.get(completedOrder).customerName));
return String.format("The order of %s is here", allOrdersPlaced.get(completedOrder).customerName);
}
The main function driving code.
orderMachine.takeNewOrder(fullMeal, "Tom");
orderMachine.takeNewOrder(halfMeal, "Bob");
daemonThread backThread = new daemonThread(orderMachine);
backThread.setDaemon(true);
backThread.assignCook();
Now for me only the last placed order("Bob") gets printed on sout. How can all calls created by Timer.schedule stay in stack.
Edits
The take new order function.
public boolean takeNewOrder(List<item> itemsInOrder, String customerName){
try {
order newOrder = new order(itemsInOrder, customerName);
allOrdersPlaced.put(newOrder.id, newOrder);
toCompleteOrders.add(newOrder.id);
return true;
} catch (Exception e) {
e.printStackTrace();
return false;
}
}
Edit 2
here is the public repo containing the complete code
https://github.com/oreanroy/Share_code_samples/tree/master/takeOrder
The problem in this code is a concurrency bug - the cookBusy variable is being written to from two different threads. To fix this, use an AtomicBoolean instead of a boolean, as this is thread safe.
AtomicBoolean cookBusy = new AtomicBoolean(false);
Use compareAndSet to ensure the shared variable is set to a known value before updating it.
public void changeCookState(boolean busy){
if (!this.cookBusy.compareAndSet(!busy, busy))
{
throw new RuntimeException("shared variable set to unexpected value");
}
}
I have written the below method to avoid concurrency.but when we deploy the code in lower envs(DEV,QA) its working fine But not in PROD because data is huge there and multiple servers will access this code concurrently, when we analyzed using Thread profiler, lock is not releasing and struck at isActive() method at line:27 indifinetely and its taking lot of CPU time and Memory..
Is there any way to modify the code?
public void registerToken(NodeRef nodeRef)
throws IdenticalContentException
{
final AtomicBoolean stillRunning = new AtomicBoolean(true);
String lockToken = null;
String nodeRefToken = getToken(nodeRef);
try
{
lockToken = this.jobLockService.getLock(getLock(nodeRefToken), 30000L, 3000L, 2);
JobLockService.JobLockRefreshCallback callback = new JobLockService.JobLockRefreshCallback()
{
public void lockReleased()
{
stillRunning.set(false);
}
public boolean isActive()
{
return stillRunning.get();
}
};
this.jobLockService.refreshLock(lockToken, getLock(nodeRefToken), 30000L, callback);
if (stillRunning.get()) {
registerNode(nodeRef, nodeRefToken);
}
}
catch (LockAcquisitionException lae)
{
LOG.error("Failed to acquire a lock for metadata checking", lae);
throw lae;
}
finally
{
stillRunning.set(false);
if (lockToken != null) {
this.jobLockService.releaseLock(lockToken, getLock(nodeRefToken));
}
}
}
I need to build a queue where the elements will be added and removed in chronological order by default. But if the client sets the priority flag for the queue I need to be able to pull the elements based on the priority order of the elements.
I am thinking of creating a priority queue backed by a map that keeps track of the queue index in priority order and based on priority flag I can pull the items from the map and pop the item from index from the queue.
However with this approach the question will be, weather I create the map by default or only if the flag is set (considering the cost of creating the map on fly being high, I am inclining towards having it by default).
Please let me know if there is a better way of doing this or if there is an existing implementation that exists.
Here is what I currently have:
import javax.naming.OperationNotSupportedException;
import java.util.Comparator;
import java.util.PriorityQueue;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;
public class DynamicPriorityQueue<ComparableQueueElement> implements IQueue<ComparableQueueElement> {
private static final int CONSTANT_HUNDRED = 100;
private boolean fetchByCustomPriority = false;
private final ReentrantLock lock;
private final PriorityQueue<ComparableQueueElement> queue;
private final PriorityQueue<ComparableQueueElement> customPriorityQueue;
public DynamicPriorityQueue() {
this(null);
}
public DynamicPriorityQueue(Comparator<ComparableQueueElement> comparator) {
this.lock = new ReentrantLock();
this.queue = new PriorityQueue<>(CONSTANT_HUNDRED);
if (comparator != null)
this.customPriorityQueue = new PriorityQueue<ComparableQueueElement>(CONSTANT_HUNDRED, comparator);
else
this.customPriorityQueue = null;
}
public void setFetchByCustomPriority(boolean fetchByCustomPriority) throws OperationNotSupportedException {
if (this.customPriorityQueue == null)
throw new OperationNotSupportedException("Object was created without a custom comparator.");
this.fetchByCustomPriority = fetchByCustomPriority;
}
public void push(ComparableQueueElement t) throws InterruptedException {
if (this.lock.tryLock(CONSTANT_HUNDRED, TimeUnit.MILLISECONDS)) {
try {
this.queue.offer(t);
if (this.customPriorityQueue != null)
this.customPriorityQueue.offer(t);
} finally {
this.lock.unlock();
}
}
}
public ComparableQueueElement peek() {
return this.fetchByCustomPriority ? this.queue.peek()
: (this.customPriorityQueue != null ? this.customPriorityQueue.peek() : null);
}
public ComparableQueueElement pop() throws InterruptedException {
ComparableQueueElement returnElement = null;
if (this.lock.tryLock(CONSTANT_HUNDRED, TimeUnit.MILLISECONDS)) {
try {
if (this.fetchByCustomPriority && this.customPriorityQueue != null) {
returnElement = this.customPriorityQueue.poll();
this.queue.remove(returnElement);
}
else {
returnElement = this.queue.poll();
if (this.customPriorityQueue != null) {
this.customPriorityQueue.remove(returnElement);
}
}
} finally {
this.lock.unlock();
}
}
return returnElement;
}
}
I deleted my comments after rereading the question, it may get complicated. You need to turn a fifo (chronological) queue into a priority queue with a flag. Your map would need to be ordered and be able to hold repeated values. Otherwise you would need to search the map to find the highest priority or search the queue. I wouldn't do it.
EDIT
What about using a wrapping class:
class Pointer<T>{
T element
}
And two queues of Pointers where the queues share the Pointers but they return them differently? The only thing you would need to do is to check that "element" is not null (you set it null when it leaves one of the queues.
The Pointer reference remains in the other queue but you check for null before returning.
EDIT
Your code doesn't have a map.
public ComparableQueueElement peek() {
return this.fetchByCustomPriority ? this.queue.peek()
: (this.customPriorityQueue != null ? this.customPriorityQueue.peek() : null);
}
is not correct. If it is not custom you should peek from this.queue
EDIT
Note that by using a wrapping class you save yourself the remove calls in the other queue. The only overhead added is that you need to check for null when fetching
For me implementation looks fine if your application have the requirement where the flag is changing very frequently. In such case you have both the queues ready to offer or poll object from queue. Although it makes your add operation heavy but retrieval is fast.
But if these changes are not frequent then you can think of re initializing the custom priority queue from FIFO queue only when flag changes.
And perform all your peek,offer operation only on one queue instead of two.
which will be more efficient if FIFO priority is being used.
and if you do that modify your push operation as below -
public void push(ComparableQueueElement t) throws InterruptedException {
if (this.lock.tryLock(CONSTANT_HUNDRED, TimeUnit.MILLISECONDS)) {
try {
this.queue.offer(t);
if (this.fetchByCustomPriority) // add to customPriorityQueue only when flag is enabled
this.customPriorityQueue.offer(t);
} finally {
this.lock.unlock();
}
}
}
So a little background;
I am working on a project in which a servlet is going to release crawlers upon a lot of text files within a file system. I was thinking of dividing the load under multiple threads, for example:
a crawler enters a directory, finds 3 files and 6 directories. it will start processing the files and start a thread with a new crawler for the other directories. So from my creator class I would create a single crawler upon a base directory. The crawler would assess the workload and if deemed needed it would spawn another crawler under another thread.
My crawler class looks like this
package com.fujitsu.spider;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.Serializable;
import java.util.ArrayList;
public class DocumentSpider implements Runnable, Serializable {
private static final long serialVersionUID = 8401649393078703808L;
private Spidermode currentMode = null;
private String URL = null;
private String[] terms = null;
private float score = 0;
private ArrayList<SpiderDataPair> resultList = null;
public enum Spidermode {
FILE, DIRECTORY
}
public DocumentSpider(String resourceURL, Spidermode mode, ArrayList<SpiderDataPair> resultList) {
currentMode = mode;
setURL(resourceURL);
this.setResultList(resultList);
}
#Override
public void run() {
try {
if (currentMode == Spidermode.FILE) {
doCrawlFile();
} else {
doCrawlDirectory();
}
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("SPIDER # " + URL + " HAS FINISHED.");
}
public Spidermode getCurrentMode() {
return currentMode;
}
public void setCurrentMode(Spidermode currentMode) {
this.currentMode = currentMode;
}
public String getURL() {
return URL;
}
public void setURL(String uRL) {
URL = uRL;
}
public void doCrawlFile() throws Exception {
File target = new File(URL);
if (target.isDirectory()) {
throw new Exception(
"This URL points to a directory while the spider is in FILE mode. Please change this spider to FILE mode.");
}
procesFile(target);
}
public void doCrawlDirectory() throws Exception {
File baseDir = new File(URL);
if (!baseDir.isDirectory()) {
throw new Exception(
"This URL points to a FILE while the spider is in DIRECTORY mode. Please change this spider to DIRECTORY mode.");
}
File[] directoryContent = baseDir.listFiles();
for (File f : directoryContent) {
if (f.isDirectory()) {
DocumentSpider spider = new DocumentSpider(f.getPath(), Spidermode.DIRECTORY, this.resultList);
spider.terms = this.terms;
(new Thread(spider)).start();
} else {
DocumentSpider spider = new DocumentSpider(f.getPath(), Spidermode.FILE, this.resultList);
spider.terms = this.terms;
(new Thread(spider)).start();
}
}
}
public void procesDirectory(String target) throws IOException {
File base = new File(target);
File[] directoryContent = base.listFiles();
for (File f : directoryContent) {
if (f.isDirectory()) {
procesDirectory(f.getPath());
} else {
procesFile(f);
}
}
}
public void procesFile(File target) throws IOException {
BufferedReader br = new BufferedReader(new FileReader(target));
String line;
while ((line = br.readLine()) != null) {
String[] words = line.split(" ");
for (String currentWord : words) {
for (String a : terms) {
if (a.toLowerCase().equalsIgnoreCase(currentWord)) {
score += 1f;
}
;
if (currentWord.toLowerCase().contains(a)) {
score += 1f;
}
;
}
}
}
br.close();
resultList.add(new SpiderDataPair(this, URL));
}
public String[] getTerms() {
return terms;
}
public void setTerms(String[] terms) {
this.terms = terms;
}
public float getScore() {
return score;
}
public void setScore(float score) {
this.score = score;
}
public ArrayList<SpiderDataPair> getResultList() {
return resultList;
}
public void setResultList(ArrayList<SpiderDataPair> resultList) {
this.resultList = resultList;
}
}
The problem I am facing is that in my root crawler I have this list of results from every crawler that I want to process further. The operation to process the data from this list is called from the servlet (or main method for this example). However the operations is always called before all of the crawlers have completed their processing. thus launching the operation to process the results too soon, which leads to incomplete data.
I tried solving this using the join methods but unfortunately I cant seems to figure this one out.
package com.fujitsu.spider;
import java.util.ArrayList;
import com.fujitsu.spider.DocumentSpider.Spidermode;
public class Main {
public static void main(String[] args) throws InterruptedException {
ArrayList<SpiderDataPair> results = new ArrayList<SpiderDataPair>();
String [] terms = {"SERVER","CHANGE","MO"};
DocumentSpider spider1 = new DocumentSpider("C:\\Users\\Mark\\workspace\\Spider\\Files", Spidermode.DIRECTORY, results);
spider1.setTerms(terms);
DocumentSpider spider2 = new DocumentSpider("C:\\Users\\Mark\\workspace\\Spider\\File2", Spidermode.DIRECTORY, results);
spider2.setTerms(terms);
Thread t1 = new Thread(spider1);
Thread t2 = new Thread(spider2);
t1.start();
t1.join();
t2.start();
t2.join();
for(SpiderDataPair d : spider1.getResultList()){
System.out.println("PATH -> " + d.getFile() + " SCORE -> " + d.getSpider().getScore());
}
for(SpiderDataPair d : spider2.getResultList()){
System.out.println("PATH -> " + d.getFile() + " SCORE -> " + d.getSpider().getScore());
}
}
}
TL:DR
I really wish to understand this subject so any help would be immensely appreciated!.
You need a couple of changes in your code:
In the spider:
List<Thread> threads = new LinkedList<Thread>();
for (File f : directoryContent) {
if (f.isDirectory()) {
DocumentSpider spider = new DocumentSpider(f.getPath(), Spidermode.DIRECTORY, this.resultList);
spider.terms = this.terms;
Thread thread = new Thread(spider);
threads.add(thread)
thread.start();
} else {
DocumentSpider spider = new DocumentSpider(f.getPath(), Spidermode.FILE, this.resultList);
spider.terms = this.terms;
Thread thread = new Thread(spider);
threads.add(thread)
thread.start();
}
}
for (Thread thread: threads) thread.join()
The idea is to create a new thread for each spider and start it. Once they are all running, you wait until each on is done before the Spider itself finishes. This way each spider thread keeps running until all of its work is done (thus the top thread runs until all children and their children are finished).
You also need to change your runner so that it runs the two spiders in parallel instead of one after another like this:
Thread t1 = new Thread(spider1);
Thread t2 = new Thread(spider2);
t1.start();
t2.start();
t1.join();
t2.join();
You should use a higher-level library than bare Thread for this task. I would suggest looking into ExecutorService in particular and all of java.util.concurrent generally. There are abstractions there that can manage all of the threading issues while providing well-formed tasks a properly protected environment in which to run.
For your specific problem, I would recommend some sort of blocking queue of tasks and a standard producer-consumer architecture. Each task knows how to determine if its path is a file or directory. If it is a file, process the file; if it is a directory, crawl the directory's immediate contents and enqueue new tasks for each sub-path. You could also use some properly-synchronized shared state to cap the number of files processed, depth, etc. Also, the service provides the ability to await termination of its tasks, making the "join" simpler.
With this architecture, you decouple the notion of threads and thread management (handled by the ExecutorService) with your business logic of tasks (typically a Runnable or Callable). The service itself has the ability to tune how to instantiate, such as a fixed maximum number of threads or a scalable number depending on how many concurrent tasks exist (See factory methods on java.util.concurrent.Executors). Threads, which are more expensive than the Runnables they execute, are re-used to conserve resources.
If your objective is primarily something functional that works in production quality, then the library is the way to go. However, if your objective is to understand the lower-level details of thread management, then you may want to investigate the use of latches and perhaps thread groups to manage them at a lower level, exposing the details of the implementation so you can work with the details.
I use Hazelcast as a non-persistent queue between two applications running in a Tomcat.
Problem: QueueListener stops listening to its queue. This means, until a certain point, the following line appears periodically in the log, then it disappears:
LOGGER.debug("No messages on {}, {}", queueName, QueueListener.this.getClass().getSimpleName());
There is no error in the logs. I have several class that extends the QueueListener, all of them listen to a different named queue. One of them just stops and I have no clue why, except one thing: it happens right after handling an item. The descendant class's handle method logs the item I can see that in the logs. Then the "No messages on {queuename}" loglines just disappear. The executor had 2 threads. Both stopped, not sure if at once.
The descendant class's handle method executes a Http request and logs the response. Note that the response did not appear in the logs for the previous two handle call, before the listener stopped.
The descendant class's handle method does not have any catch block so it will not swallow any Exceptions. No exception was logged in the QueueListener.
My question, how to proceed to find the cause of this? Where to look for it?
The application that send messages into this queue runs in the same Tomcat as the one that listens to this queue. Multicast is enabled (see full HazelCast config bellow). There is an other Tomcat that runs on the same host and some other Tomcats running on different hosts, all connecting to this same Hazelcast instance. They're using the same confing.
Hazelcast version: 2.6
QueueListener.java:
package com.mi6.publishers;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
public abstract class QueueListener<T> {
private static final long TIMEOUT = 10000L;
private static final Logger LOGGER = LoggerFactory.getLogger(QueueListener.class);
/**
* queue which is processed
*/
private IQueue<T> queue;
private final String queueName;
#Autowired
private HazelcastInstance instance;
private ExecutorService svc;
private final int threadCount;
private volatile boolean shutdown = false;
/**
* Constructor
*
* #param queueName
* #param threadCount
*/
public QueueListener(String queueName, int threadCount) {
this.queueName = queueName;
this.threadCount = threadCount;
}
/**
* #PostConstuct Start background threads
*/
#PostConstruct
public void init() {
LOGGER.info("Constructing hazelcast listener for {}", getClass().getSimpleName());
if (instance != null) {
queue = instance.getQueue(queueName);
svc = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < threadCount; i++) {
svc.submit(new Runnable() {
#Override
public void run() {
while (!shutdown) {
try {
T item = queue.poll(TIMEOUT, TimeUnit.MILLISECONDS);
if (item != null) {
handle(item);
} else {
LOGGER.debug("No messages on {}, {}", queueName, QueueListener.this.getClass().getSimpleName());
}
} catch (InterruptedException ex) {
// do nothing if interrupted
} catch (Exception ex) {
LOGGER.error("Error while receiving messages from queue:{}", queueName);
LOGGER.error("Error while receiving messages", ex);
}
}
}
});
}
} else {
throw new IllegalStateException("Hazelcast instance cannot be null");
}
}
/**
* call before stop
*/
#PreDestroy
public void destroy() {
shutdown = true;
if (svc != null) {
svc.shutdown();
}
}
/**
* Event handler
*
* #param item
*/
public abstract void handle(T item);
public String getQueueName() {
return queueName;
}
}
This is how Hazelcast is configured:
#Value("${hazelcast.multicast:True}")
private Boolean hazelcastMulticast;
#Value("${hazelcast.group:groupNameNotSet}")
private String hazelcastGroup;
#Bean(destroyMethod = "shutdown")
public HazelcastInstance hazelcastInstance() {
Config cfg = new Config();
cfg.setInstanceName(hazelcastGroup);
NetworkConfig network = cfg.getNetworkConfig();
network.setPortAutoIncrement(true);
Join join = network.getJoin();
join.getMulticastConfig().setEnabled(hazelcastMulticast);
cfg.getGroupConfig().setName(hazelcastGroup);
cfg.getGroupConfig().setPassword(hazelcastGroup);
QueueConfig sms = new QueueConfig();
sms.setName("some-queue-name1");
cfg.addQueueConfig(sms);
QueueConfig flash = new QueueConfig();
flash.setName("some-queue-name2");
cfg.addQueueConfig(flash);
QueueConfig apns = new QueueConfig();
apns.setName("some-queue-name3");
cfg.addQueueConfig(apns);
QueueConfig gcm = new QueueConfig();
gcm.setName("some-queue-name4");
cfg.addQueueConfig(gcm);
return Hazelcast.newHazelcastInstance(cfg);
}