Develop Reference

Dynamic priorities for a heterogeneous task set

I've got a bunch of repeating tasks to schedule. They query the database to find out what to do and then execute some action like statistics updates, sending emails, fetching files and importing them. Currently, there are maybe ten of them and this number it's expected to grow a lot. I'm not given any timing constraints, actually, it's my job to choose an algorithm so that nobody complains. :D
Currently, I'm using an ad-hoc combination of threads and periodically scheduled tasks like
for the most important task, there's an own thread falling back to a short sleep when idle (from which it can be woken up, when new important work arrives).
another important task is scheduled once per hour in its own thread
medium importance tasks are scheduled periodically to "fill the holes", so that probably only one of them runs at any moment
the least important tasks are all processed by a single dedicated thread
It seems to work well at the moment, but it's not future-proof and it doesn't feel right for these reasons:
As the queue for the least important tasks may grow a lot, such tasks may be delayed indefinitely.
Filling the holes may go wrong and there may be many tasks running at once.
The number of tasks running at any given moment should depend on the server load. (*)
(*) It's primarily a web server and serving requests is actually the highest priority. Getting a separate server wouldn't help, as the bottleneck is usually the database. Currently, it works fine, but I'm looking for a better solution as we hope that the load grows by a factor of 100 in a year or two.
My idea is to increase the priority of a job, when it was delayed too much. For example, there are statistics running hourly and delaying them by a few hours is no big deal, but it shouldn't be a whole day and it mustn't be a whole week.
I'd be happy to replace all my AbstractExecutionThreadServices and AbstractScheduledServices by something working like follows:
Start the highest priority tasks immediately, no matter what.
Start the medium priority tasks only when the total load is "small".
Start the lowest priority tasks only when the system is "mostly idle".
Increase the priorities for delayed tasks using a supplied formula.
This surely sounds pretty fuzzy and getting it more precise is a part of what I'm asking. My competing goals are
Never delay the important tasks needlessly.
Never let too many concurrently running tasks slow down the server too much.
There are no hard deadlines and there's no need to minimize the number of threads used. I don't insist on a solution doing exactly what I described, I'm not looking for a library (nor I insist on reinventing the wheel). I don't think that a cron-like scheduler is the right solution.

Working with the ExecutorService model, the classic solution to reordering executor tasks is to create a ThreadPoolExecutor with a PriorityBlockingQueue feeding it the tasks - as described here.
However needing to schedule the tasks as well puts a spin on it. ScheduledThreadPoolExecutor uses an internal custom BlockingQueue to feed the tasks in when the schedule is ready, but as I think you're well aware, it's not easily open to further customisation.
At a glance, DelayQueue looks like it fits the bill perfectly - it can prioritise the next Delayed element or task. And this handles a late decision by Delayed.getDelay() about whether it is ready to go.
The fly in the ointment with this plan is when you try to pass something like DelayQueue<DelayedRunnable> into the constructor of ThreadPoolExecutor. This will only accept a BlockingQueue<Runnable>, not BlockingQueue<? extends Runnable>.
One way out of this is to create a minimum implementation of BlockingQueue<Runnable> that delegates to a BlockingQueue. The basics are here:
public class BlockingDelayQueue extends AbstractQueue<Runnable>
implements BlockingQueue<Runnable> {
private final DelayQueue<DelayedRunnable> delayQueue;
public BlockingDelayQueue(DelayQueue<DelayedRunnable> delayQueue) {
this.delayQueue = delayQueue;
}
#Override
public boolean isEmpty() {
return delayQueue.isEmpty();
}
#Override
public Runnable poll(long timeout, TimeUnit unit)
throws InterruptedException {
DelayedRunnable delayedRunnable = delayQueue.poll(timeout, unit);
if (delayedRunnable == null)
return null;
return delayedRunnable.getCommand();
}
...
}
The experimental version of DelayedRunnable used to prove the idea there uses a simple Priority enum that checks the 'busyness' of the executor:
LOW {
boolean isReady(ThreadPoolExecutor executor) {
return executor.getActiveCount() == 0;
}
},
MEDIUM {
boolean isReady(ThreadPoolExecutor executor) {
return executor.getActiveCount() <= 1;
}
},
HIGH {
boolean isReady(ThreadPoolExecutor executor) {
return true;
}
};
Which DelayedRunnable.getDelay() can then check:
#Override
public long getDelay(TimeUnit unit) {
long millis;
if (!priority.isReady(executor))
millis = 1000;
else
millis = time - System.currentTimeMillis();
return unit.convert(millis, TimeUnit.MILLISECONDS);
}
- so long as it doesn't return <= 0 if the priority isn't ready yet.
This seemed to work well, e.g. launching a standard 2s sleep task here...
DelayedScheduler scheduler = new DelayedScheduler();
scheduler.schedule(task("Low 1"), 1, TimeUnit.SECONDS, Priority.LOW);
scheduler.schedule(task("Low 2"), 2, TimeUnit.SECONDS, Priority.LOW);
scheduler.schedule(task("Low 3"), 3, TimeUnit.SECONDS, Priority.LOW);
scheduler.schedule(task("Medium 1"), 1, TimeUnit.SECONDS, Priority.MEDIUM);
scheduler.schedule(task("Medium 2"), 2, TimeUnit.SECONDS, Priority.MEDIUM);
scheduler.schedule(task("Medium 3"), 3, TimeUnit.SECONDS, Priority.MEDIUM);
scheduler.schedule(task("High 1"), 1, TimeUnit.SECONDS, Priority.HIGH);
scheduler.schedule(task("High 2"), 2, TimeUnit.SECONDS, Priority.HIGH);
scheduler.schedule(task("High 3"), 3, TimeUnit.SECONDS, Priority.HIGH);
... produced about the right results:
High 1 started at 1087ms
Medium 1 started at 1087ms
High 2 started at 2087ms
Medium 1 ended at 3087ms
High 1 ended at 3087ms
High 3 started at 3087ms
High 2 ended at 4088ms
Medium 2 started at 4088ms
High 3 ended at 5088ms
Medium 3 started at 5088ms
Medium 2 ended at 6088ms
Medium 3 ended at 7089ms
Low 1 started at 7089ms
Low 1 ended at 9089ms
Low 2 started at 9089ms
Low 2 ended at 11089ms
Low 3 started at 11089ms
Low 3 ended at 13089ms
- Medium priority tasks were allowed while there was only one High priority task running, Low while there was nothing else going.
(DelayedScheduler and the other unseen bits on GitHub).

I think your pretty close to what you want, maybe a little encouragement/approval/aggreement is all that's needed
My thoughts would be "If I know the max number of concurrent threads I can run then how will I share those against 3 thread queues".
Once I know this I can setup 3 queues, each with a different share of the available threads.
- Priority 1 (Highest) gets 50% of work
- Priority 2 gets 35% of work
- Priority 3 (Lowest) gets 15% of work

When does concurrency/multithreading help improve performance?

I have been planning to use concurrency in project after learning it indeed has increased through put for many.
Now I have not worked much on multi threading or concurrency so decided to learn and have a simple proof of concept before using it in actual project.
Below are the two examples I have tried:
1. With use of concurrency
public static void main(String[] args)
{
System.out.println("start main ");
ExecutorService es = Executors.newFixedThreadPool(3);
long startTime = new Date().getTime();
Collection<SomeComputation> collection = new ArrayList<SomeComputation>();
for(int i=0; i< 10000; i++){
collection.add(new SomeComputation("SomeComputation"+i));
}
try {
List<Future< Boolean >> list = es.invokeAll(collection);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("\n end main "+(new Date().getTime() - startTime));
}
2. Without use of concurrency
public static void main(String[] args) {
System.out.println("start main ");
long startTime = new Date().getTime();
Collection<SomeComputation> collection = new ArrayList<SomeComputation>();
for(int i=0; i< 10000; i++){
collection.add(new SomeComputation("SomeComputation"+i));
}
for(SomeComputation sc:collection)
{
sc.compute();
}
System.out.println("\n end main "+(new Date().getTime() - startTime));
}
Both share a common class
class SomeComputation implements Callable<Boolean>
{
String name;
SomeComputation(String name){this.name=name;}
public Boolean compute()
{
someDumbStuff();
return true;
}
public Boolean call()
{
someDumbStuff();
return true;
}
private void someDumbStuff()
{
for (int i = 0;i<50000;i++)
{
Integer.compare(i,i+1);
}
System.out.print("\n done with "+this.name);
}
}
Now the analysis after 20 odd runs of each approach.
1st one with concurrency takes on average 451 msecs.
2nd one without concurrency takes on average 290 msecs.
Now I learned this depends on configuration , OS , version(java 7) and processor.
But all was same for both approaches.
Also learned the cost of concurrency is affordable on when computation is heavy.But this point wasn't clear to me.
Hope some one can help me understand this more.
PS: I tried finding similar questions but could find this kind.Please comment the link if you do.

Concurrency has at least two different purposes: 1) performance, and 2) simplicity of code (like 1000 listeners for web requests).
If your purpose is performance, you can't get any more speedup than the number of hardware cores you put to work.
(And that's only if the threads are CPU-bound.)
What's more, each thread has a significant startup overhead.
So if you launch 1000 threads on a 4-core machine, you can't possibly do any better than a 4x speedup, but against that, you have 1000 thread startup costs.

As mentioned in one of answers, one use of concurrency is to have simplicity of code i.e. there are certain problems which are logically concurrent so there is no way to model those problems in a non-concurrent way like producer - consumer problems, listeners to web requests etc
Other than that, a concurrent program adds to performance only if its able to save CPU Cycles for you i.e. goal is to keep CPU or CPUs busy all the time and not waste its cycles, which further means that you let your CPU do something useful when your program is supposed to be busy in something NON - CPU tasks like waiting for Disk I/O, Wait for Locks , Sleeping, GUI app user wait etc - these times simply add time to your total program run time.
So the question is, what your CPU doing when your program not using it? Can I complete a portion of my program during that time and segregate waiting part in another thread? Now a days, most of modern systems are multiprocessor and multi core systems further leading to wastage if programs are not concurrent.
The example that you wrote is doing all the processing in memory without going into any of wait states so you don't see much gain but only loss in setting up of threads and context switching.
Try to measure performance by hitting a DB, get 1 million records, process those records and then save those records to DB again. Do it in one go sequentially and in small batches in concurrent way so you notice performance difference because DB operations are disk intensive and when you are reading or writing to DB, you are actually doing Disk I/O and CPU is free wasting its cycles during that time.
In my opinion, good candidates for concurrency are long running tasks involving one of wait operations mentioned above otherwise you don't see much gain.Programs which need some background tasks are also good candidates for concurrency.
Concurrency has not to be confused with multitasking of CPU i.e. when you run different programs on same CPU at the same time.
Hope it helps !!

concurrecy is needed when the threads are sharing the same data sources
so when some thread is working with this source the others must wait until it
finish the job than they have the acces
so you need to learn synchronized methode and bluck or some thing like that
sorry for my english read this turial it's helpful
https://docs.oracle.com/javase/tutorial/essential/concurrency/syncmeth.html

Java ScheduledExecutorService behind the scenes

How does things like scheduleAtFixedRate work? How does it work behind the scenes and is there a penalty to using it?
More specifically, I have a task that I want to run periodically, say every 12 hours. The period is not strict at all, so my first instinct was to check in every request (tomcat server) if it's been more than >12 hours since the task last executed and if so, execute it and reset the timer. The downside of this is that I have to do a small time check on every request, make sure the task is run only once (using a semaphore or something similar) and the task might not execute in a long time if there's no requests.
scheduleAtFixedRate makes it easier to schedule a recurring task, but since I don't know how it does it, I don't know what the performance impact is. Is there a thread continually checking if the task is due to run? etc.
edit:
In Timer.java, there's a mainLoop function which, in my understanding, is something like this (overly simplified):
while(true) {
currentTime = System.currentTimeMillis();
if(myTask.nextExecutionTime == currentTime) myTask.run();
}
Won't this loop try to run as fast as possible and use a ton of CPU (I know, obviously not, but why)? There's no Thread.sleep in there to slow things down.

You can read the code if you wish to work out how it works.
There is an overhead using ScheduledExecutorService in terms of CPU and memory, however on the scale of hours, minutes, second even milli-seconds, it probably not work worrying about. If you have a task running in the range of micro-seconds, I would consider something more light weight.
In short, the overhead is probably too small for you to notice. The benefit it gives you is ease of use, and it is likely to be worth it.

Constantly nulling out and re-creating Timer, TimerTask in Java

I know using Timer and TimerTask is no longer the current accepted practice (some have suggested using threads, others suggest using ScheduledExecutorService or its variants), so this question is not so much about good programming practice, but about the possibility of actual errors or exception.
Essentially, what I have is a servlet that keeps a running counter (which is a static Calendar object) that gets incremented every second. When a specified deadline is met (when we reach 10 minutes by default), I make a call from my application to a NIST time server to get the current time, which I then use to re-set my counter.
The same TimerTask-defined method that increments the counter (every second) is also the one that must be paused and re-scheduled every time I make a call to a NIST server (every ten minutes). I have been unsuccessful in pausing/cancelling the existing Timer/TimerTask objects before the NIST server call and re-scheduling the TimerTask after the call.
The exceptions that occur from this are described here:
How do I use a timer in Java when required to set and cancel multiple times?
Suffice it to say, neither TimerTask nor Timer can be scheduled more than once, even by using purge() or cancel(), which appear to be only good for setting those objects as eligible for Java garbage collection.
Using wait() and notify() resulted in synchronization exceptions that I, unfortunately, did not have the time to figure out, so my initial experiment with threading was a failure.
What I ended up doing is this:
secondTickerTask.cancel();
secondTicker.purge();
secondTicker.cancel();
secondTickerTask = null;
secondTicker = null;
Date newCurrentTime = getNistTimeFromFirstWorkingServer();
// Save new date to current time, reset second counter.
setCurrentTimeAndDeadline(newCurrentTime);
startSecondIncrementTimer(newCurrentTime);
secondTicker = new Timer();
secondTickerTask = new TimerTask(){
public void run(){
incrementCurrentTimeAndTestDeadline();
}
I ran this code over-night a few times, at 10-minute and 1-minute intervals between NIST server calls, and it worked smoothly.
So, after that long lead-up (thank you for your patience), this is what my question is: Being forced, for the moment, to use the code that I have, is there any damage that could result in the long run? If I keep making new TimerTask and Timer objects while nulling out the old ones over, let's say, a period of a month, or six months, will I force the Server to run out of memory? Is Java's garbage collection robust enough to handle this sort of use? Can any other scary thing happen?
Thank you very much for your time,
- Eli

Java will handle the creation and abandonment of the timer tasks just fine. You need to ensure that you drop all references to the timers when you are done with them, which it appears you are doing, and then when the GC runs it will clean up any garbage the Timers introduced.
You are safe.
You should note that, over long periods of time, some Java processes tend to keep allocating memory until they hit their -Xmx limit. This does not mean that there is a problem (because that space will be reused by the GC), but it also means that, if you want a long-running Java process to have a relatively small footprint that you should not specify an -Xmx much larger than what you actually need.

Does it make sense to reuse Runnables in a thread pool?

I'm implementing a thread pool for processing a high volume market data feed and have a question about the strategy of reusing my worker instances that implement runnable which are submitted to the thread pool for execution. In my case I only have one type of worker that takes a String and parses it to create a Quote object which is then set on the correct Security. Given the amount of data coming off the feed it is possible to have upwards of 1,000 quotes to process per second and I see two ways to create the workers that get submitted to the thread pool.
First option is simply creating a new instance of a Worker every time a line is retrieved from the underlying socket and then adding it to the thread pool which will eventually be garbage collected after its run method executed. But then this got me thinking about performance, does it really make sense to instantiate 1,0000 new instances of the Worker class every second. In the same spirit as a thread pool do people know if it is a common pattern to have a runnable pool or queue as well so I can recycle my workers to avoid object creation and garbage collection. The way I see this being implemented is before returning in the run() method the Worker adds itself back to a queue of available workers which is then drawn from when processing new feed lines instead of creating new instances of Worker.
From a performance perspective, do I gain anything by going with the second approach or does the first make more sense? Has anyone implemented this type of pattern before?
Thanks - Duncan

I use a library I wrote called Java Chronicle for this. It is designed to persist and queue one million quotes per second without producing any significant garbage.
I have a demo here where it sends quote like objects with nano second timing information at a rate of one million messages per second and it can send tens of millions in a JVM with a 32 MB heap without triggering even a minor collection. The round trip latency is less than 0.6 micro-seconds 90% of the time on my ultra book. ;)
from a performance perspective, do I gain anything by going with the second approach or does the first make more sense?
I strongly recommend not filling your CPU caches with garbage. In fact I avoid any constructs which create any significant garbage. You can build a system which creates less than one object per event end to end. I have a Eden size which is larger than the amount of garbage I produce in a day so no GCs minor or full to worry about.
Has anyone implemented this type of pattern before?
I wrote a profitable low latency trading system in Java five years ago. At the time it was fast enough at 60 micro-seconds tick to trade in Java, but you can do better than that these days.
If you want low latency market data processing system, this is the way I do it. You might find this presentation I gave at JavaOne interesting as well.
http://www.slideshare.net/PeterLawrey/writing-and-testing-high-frequency-trading-engines-in-java
EDIT I have added this parsing example
ByteBuffer wrap = ByteBuffer.allocate(1024);
ByteBufferBytes bufferBytes = new ByteBufferBytes(wrap);
byte[] bytes = "BAC,12.32,12.54,12.56,232443".getBytes();
int runs = 10000000;
long start = System.nanoTime();
for (int i = 0; i < runs; i++) {
bufferBytes.reset();
// read the next message.
bufferBytes.write(bytes);
bufferBytes.position(0);
// decode message
String word = bufferBytes.parseUTF(StopCharTesters.COMMA_STOP);
double low = bufferBytes.parseDouble();
double curr = bufferBytes.parseDouble();
double high = bufferBytes.parseDouble();
long sequence = bufferBytes.parseLong();
if (i == 0) {
assertEquals("BAC", word);
assertEquals(12.32, low, 0.0);
assertEquals(12.54, curr, 0.0);
assertEquals(12.56, high, 0.0);
assertEquals(232443, sequence);
}
}
long time = System.nanoTime() - start;
System.out.println("Average time was " + time / runs + " nano-seconds");
when set with -verbose:gc -Xmx32m it prints
Average time was 226 nano-seconds
Note: there are no GCes triggered.

I'd use the Executor from the concurrency package. I believe it handles all this for you.

does it really make sense to instantiate 1,0000 new instances of the Worker class every second.
Not necessarily however you are going to have to be putting the Runnables into some sort of BlockingQueue to be able to be reused and the cost of the queue concurrency may outweigh the GC overhead. Using a profiler or watching the GC numbers via Jconsole will tell you if it is spending a lot of time in GC and this needs to be addressed.
If this does turn out to be a problem, a different approach would be to just put your String into your own BlockingQueue and submit the Worker objects to the thread-pool only once. Each of the Worker instances would dequeue from the queue of Strings and would never quit. Something like:
public void run() {
while (!shutdown) {
String value = myQueue.take();
...
}
}
So you would not need to create your 1000s of Workers per second.

Yes of course, something like this, because OS and JVM don't care about what is going on a thread, so generally this is a good practice to reuse a recyclable object.

I see two questions in your problem. One is about thread pooling, and another is about object pooling. For your thread pooling issue, Java has provided an ExecutorService . Below is an example of using an ExecutorService.
Runnable r = new Runnable() {
public void run() {
//Do some work
}
};
// Thread pool of size 2
ExecutorService executor = Executors.newFixedThreadPool(2);
// Add the runnables to the executor service
executor.execute(r);
The ExecutorService provides many different types of thread pools with different behaviors.
As far as object pooling is concerned, (Does it make sense to create 1000 of your objects per second, then leave them for garbage collection, this all is dependent on the statefulness and expense of your object. If your worried about the state of your worker threads being compromised, you can look at using the flyweight pattern to encapsulate your state outside of the worker. Additionally, if you were to follow the flyweight pattern, you can also look at how useful Future and Callable objects would be in your application architecture.