I've got a bunch of repeating tasks to schedule. They query the database to find out what to do and then execute some action like statistics updates, sending emails, fetching files and importing them. Currently, there are maybe ten of them and this number it's expected to grow a lot. I'm not given any timing constraints, actually, it's my job to choose an algorithm so that nobody complains. :D
Currently, I'm using an ad-hoc combination of threads and periodically scheduled tasks like
for the most important task, there's an own thread falling back to a short sleep when idle (from which it can be woken up, when new important work arrives).
another important task is scheduled once per hour in its own thread
medium importance tasks are scheduled periodically to "fill the holes", so that probably only one of them runs at any moment
the least important tasks are all processed by a single dedicated thread
It seems to work well at the moment, but it's not future-proof and it doesn't feel right for these reasons:
As the queue for the least important tasks may grow a lot, such tasks may be delayed indefinitely.
Filling the holes may go wrong and there may be many tasks running at once.
The number of tasks running at any given moment should depend on the server load. (*)
(*) It's primarily a web server and serving requests is actually the highest priority. Getting a separate server wouldn't help, as the bottleneck is usually the database. Currently, it works fine, but I'm looking for a better solution as we hope that the load grows by a factor of 100 in a year or two.
My idea is to increase the priority of a job, when it was delayed too much. For example, there are statistics running hourly and delaying them by a few hours is no big deal, but it shouldn't be a whole day and it mustn't be a whole week.
I'd be happy to replace all my AbstractExecutionThreadServices and AbstractScheduledServices by something working like follows:
Start the highest priority tasks immediately, no matter what.
Start the medium priority tasks only when the total load is "small".
Start the lowest priority tasks only when the system is "mostly idle".
Increase the priorities for delayed tasks using a supplied formula.
This surely sounds pretty fuzzy and getting it more precise is a part of what I'm asking. My competing goals are
Never delay the important tasks needlessly.
Never let too many concurrently running tasks slow down the server too much.
There are no hard deadlines and there's no need to minimize the number of threads used. I don't insist on a solution doing exactly what I described, I'm not looking for a library (nor I insist on reinventing the wheel). I don't think that a cron-like scheduler is the right solution.
Working with the ExecutorService model, the classic solution to reordering executor tasks is to create a ThreadPoolExecutor with a PriorityBlockingQueue feeding it the tasks - as described here.
However needing to schedule the tasks as well puts a spin on it. ScheduledThreadPoolExecutor uses an internal custom BlockingQueue to feed the tasks in when the schedule is ready, but as I think you're well aware, it's not easily open to further customisation.
At a glance, DelayQueue looks like it fits the bill perfectly - it can prioritise the next Delayed element or task. And this handles a late decision by Delayed.getDelay() about whether it is ready to go.
The fly in the ointment with this plan is when you try to pass something like DelayQueue<DelayedRunnable> into the constructor of ThreadPoolExecutor. This will only accept a BlockingQueue<Runnable>, not BlockingQueue<? extends Runnable>.
One way out of this is to create a minimum implementation of BlockingQueue<Runnable> that delegates to a BlockingQueue. The basics are here:
public class BlockingDelayQueue extends AbstractQueue<Runnable>
implements BlockingQueue<Runnable> {
private final DelayQueue<DelayedRunnable> delayQueue;
public BlockingDelayQueue(DelayQueue<DelayedRunnable> delayQueue) {
this.delayQueue = delayQueue;
}
#Override
public boolean isEmpty() {
return delayQueue.isEmpty();
}
#Override
public Runnable poll(long timeout, TimeUnit unit)
throws InterruptedException {
DelayedRunnable delayedRunnable = delayQueue.poll(timeout, unit);
if (delayedRunnable == null)
return null;
return delayedRunnable.getCommand();
}
...
}
The experimental version of DelayedRunnable used to prove the idea there uses a simple Priority enum that checks the 'busyness' of the executor:
LOW {
boolean isReady(ThreadPoolExecutor executor) {
return executor.getActiveCount() == 0;
}
},
MEDIUM {
boolean isReady(ThreadPoolExecutor executor) {
return executor.getActiveCount() <= 1;
}
},
HIGH {
boolean isReady(ThreadPoolExecutor executor) {
return true;
}
};
Which DelayedRunnable.getDelay() can then check:
#Override
public long getDelay(TimeUnit unit) {
long millis;
if (!priority.isReady(executor))
millis = 1000;
else
millis = time - System.currentTimeMillis();
return unit.convert(millis, TimeUnit.MILLISECONDS);
}
- so long as it doesn't return <= 0 if the priority isn't ready yet.
This seemed to work well, e.g. launching a standard 2s sleep task here...
DelayedScheduler scheduler = new DelayedScheduler();
scheduler.schedule(task("Low 1"), 1, TimeUnit.SECONDS, Priority.LOW);
scheduler.schedule(task("Low 2"), 2, TimeUnit.SECONDS, Priority.LOW);
scheduler.schedule(task("Low 3"), 3, TimeUnit.SECONDS, Priority.LOW);
scheduler.schedule(task("Medium 1"), 1, TimeUnit.SECONDS, Priority.MEDIUM);
scheduler.schedule(task("Medium 2"), 2, TimeUnit.SECONDS, Priority.MEDIUM);
scheduler.schedule(task("Medium 3"), 3, TimeUnit.SECONDS, Priority.MEDIUM);
scheduler.schedule(task("High 1"), 1, TimeUnit.SECONDS, Priority.HIGH);
scheduler.schedule(task("High 2"), 2, TimeUnit.SECONDS, Priority.HIGH);
scheduler.schedule(task("High 3"), 3, TimeUnit.SECONDS, Priority.HIGH);
... produced about the right results:
High 1 started at 1087ms
Medium 1 started at 1087ms
High 2 started at 2087ms
Medium 1 ended at 3087ms
High 1 ended at 3087ms
High 3 started at 3087ms
High 2 ended at 4088ms
Medium 2 started at 4088ms
High 3 ended at 5088ms
Medium 3 started at 5088ms
Medium 2 ended at 6088ms
Medium 3 ended at 7089ms
Low 1 started at 7089ms
Low 1 ended at 9089ms
Low 2 started at 9089ms
Low 2 ended at 11089ms
Low 3 started at 11089ms
Low 3 ended at 13089ms
- Medium priority tasks were allowed while there was only one High priority task running, Low while there was nothing else going.
(DelayedScheduler and the other unseen bits on GitHub).
I think your pretty close to what you want, maybe a little encouragement/approval/aggreement is all that's needed
My thoughts would be "If I know the max number of concurrent threads I can run then how will I share those against 3 thread queues".
Once I know this I can setup 3 queues, each with a different share of the available threads.
- Priority 1 (Highest) gets 50% of work
- Priority 2 gets 35% of work
- Priority 3 (Lowest) gets 15% of work
Related
I've encountered a problem I'm not sure how to solve.
I'm trying to parallelize a part of the code that we've done sequentially up until now. To do so I've divided the task into several smaller orthogonal tasks.
I've created an executorService and I'm running:
executorService.invokeAll(callableList, timeBudget, TimeUnit.NANOSECONDS);
Each callable is has several IO tasks within it (Like going to a database and external services) the overall time-budget is 200ms+-. The reason to use invokeAll is since I have an overall timeBudget for all of the request. Thus, I need a way to limit all the futures with a single budget.
In order to test myself I've added different metrics that report back to some logging visualisation tool that we have. I've noticed that:
The median (and 75th percentile) latency of that part of the code has faster.
The 95th+ percentiles has actually gotten worse.
After thorough investigation (Where I've benchmarked different parts of the code) I've noticed that invokeAll 99th percentile running time was actually 500ms and even more sometimes. This thing really screws up the optimization. Any ideas on what may cause this? Any other suggestions? Are there alternatives to invokeAll?
While I don't have answer to why invokeAll with timeouts sometimes takes much more time than the given budget. I have an answer to the question: How to run a list of futures simultaneously with a given budget?
ListeningExecutorService executor = <init>;
List<ListenableFuture<G>> futures = new ArrayList<>();
for (T chunk : chunks) {
futures.add(executorService.submit(() -> function(chunk, param1, param2, ...)));
}
Futures.allAsList(futures).get(budgetInNanos, TimeUnit.NANOSECONDS);
The code above has used Guava library.
The problem with this approach is that I'm not getting the status of each future, because I'm getting a timeout exception if the time is up - but at least in terms of budgeting the behaviour is as expected.
With vert.x, how many Timers (one-shot mainly) can I create without thinking about reduced performance, memory issues, etc.? Are there some limits?
The scenario of creating those Timers is not artificial, I don't want to loop and create at a time a few thousand Timers, but rather in a real world scenario over the time Timers might be created continuously, a few at a time or a few 10th in an hour, or 100 or so over a day, but it might happen, that there are a few thousand created overall and waiting to run.
And what to expect if for example a few hundred of them are executed at the same time or are running at almost the same time? How does vert.x handle such load?
Cheers
Short answer: nothing much will happen, even if you have million of periodic timers set.
You can try it out for yourself:
Vertx vertx = Vertx.vertx();
AtomicLong counter = new AtomicLong(0);
for (int i = 0; i < 1_000_000; i++) {
vertx.setPeriodic(1, (l) -> {
if (counter.incrementAndGet() % 1_000_000 == 0) {
System.out.print(".");
}
});
}
System.out.println("Done deploying");
Here you can see that I creat 1M timers, each of them incrementing a counter every millisecond. This will peak your CPU, and consume a lot of memory, but it will work.
And we're talking about actively running timers. The only resource your sleeping timers will consume is memory (since they are couple of objects sitting is a data structure).
As long as you don't run of memory (and you'll need hundreds of thousands of them for that to happen), you should be fine.
And if you're really curious how timers in Vert.x work, you can start looking from here: https://github.com/eclipse/vert.x/blob/master/src/main/java/io/vertx/core/impl/VertxImpl.java#L372
I have implemented a 5-Stage CPU instruction pipeline simulator in Java using multi-threading.
Each Stage is a thread that performs mainly below 3 functions, also there is a queue (of capacity 1) in-between every two stages.
Receive from the previous stage.
Process i.e. perform its main responsibility.
Forward to the next stage.
#Override
public void run() {
while (!(latchQueue.isEmpty())) {
fetch();
process();
forward();
}
}
Simulation works fine. This is where I’m stuck, I want to be able to simulate only a specified number of clock cycles. so, the simulator should stop/pause once it has reached the specified number of cycles.
As of now, I have started all the 5 threads and let it simulate the processing of all the instructions rather than limiting it by clock cycles.
How can I accomplish this? do I need to pause thread when specified clock cycles have reached? If so how can I gracefully handle suspending/stopping the threads? Please help me in choosing the best possible approach.
Thanks in advance :)
You are already using some concurrent queue to communicate between the threads (exactly how it works isn't clear because your code example is quite incomplete).
So you can count cycles at the first stage, and use that same mechanism to communicate: shove a sentinel object, which represents "time to stop/pause this thread", onto the queue for the first stage, and when processed it pauses the processor (and still forwards it to the next stage, so all stages will progressively shut down). For example, you could extend the type of objects passed in your queue so that the hierarchy contains both real payload objects (e.g., decoded instructions, etc) or "command objects" like this stop/pause sentinel.
Another asynchronous solution would be to Thread.interrupt each thread and add an interrupt check in your processing loop - that's mostly to gracefully shut down, and not so much to support a "pause" functionality.
Will following work?
Share following class CyclesCounter between all your threads representing stages. It has tryReserve method, getting true from it means thread has got enough "clock cycles" for its' next run. Getting false means there's not enough cycles left. Class is thread-safe.
After getting false, perhaps, your thread should just stop then (i.e., by returning from run()) -- no way it can get enough nr of cycles (due to your requirements, as I understood them), until whole session is run again.
class CyclesManager {
private final AtomicInteger cycles;
CyclesManager(int initialTotalCycles) {
if (initialTotalCycles < 0)
throw new IllegalArgumentException("Negative initial cycles: " + initialTotalCycles);
cycles = new AtomicInteger(initialTotalCycles);
}
/**
* Tries to reserve given nr of cycles from available total nr of cycles. Total nr is decreased accordingly.
* Method is thread-safe: total nr of is consistent if called from several threads concurrently.
*
* #param cyclesToReserve how many cycles we want
* #return {#code true} if cycles are ours, {#code false} if not -- there's not enough left
*/
boolean tryReserve(int cyclesToReserve) {
int currentCycles = cycles.get();
if (currentCycles < cyclesToReserve)
return false;
return cycles.compareAndSet(currentCycles, currentCycles - cyclesToReserve);
}
}
I have a RESTful-styled RPC (remote procedure call) API running on a tomcat server that processes data of N users with M tasks on K threads. Mostly one user has around 20 to 500 tasks (but M could be between 1 to 5000). One task needs around 10 to 20 seconds to complete, but can be between 1 second and 20 minutes. Currently, mostly the system has one user, sometimes up to three, but it increases to around 10 users at the same time in the near future. Our server has 10 cores, therefore I'd like to use 10 threads. At the moment every user gets 5 threads for processing, which works fine. But a) most of the time the machine is only utilized 50% (which results in needles waiting in the "30-minute" range), sometimes the server load is up to 150%.
Requirements to solution:
at all times the server is utilized to 100% (if there are tasks)
that all users are treated the same regarding thread execution (same amount of threads finished as every other user)
a new user does not have to wait until all tasks of a earlier user are done (especially in the case where user1 has 5000 tasks and user2 has 1 this is important)
Solutions that come to mind:
just use a FixedThreadPoolExecutor with 10 threads, violates condition 3
use the PriorityBlockingQueue and implement the compareTo method in my task -> can not use the threadpoolExecutors submit method (and therefore I do not know when a submitted task is over)
implement a "round robin" like blocking queue, where the K threads (in our case 10) take new tasks from the N internal queues in a round robin way -> to be able to put a task into the right queue, I need a "submit"-method that takes more than one parameter (I need to implement a ThreadPoolExecutor, too)
I tried to make an illustration of what I mean by round robin like blocking queue (if not helpful feel free to edit it out):
-- --
-- -- -- -- queue task load,
-- -- -- -- -- -- -- one task denoted by --
-- -- -- -- -- -- -- --
| Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | QN |
| * ^ |
| last| |next |
| -------------
\ /
\ | | | | |
| T1 | T2 | T3 | T4 | TK |
Is there an elegant solution to use mostly Java standard APIs (or any other widespread Java API) for achieving this kind of processing behavior (might it be one of my proposed solutions or any another solution)? Or do you have any other hints on how to tackle this issue?
Addressing your requirements:
1) maximizing thread usage: any ThreadPoolExecutor will take care of this.
2) all users are treated the same: essentially requires a round-robin setup.
3) avoid new users waiting in FIFO order: same as #2.
Also you mentioned the ability to submit and get a result.
You might consider a standalone PriorityBlockingQueue<Job> using a wrapper object, e.g.:
class Job implements Comparable<Job> {
private int priority;
private YourCallable task;
public Job(int priority, YourCallable task) {
this.priority = priority;
this.task = task;
}
#Override
public int compareTo(Job job) {
// use whatever order you prefer, based on the priority int
}
}
Your producer offers a Job to the PriorityBlockingQueue with a priority assigned (based on your round-robin rule or whatever), and a task that implements Callable. Your consumer then does queue.poll for a Job.
Once you have that in hand, you can grab the task contained inside that Job object and send it off for processing on a ThreadPoolExecutor of your choosing.
If you agree that minimizing the overall task latency is a good replacement for requirements 2 and 3, and you have good-enough task runtime estimates, then I may have an answer.
You store the task submit time with each task, so that later you can always compute its estimated latency.
You can then build a PriorityBlockingQueue that, when inserting a new task, always inserts it at a queue position that provides some fairness and attempts to minimize overall latency. This will, at first, put long-running tasks at a disadvantage. I have not tried it myself, but I would assign a task priority based on your estimated run time, and use estimatedRuntime-waitingTime as priority (taking the lowest-priotity job first). This will give heavy tasks a chance after they waited enough to have negative priority. Until then, light tasks will have a better chance to be first, even if they have just been submitted. This scheduling will only be so fair as your estimates allow, though.
As for the round-robin requirement: If that is really important, you can handle this in the queue as well. Basically, when you use a thread pool, you can implement yur scheduling strategy in terms of where you insert new jobs in the queue. If you can estimate job latency, you can balance that across your users, too, by inserting at the right position.
I have been working on a solution similar to the round-robin setup. It gets complicated real fast but I think I came up with a decent implementation. It is probably not a very elegant solution but there are unit-tests showing some functions. Unfortunately, TaskQ is not yet at a "1.0" stage.
It does cover your points 1 to 3:
you can specify the amount of threads to use and if there are enough tasks, all threads are used.
each user will get a turn as threads become available
if one user A has 500 tasks in queue and another user B comes along with 1 task, the task from user B will get executed as soon as a thread is available.
There is no manual/documentation yet, I hope you can find the time to investigate. A unit test showing some usage is here, the main class to extend/use is RunnableTaskQWithQos.
I want a mechanism that will start a java program ( quite a big one ) depending on 2 conditions:
N new inserts in a MySQL table
Every 5 minutes interval.
I know that I can do this through crontab or using Timer or using Stored Procedure etc.
My plan is to write a Java class ( I am most familiar with ), Listener having two threads in parallel - Database Listener and Time listener threads each of them monitoring one of these conditions. If one says, yes, the parent class will start a new thread to run the Program.
I feel that it will be a heavy weight program. Is there some other option that I am overlooking at?
Write a single job. Have it execute regularly.
Effectively, you'll be doing some something of the nature of:
SELECT count(*) FROM table WHERE new = 1;
(or whatever)
Run that every second, 5 seconds, 10 seconds, whatever seems reasonable based on your activity.
When count == N, run your process. When "time since last run" == 5 minutes, run your process.
The process is the same, you just check it more often with the two criteria.
This offers an advantage that you won't get rogue race condition where the job fires TWICE (because Job A found the insert count that just-so-happens to have been 5 minutes from when the last job ran). Rare, yes, but race conditions always seem to actively seek "rare" events that "never happen".
As for scheduling, a crontab is easy because you don't have to maintain your process, keep it alive, daemonize, etc. etc.
If you're already running in a long running container (app server, tomcat, etc.) then that problem is already solved and you can just leverage that.
Downside of cron is it's granularity, it only runs at most every minute. If that too long, it won't work for you. But if it's ok, then there's real value in having a simple process that just lights up, does it's check, and quits. Of course, it will have to persist it's state somehow (it could look in a job log to see when the last job ran, for example).
Within java, there are lots of options: raw threads, sleeping, Timers, ScheduledExecutorService, something like Quartz, EJB Timer beans (if you're running a Java EE container).
But, I'm a KISS fan. If a cron job can do it, let it, and do it once.
It is actually not that big using a ScheduledExecutorService:
private static final Runnable PROGRAM_RUNNABLE = new Runnable() {
#Override
public void run() {
// run the program
}
}
private ScheduledExecutorService ses = Executors.newScheduledThreadPool(2);
public static void main(String[] args) {
// database based
ses.scheduleAtFixedRate(new Runnable() {
#Override
public void run() {
boolean inserted = checkDatabase(); // check the insert in the db
if(inserted) {
PROGRAM_RUNNABLE.run();
}
}
}, 0, 1, TimeUnit.MINUTES);
// time based
ses.scheduleAtFixedRate(PROGRAM_RUNNABLE, 5, 5, TimeUnit.MINUTES);
}