I want a mechanism that will start a java program ( quite a big one ) depending on 2 conditions:
N new inserts in a MySQL table
Every 5 minutes interval.
I know that I can do this through crontab or using Timer or using Stored Procedure etc.
My plan is to write a Java class ( I am most familiar with ), Listener having two threads in parallel - Database Listener and Time listener threads each of them monitoring one of these conditions. If one says, yes, the parent class will start a new thread to run the Program.
I feel that it will be a heavy weight program. Is there some other option that I am overlooking at?
Write a single job. Have it execute regularly.
Effectively, you'll be doing some something of the nature of:
SELECT count(*) FROM table WHERE new = 1;
(or whatever)
Run that every second, 5 seconds, 10 seconds, whatever seems reasonable based on your activity.
When count == N, run your process. When "time since last run" == 5 minutes, run your process.
The process is the same, you just check it more often with the two criteria.
This offers an advantage that you won't get rogue race condition where the job fires TWICE (because Job A found the insert count that just-so-happens to have been 5 minutes from when the last job ran). Rare, yes, but race conditions always seem to actively seek "rare" events that "never happen".
As for scheduling, a crontab is easy because you don't have to maintain your process, keep it alive, daemonize, etc. etc.
If you're already running in a long running container (app server, tomcat, etc.) then that problem is already solved and you can just leverage that.
Downside of cron is it's granularity, it only runs at most every minute. If that too long, it won't work for you. But if it's ok, then there's real value in having a simple process that just lights up, does it's check, and quits. Of course, it will have to persist it's state somehow (it could look in a job log to see when the last job ran, for example).
Within java, there are lots of options: raw threads, sleeping, Timers, ScheduledExecutorService, something like Quartz, EJB Timer beans (if you're running a Java EE container).
But, I'm a KISS fan. If a cron job can do it, let it, and do it once.
It is actually not that big using a ScheduledExecutorService:
private static final Runnable PROGRAM_RUNNABLE = new Runnable() {
#Override
public void run() {
// run the program
}
}
private ScheduledExecutorService ses = Executors.newScheduledThreadPool(2);
public static void main(String[] args) {
// database based
ses.scheduleAtFixedRate(new Runnable() {
#Override
public void run() {
boolean inserted = checkDatabase(); // check the insert in the db
if(inserted) {
PROGRAM_RUNNABLE.run();
}
}
}, 0, 1, TimeUnit.MINUTES);
// time based
ses.scheduleAtFixedRate(PROGRAM_RUNNABLE, 5, 5, TimeUnit.MINUTES);
}
Related
I have a method that I would like to run a cron job, or something similar enough to a cron job (but not a Thread, TaskTimer, or something similar to those that will take up more resources in my case) for only a small part of that method.
#Override
public void notifyUsers(Data data) {
// do some stuff with data
// want a cron job here
doImportantStuff(data);
}
Normally, I could just set up my .xml file to have the entire method to independently run the task and figure everything out, however, this method is dependent on some other data is is being processed by another external service, so I am unfortunately dependent on that service to provide me with the information I need, and I can't fetch that data in my current service due to our way of working and organization of data and whatnot.
Additionally, the work that will be done in doImportantStuff() will basically iterate of a list of objects, each with a list of their own times to be executed (only once, however). So for example, data.list[0] will be executed in 30 and 45 minutes (once for each time, for the entire life span, so 2 executions total) where as data.list[1] will be executed in 15, 30, and 60 minutes (3 executions total). These will and be be variable times too for a variable amount of data. (To me it would make more sense to use a thread, but like I said, I can't).
I would like to know is there a way for me to execute a cron, or something similar or 'small' enough as a cron to run in the background, for a block of code inside a method?
I originally was going to create a new Thread and have that thread doImportantStuff() and then sleep for X amount of time, but then after further investigation I realized that it would take up too many resources since, in theory, there could be hundreds of other calls to this method, and then each of them creating their own thread.
I am not an expert on this particular matter, obviously, so apologies if my understanding on the topic is wrong.
I suppose you can use Spring Schedule in java as well. Because in Spring has this kind of feature. It is easy and useful
#Scheduled(cron = "0 15 10 15 * ?")
public void scheduledMethod(){
//code..
}
You can find some more info here
I had one quartz job doing some stuff and taking lot of time for one particular case and others are very quick. So I don't want to stop my job waiting when one slow case is running for very long time and other things are complete. This job runs and checks doStuff flag. when doStuff==1 run do things and make value to 0. So that when next time job is triggered it should not run already done stuff.
So to solve above problem I created 2 jobs. A- quick performing job, B-slow moving job.
I have 2 quartz jobs A & B running concurrently and using one flag to decide whether to run job & do stuff or to run and exit without doing any stuff.
I have a flag doStuff and it has initial value as 0 . when it's value is 1 then both job should start doing stuff and when they finish they should update value back to 0. When value is 0 and job started then this job won't do it'a stuff.
The work both job A & B doing is exactly same but different parameters.
Ideally both job should run when flag is 1 and do stuff and update value to 0.
But when first job runs and completes before starting second job, it is updating value to 0 and when second runs it reads value 0 and won't do any stuff.
The doStuff flag is DB column. I can add one more column for job B, but that's not a good design as we may get requirement of job C, job D and I can't add column for each requirement.
Is there any better way to handle this problem?
Note: I am new to Quartz framework.
Tried using new column for flag but not a good design. Tried locking but that also won't solve problem .
Below is how execute method of InterruptableJob looks like
public void execute(JobExecutionContext jobContext) throws JobExecutionException {
JobDataMap jobDataMap = jobContext.getMergedJobDataMap();
final long customerId = jobDataMap.getLong(CUSTOMER_ID_KEY);
List < QAgentAssetSourceInfo > customerAgents = getValuesBasedOnDoStuff();
// below code is in Runnable run method and will be started in new thread
{
failedAgents.putAll(agentMgmtHelper.get().doStuff(customerAgents, customer.getId(), null, batchSize));
// we are done with stuff now make value to 0
unsetDoStuff();
});
}
I'm trying to create a method that executes a given task in a maximum amount of time. If it fails to finish in that time, it should be retried a number of times before giving up. It should also wait a number of seconds between each try. Here's what I've come up with and I'd like some critiques on my approach. Is their a simpler way to do this using the ScheduledExecutorService or is my way of doing this suffice?
public static <T> T execute(Callable<T> task, int tries, int waitTimeSeconds, int timeout)
throws InterruptedException, TimeoutException, Exception {
Exception lastThrown = null;
for (int i = 0; i < tries; i++) {
try {
final Future<T> future = new FutureTask<T>(task);
return future.get(timeout, TimeUnit.SECONDS);
} catch (TimeoutException ex) {
lastThrown = ex;
} catch (ExecutionException ex) {
lastThrown = (Exception) ex.getCause();
}
Thread.sleep(TimeUnit.SECONDS.toMillis(waitTimeSeconds));
}
if (lastThrown == null) {
lastThrown = new TimeoutException("Reached max tries without being caused by some exception. " + task.getClass());
}
throw lastThrown;
}
I think, but it's my opinion, that if you are scheduling network related tasks, you should not retry but eventually run them in parallel. I describe this other approach later.
Regarding your code, you should pass the task to an executor, or the FutureTask to a thread. It will not spawn a thread or execute by itself. If you have an executor (see ExecutorService), you don't even need a FutureTask, you can simply schedule it and obtain a callable.
So, given that you have an ExecutorService, you can call :
Future<T> future = yourExecutor.submit(task);
Future.get(timeout) will wait for that timeout and eventually return with TimeoutException even if the task has never started at all, for example if the Executor is already busy doing other work and cannot find a free thread. So, you could end up trying 5 times and waiting for seconds without ever giving the task a chance to run. This may or may not be what you expect, but usually it is not. Maybe you should wait for it to start before giving it a timeout.
Also, you should explicitly cancel the Future even if it throws TimeoutException, otherwise it may keep running, since nor documentation nor code says it will stop when a get with timeout fails.
Even if you cancel it, unless the Callable has been "properly written", it could keep running for some time. Nothing you can do it about it in this part of code, just keep in mind that no thread can "really stop" what another thread is doing in Java, and for good reasons.
However I suppose your tasks will mostly be network related, so it should react correctly to a thread interruption.
I usually use a different strategy is situations like this:
I would write public static T execute(Callable task, int maxTries, int timeout), so the task, max number of tries (potentially 1), max total timeout ("I want an answer in max 10 seconds, no matter how many times you try, 10 seconds or nothing")
I start spawning the task, giving it to an executor, and then call future.get(timeout/tries)
If I receive a result, return it. If I receive an exception, will try again (see later)
If however i get a timeout, I DON'T cancel the future, instead I save it in a list.
I check if too much time has passed, or too many retries. In that case I cancel all the futures in the list and throw exception, return null, whatever
Otherwise, I cycle, schedule the task again (in parallel with the first one).
See point 2
If I have not received a result, I check the future(s) in the list, maybe one of the previous spawned task managed to do it.
Assuming your tasks can be executed more than once (as I suppose they are, otherwise no way to retry), for network stuff I found this solution to work better.
Suppose your network is actually very busy, you ask for a network connection, giving 20 retries 2 seconds each. Since your network is busy, none of the 20 retries manages to get the connection in 2 seconds. However, a single execution lasting 40 seconds may manage to connect and receive data. It's like a person pressing f5 compulsively on a page when the net is slow, it will not do any good, since every time the browser has to start from the beginning.
Instead, I keep the various futures running, the first one that manages to get the data will return a result and the others will be stopped. If the first one hangs, the second one will work, or the third one maybe.
Comparing with a browser, is like opening another tab and retrying to load the page there without closing the first one. If the net is slow, the second one will take some time, but not stopping the first one, which will eventually load properly. If instead the first tab was hung, the second one will load rapidly. Whichever loads first, we can close the other tab.
The thread on which your execute is called will block for so much time. Not sure if this is correct for you. Basically , for these types of tasks , ScheduledExecutorService is best.You can schedule a task and specify the timings. Take a look at ScheduledThreadPoolExecutor
I was adding a loading bar feature to my program in Java and I was wondering if I could set a list of processes that need to be done, so that it knows how many processes it needs to complete.
For example, saving them as strings?
ArrayList<String> processes = new ArrayList<String>();
processes.add("CSVWriter.createFileOfCompany(\"Random Company\");");
processes.add("FileConverter.convertCSVToXLS(classPath +
\"/Output/Random_Company.csv\");");
for (int i = 0; i < processes.size(); i++) {
// run process
// update loading bar
}
These aren't real methods to my program, but they are pretty similar to what I want to accomplish.
I'm sure this isn't the best way, but I want to have some way to create a list like this so I know how many processes there are before I run them. I have a system set up to break down these processes even further and show their progress, so this bar is pretty precise at the moment, but I have to number each of the processes =/.
Maybe I'm just missing the point. Creating progress bars is totally new to me.
If there are any good articles on progress bar creation, feel free to send them my way as well. Keep in mind that I'm not using an actual swing-based GUI. This bar is all S.O.P text.
Many thanks,
Justian Meyer
Closures will hopefully be coming soon in the next version of Java, but until then you can use anonymous classes implementing a known interface:
List<Runnable> jobs = new ArrayList<Runnable>();
jobs.add(new Runnable() {
public void run() {
CSVWriter.createFileOfCompany("Random Company");
}
});
jobs.add(new Runnable() {
public void run() {
FileConverter.convertCSVToXLS(classPath + "/Output/Random_Company.csv");
}
});
for (Runnable job : jobs) {
job.run();
}
Here's a scheme that just came to my mind:
interface WorkProcess
void setWorkUnits(int units)
void setArguments(Object obj1,...)
void execute()
So you encapsulate all your tasks with an interface that does execute as per the classic command pattern; it's also told (see below) how much work that job will probably take. You'll probably want some mechanism to pass data into these tasks for them to work with.
class WorkProcessFactory
static WorkProcess makeWorkProcess()
static int getTotalWorkUnitsAllocated()
static synchronized int reportWorkDone(int units)
static void addProgressListener(ProgressListener xxx)
When you have a job to do, you ask the factory to churn out one of these processes for you. In doing so, it does the estimate on work units and passes that to the WorkProcess it just created. It also keeps a tally of the total of those units for setting up the top of the progress indicator. One cool side effect of this is that you don't have to wait for all your jobs to be prepared: You can increment the maximum value for your progress report every time you add a job even as jobs are being processed and newly created; your bar will always show a realistic ratio.
You can stuff your WorkProcesses into a queue of some sort. ExecutorService comes to mind if your WorkProcess also implements the Runnable interface (which is a good idea). You'll be at liberty to process those jobs sequentially single-file or concurrently - whatever your execution queue machinery supports.
The WorkProcess' execute() method basically wraps the run() method (or maybe the other way around, since ExecutorService expects a run() method but our work unit magic is in execute()); but when that's done, it conscientiously calls back to reportWorkDone with the number of units it was told its job was worth. Upon receiving these reports, the factory has an updated value for units of work done, which it can report back to the snooping ProgressListener. That, in turn, updates the bar in the GUI.
Done. (I think). Probably needs some refining.
I have a web application that synchronizes with a central database four times per hour. The process usually takes 2 minutes. I would like to run this process as a thread at X:55, X:10, X:25, and X:40 so that the users knows that at X:00, X:15, X:30, and X:45 they have a clean copy of the database. It is just about managing expectations. I have gone through the executor in java.util.concurrent but the scheduling is done with the scheduleAtFixedRate which I believe provides no guarantee about when this is actually run in terms of the hours. I could use a first delay to launch the Runnable so that the first one is close to the launch time and schedule for every 15 minutes but it seems that this would probably diverge in time. Is there an easier way to schedule the thread to run 5 minutes before every quarter hour?
You can let the Runnable schedule its "next run".
Such as,
class Task implements Runnable {
private final ScheduledExecutorService service;
public Task(ScheduledExecutorService service){
this.service = service;
}
public void run(){
try{
//do stuff
}finally{
//Prevent this task from stalling due to RuntimeExceptions.
long untilNextInvocation = //calculate how many ms to next launch
service.schedule(new Task(service),untilNextInvocation,TimeUnit.MILLISECONDS);
}
}
}
Quartz would be a good fit since you're application is web-based. It will provide the fine-grained time based scheduling you need.
Quartz is a full-featured, open source
job scheduling service that can be
integrated with, or used along side
virtually any Java EE or Java SE
application - from the smallest
stand-alone application to the largest
e-commerce system. Quartz can be used
to create simple or complex schedules
for executing tens, hundreds, or even
tens-of-thousands of jobs; jobs whose
tasks are defined as standard Java
components that may executed virtually
anything you may program them to do.
The Quartz Scheduler includes many
enterprise-class features, such as JTA
transactions and clustering.
TimerTask handles this case.
See schedule(TimerTask, Date)
If you don't want to have to keep scheduling the jobs, you may want to look into a job scheduling tool like Quartz.