Develop Reference

Kafka Streams: how to avoid scheduler punctuator to be suppressed before completion

I have a Kafka stream application that needs to process some data into a StateStore every x minutes.
I have created a transformer receiving data and storing them into the StateStore. The Transformer is instantiated automatically by the kafka stream framework via a TransformerSupplier.
TransformerSupplier<String, Message, KeyValue<String, Message>> getTransformerSupplier(){
if(transformerSupplier == null)
transformerSupplier = () -> new MyTransformer();
return transformerSupplier;
}
The transformer then Schedule a Punctuator into the context when instantiated.
this.context.schedule(Duration.ofSeconds(schedulerSeconds), PunctuationType.WALL_CLOCK_TIME, new MyScheduler);
The scheduler overrides the punctuate function containing the code needed to handle the data into the StateStore. I have added some log lines to verify that the punctuate operation is completed before the Scheduler ends.
#Override
public void punctuate(long timestamp) {
logger.info("Scheduler started at {}", Instant.now());
processMyDataIntoStateStore();
logger.info("Scheduler ended at {}", Instant.now());
}
The problem I am seeing is that when searching for the above log lines I should expect the same number for 'started' and 'ended', but I actually see that half of the executions don't reach the 'Scheduler ended'.
It seems that the punctuator operation is stopped prematurely before completed by the context, why is this happening? is there a way to avoid stopping before completion?
Another scheduler is anyway instantiated for the next execution but I would prefer that the operation is not stopped without first complete the previous.
thanks for any help

probably moving it on to a separate thread would do the job as the punctuator is executed on the same thread of the transformer.

How can I build a cron-like job executor in Java?

I'd like to develop a Java program that executes tasks registered in a database. The tasks have their own cron-like schedule, which is an object of CronExpression of Quartz Scheduler, and saved in the database after being serialized.
Tasks should be executed anytime according to its schedule, so I think the program should be daemonized, and may be able to be restarted or stopped outside the program (like an usual service beneath /etc/init.d/)
I'm studying the examples of Quartz
and saw the program running continuously even if there's no sleep and shutdown method. This seems nice to achieve my purpose, but I'm not sure if this way can generate a daemon process.
// TODO: Retrieve cron format from the database
Trigger trigger = org.quartz.TriggerBuilder.newTrigger()
.withIdentity("trigger1", "group1")
.withSchedule(cronSchedule("* * * ? * MON-FRI"))
.startNow()
.build();
try {
sched.scheduleJob(job, trigger);
sched.start();
// Thread.sleep(90L * 1000L);
// sched.shutdown(true);
} catch (SchedulerException e) {
...
My question is
What is the best way to build a cron job scheduler which runs continuously on a server?
Thank you in advance, and any opinions or questions would be appreciated.

Quartz Scheduler pausing running job internally upon shutdown?

I am noticing that sometimes quartz scheduler is automatically pausing some of the jobs. Is this a bug or any configuration issue?
All scheduled jobs are using CronTrigger.
I am suspecting that whenever server is stopped, it maybe automatically pausing running job? I have following code for ServletContextListener for shutdown of application.
public void shutdownScheduler(Scheduler scheduler) {
try {
if (null != scheduler) {
scheduler.shutdown();
}
} catch (Exception e) {
log.error(e);
}
}
Some of the quartz properties are listed below...
org.quartz.scheduler.wrapJobExecutionInUserTransaction = false
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 20
org.quartz.threadPool.threadPriority = 5
org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread = true
org.quartz.jobStore.misfireThreshold = 60000

Yes, it does go to standBy mode by default. If you check the code you will see that one of the first thing the code does after calling shutdown is invoking the standby() function.
If you want to avoid this behaviur, you have the option to call shutdown with a parameter:
shutdown(true)
which will force the Scheduler to wait for running jobs to complete first.
Confirmation is in the doc, but not too much about the details unfortunately.

ThreadPool Executor not executing threads in GAE

I am trying to use executor framework in Google App engine. Bellow is the code that I am trying to run.
Thread thread = ThreadManager.createBackgroundThread(new Runnable(){
public void run(){
try{
LOGGER.info( "Checking background thread");
Thread.sleep(10);
}
catch (InterruptedException ex){
throw new RuntimeException("Exception:", ex);
}
}
});
ScheduledThreadPoolExecutor executor = (ScheduledThreadPoolExecutor) Executors.newScheduledThreadPool(10, ThreadManager.backgroundThreadFactory());
executor.scheduleAtFixedRate(thread, 0, 30, TimeUnit.MINUTES);
But this doesn't start the thread. But if I use thread.start() it works properly. I have checked Whitelisted Classes and it does provide Executor classes. So where I am doing it wrong ?

Saikat,
You should always try to avoid creating threads on App Engine, because of it's distributed and dynamic nature it tends to have really bad/unexpected results.
In your case multiple instances will spawn multiple (local) threads sending many times the same notification. Also, bear in mind GAE front end instances have a 1 minute request limit, so after that time the server will kill that request.
Fortunately App Engine provides the Cron service for exactly this situations.
The Cron Service will allow you to schedule a job to run at a given time or every given period. When the cron is triggered GAE will call a configured URL so you can do your process, in your case send notifications.
Eg:(from the link provided)
<cron>
<url>/weeklyreport</url>
<description>Mail out a weekly report</description>
<schedule>every monday 08:30</schedule>
<timezone>America/New_York</timezone>
</cron>
will make an HTTP request to /weeklyreport every monday #8:30.

difference between pauseJob and pauseTrigger in quartz scheduler?

What is the difference between pauseJob() and pauseTrigger() in quartz scheduler?
How can select one among them for use? now i want to pause/interept a specific job how can i do
my scheduler code is given bellow
JobDetail job = new JobDetail();
job.setName("pollerjob"+pollerId);
job.setJobClass(Pollersheduller.class);
job.getJobDataMap().put("socialMediaObj", socialMediaObj);
job.getJobDataMap().put("queue", queue);
//configure the scheduler time
SimpleTrigger trigger = new SimpleTrigger();
trigger.setName("pollerSocial"+pollerId);
trigger.setStartTime(new Date(System.currentTimeMillis() + 1000));
trigger.setRepeatCount(SimpleTrigger.REPEAT_INDEFINITELY);
trigger.setRepeatInterval(Long.parseLong(intervel));
//schedule it
Scheduler scheduler = null;
try {
scheduler = new StdSchedulerFactory().getScheduler();
scheduler.start();
scheduler.scheduleJob(job, trigger);
} catch (SchedulerException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

As you have probably noticed, in Quartz a single job can be associated with multiple triggers. And if you look into Quartz sources, you will see that the pauseJob method simply retrieves all triggers associated with the specified job and pauses them, whereas the pauseTrigger method pauses only a particular trigger. So that is the main difference.
Please note that pausing a job in Quartz does not pause a currently running running job, it merely prevents the job from being run in the future!
If you want to interrupt a running job, then you can use the interruptJob method defined in the org.quartz.Interruptable interface the job must implement. If your job implements this interface, then it is entirely up to you to implement the interrupting logic. For example, you can set some sort of a flag when the interruptJob method is called and then you need to check the value of this flag in the job's execute method.