Develop Reference

registerEventTimeTimer is not firing with watermark

I'm using KafkaSource to read kafka messages of Events type, as per documentation providing event time extractor is optional for source kafka
KafkaSource<Events> source =
KafkaSource.<Events>builder()
.setProperties(kafkaProperties)
.setBootstrapServers(parameters.get("bootstrap-servers-source"))
.setTopics(parameters.get("source-topic"))
.setGroupId("visit-events-flink-mvp")
.setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST))
//.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(new EventsDeserializationSchema())
.build();
// event stream from kafka source
DataStream<Events> eventStream =
env.fromSource(source, WatermarkStrategy.forMonotonousTimestamps(), "Kafka Source")
//should be a unique id
.uid("kafka-source");
//stream is keyed based on the anonymousId
DataStream<Events> keyedStream =
eventStream.keyBy(Events::getAnonymousId)
// .process(new KeyedProcessing(Long.parseLong(parameters.get("ttl"))))
.process(new KeyedProcessingWithCallBack(Long.parseLong(parameters.get("ttl"))))
.uid("engager-events-keyed-processing");
In my KeyedProcessingWithCallBack, I'm setting event time timer for 60 secs and the call back is not triggering at all.
My kafka source has 8 partitions and I'm running job with parallelism 1
public void processElement(EngagerEvents value, KeyedProcessFunction<String, EngagerEvents, String>.Context ctx, Collector<String> out) throws Exception {
ObjectMapper objectMapper = new ObjectMapper();
JsonNode jsonNode = objectMapper.readTree(value.getEventString());
System.out.println("time : " +jsonNode.get("EVENT_TIMESTAMP").textValue());
if (anonymousIdHasBeenSeen.value() == null) {
System.out.println("time stamp emitting: " +jsonNode.get("EVENT_TIMESTAMP").textValue());
// key is not available in the state
anonymousIdHasBeenSeen.update(true);
System.out.println("TIMER START TIME: " +ctx.timestamp());
out.collect(value.getEventString());
ctx.timerService().registerEventTimeTimer(ctx.timestamp() + (stateTtl * 1000));
}
}
// not getting triggered
#Override
public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out)
throws Exception {
// triggers after ttl has passed
System.out.println("Call back triggered : time : " +timestamp + " value : " +anonymousIdHasBeenSeen.value());
anonymousIdHasBeenSeen.clear();
}
TEST Simulator code which will send event with anonymousId=111 with different event time stamp
try {
for (int i = 0; i < 500; i++) {
String[] anonymousId = {"111"};
String key = String.valueOf(new Random().nextInt(10));
ProducerRecord<String, String> record = new ProducerRecord<>(
"flink-visits-mvp-test-source",
key,
// getEvent(UUID.randomUUID().toString() + "-" +Thread.currentThread().getName() , event[new Random().nextInt(1)]));
// getEvent(anonymousId[new Random().nextInt(1)], event[new Random().nextInt(1)]));
getEvent(anonymousId[new Random().nextInt(1)],
System.currentTimeMillis(),
event));
//System.out.println(record.value().toString());
producer.send(record);
try {
Thread.sleep(200);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}
Am I doing something wrong here ? Why is my event time timer call back not trgiggering ?

I also experience the same error with Flink V1.16. The processElement method is invoked as expected however, observed that ctx.timerService().currentWatermark() always print as -9223372036854775808. onTimer method never invoked with 60 seconds timer.
With lots of trial and error I found that invoking env.setParallelism() method resolve the issue.
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
After fix, the currentWatermark() method return correct watermark time and onTimer method is invoked every 60 seconds.
Here is the log entries before fix:
Event Time = 1669610765000, End Of Window Time = 1669610819999,
current water mark time = -9223372036854775808
Event Time = 1669610807000, End Of Window Time = 1669610819999,
current water mark time = -9223372036854775808
Here is the log entries after fix:
Event Time = 1669610393000, End Of Window Time = 1669610399999,
current water mark time = 1669610387999
Event Time = 1669610450000, End Of Window Time = 1669610459999,
current water mark time = 1669610435999

Quartz How can i stop/pause/start/interrupt the Scheduler from Jsp i.e on click of particular button

Please can any one forward me the sample code related. Cause i tried a lot and on internet no useful info or links i can found related to it.
Thanks in Advance

This might be a workaround. But it works!
In your scheduler, have a default thread running every 1 minute (Or interval of your choice) that pings a file or DB for any changes.
The scheduler should be refreshed if the scheduler finds an entry in the DB.
From your JSP, on click of a button, create a relevant entry in the DB.
While pinging the DB, if the scheduler finds an entry, then it will do the necessary action.
Code snippet
// Default constructor.
public Scheduler()throws SchedulerException, Exception
{
try
{
SchedulerFactory sf = new StdSchedulerFactory();
sche = sf.getScheduler();
sche.start();
if(sche.isShutdown())
{
SendAlerts.sendMsgToGroup("Scheduler Failed To Start at "+sdtf3.format(new Date())+" hrs.",defaultMsgGroup);
logger.fatal("Scheduler Failed To Start At = " + sdtf1.format(new Date()) );
}
else
{
SendAlerts.sendMsgToGroup("Scheduler started at "+sdtf3.format(new Date())+" hrs.",SchStartAlertGroup);
logger.fatal("Scheduler Started At = " + sdtf1.format(new Date()) );
}
sysdate = new Date();
readFromDBAndConfigureSchedules();
while (true)
{
if(sche.isShutdown())
{
SendAlerts.sendMsgToGroup("Scheduler Failed To Start at "+sdtf3.format(new Date())+" hrs.",defaultMsgGroup);
logger.fatal("Scheduler Failed To Start At = " + sdtf1.format(new Date()) );
}
else
{
logger.info("Scheduler is Running. Table Last Pinged at : "+sdtf1.format(sysdate));
}
/*
-----------------
IN THE CHECK DB TABLE METHOD, HANDLE REQUESTS FOR STOP, PAUSE, RE-SCHEDULE ETC
------------------
*/
SchRunJob.checkDBTable();
// Loop will repeat every 1 hour = 60 minutes * 60 seconds = 3600 seconds
Thread.sleep (3600 * 1000);
} // End of while Start Flag is Y
} // End of try block
catch (Exception e)
{
SendAlerts.sendMsgToGroup( "Fatal Exception Caught.Scheduler Shut Down at " + sdtf1.format(new Date()),defaultMsgGroup);
logger.fatal("Fatal Exception Caught.Scheduler Shut Down at " + sdtf1.format(new Date()));
e.printStackTrace();
System.exit(0);
}
} // End of default constructor**

Timer and TimerTask not update

I have an app that could set the time for processing. The problems is when I update the time, the processing will increase. For example:
Initially the timer start at 07:00AM
Let say, I update the timer to 08:00AM then the next day onwards, the program will run again at 07:00AM and also at 08:00AM. (The 07:00AM is still in scheduler, how to remove the 07:00AM?)
How to make the scheduler to only run the 08:00AM the next day?
public void setKonfigurasi(String name, String value) {
log.info(SERVLET_NAME + "Entering setKonfigurasi");
amBean.setParam(name,value); //update the time into database
//name = 'processFileConf|kodPT|userA|20140312 08:30 AM'
// reschedule timer after configured by user
try {
String kodPT = name.substring(name.indexOf("|") + 1, name.indexOf("|",name.indexOf("|") + 1));
String configStr = value.substring(2); //get the new time
String currentStr = CommonUtil.getCurrentDate();
DateFormat dateformat = new SimpleDateFormat("dd/MM/yyyy KK:mm:ss a");
Date currentDate=new Date() ;
Date configDate = dateformat.parse(currentStr+" "+configStr);
long config = configDate.getTime();
long current = currentDate.getTime();
// today
long delay = config-current;
if (delay < 0)
// tomorrow
delay += (1000*60*60*24);
// create the timer and timer task objects
Timer timer = new Timer();
timer.scheduleAtFixedRate(new TimerTask() {
public void run() {
System.out.println("showtime for "+kodPT);
processFile("auto"+kodPT);
}
}, delay, 1000*60*60*24);
ServletContextEvent servletContextEvent = EtServletContextListener.getContext();
ServletContext servletContext = servletContextEvent.getServletContext();
servletContext.removeAttribute ("timer");
servletContext.setAttribute ("timer", timer);
} catch (Exception e) {
log.error("Exception on date format : "+e.getMessage());
}
log.info(SERVLET_NAME + "Exiting setKonfigurasi");
}

You need to call cancel() on the previous timer and create a new one. The javadoc says:
Terminates this timer, discarding any currently scheduled tasks. Does
not interfere with a currently executing task (if it exists). Once a
timer has been terminated, its execution thread terminates gracefully,
and no more tasks may be scheduled on it.

I have found out about how to do what I want. Instead of using java.util.Timer to create the timer, we should use javax.ejb.Timer since the Timer in ejb have an info to identified each timer.
Timer Service EJB

Cron trigger not firing for the specified timezone

I have list of timezones; and for each timezone, I have to start schedule job. Following is the code
cronexpr = 0 30 8,12,15,17 * * ?
if(cronexpr != null){
for(int i=0;i<tList.size();i++) {
job = new JobDetailImpl("runSMSJob"+i,"SMSJobgrp"+i,SMSJob.class);
trigger = new CronTriggerImpl("runMeJobTesting"+i, "group", "runSMSJob"+i, "SMSJobgrp"+i, cronexpr, tList.get(i));
logger.info("TIMEZONE is "+trigger.getTimeZone());
schd.start();
schd.scheduleJob(job, trigger);
}
Here tList is a list containing several timezones. On my local system its running properly but on server where the timezone is BST, even though the locale is India timezone, it's firing at the BST time.
Do any one have any idea on why its failing to take the specified timezone properly?

Quartz CronTrigger executing jobs on wrong date/time

I am using following cron expression to execute a job on every Friday at specified time of day (in sample below it's 1:13 PM).
0 13 13 ? * FRI
So expected behaviour should be if I initialize this trigger any day other then Friday then it should not start executing until next Friday. But whats happening in my case is even if I initialized this trigger today (as today is Wednesday), it starts executing jobs at the very moment.
Relevant java source:
CronTrigger cronTrigger = new CronTrigger("trigger_" + groupName, groupName, cronExpression);
cronTrigger.setStartTime(startDate); //startDate = 1-Mar-2012
cronTrigger.setEndTime(endDate); //endDate = 30-Apr-2012

Your issue is configuring the startTime. startTime is meant to be the time at at which the trigger should occur. Since the date is old this causes a misfire in the scheduler and the default behavior is for the scheduler to immediately refire.
Remove setStartTime, the default behavior is for startTime to be set to the current time and the first trigger time will be the match to the cron trigger after the start time so this Thursday.
Quick little test I through together to verify:
public class Test {
public static void main(String[] args) throws ParseException, SchedulerException {
String groupName = "group";
String cronExpression = "0 13 13 ? * THUR";
CronTrigger cronTrigger = new CronTrigger("trigger_" + groupName, groupName, cronExpression);
cronTrigger.setStartTime(new Date(0));
Scheduler scheduler = StdSchedulerFactory.getDefaultScheduler();
JobDetail detail = new JobDetail("testJob", groupName, TestJob.class);
scheduler.scheduleJob(detail, cronTrigger);
scheduler.start();
try {
Thread.sleep(50001);
} catch (Exception ignore) {
}
}
public static class TestJob implements Job {
public void execute(JobExecutionContext context) throws JobExecutionException {
System.out.println("TEST");
}
}
}
When removing the setStartTime my print message does not trigger. With it there the print message triggers.