Better way to implement synchronous repeatable task using Spring/Spring-Batch - java

I have scenario when I need to poll database for specific result. I cant go on within my code until I get the expected result(except the case of passing the timeout interval)
Step A -> Steb B -> Step C
Simple way of doing this(but doesnt feel right for me) was:
numOfRetry=0;
invokeStepA();
while(true)
{
numOfRetry++
boolen result=invokeStepB();
if(result || numOfRetry==3)
{
break;
}
else
{
Thread.sleep(100000)
}
invokeStepC();
Assume the database polling is occurring on Step B.
It doesnt feel right having this while loop on my Spring bean service while calling those jobs.
Maybe I could implement this better?
Thank you.
Farther explanation about my process:
Step A is Invoking external service to do some logic.
Step B need to poll another service which checking if Step A has finished it's work(In case it has finished I can proceed to StepC else I need to try again in X seconds and to check again)
StepC - another logic which must be accomplished only after StepB returned true.
The logic which Step A is doing happens on external service.

In the asynchronous way it happens like
int count = Runtime.getRuntime().availableProcessors();
ExecutorService threadPool = Executors.newFixedThreadPool(count);
invokeStepA();
for (int i = 0; i < RETRY_COUNT; i++) {
Future f = threadPool.submit(new Callable() {
#Override
public Object call() {
return invokeStepB();
}
}
result = (YOUR_DATA_STRUCTURE) f.get();
if (resultIsOK(result)) {
break;
}
}
However, I think since your task is ordered and assuming you cannot go to do something else, using asynchronous isn't really that effective. Please tell me more about your background in case you have special requirements.
EDIT: I think your new requirement looks like you need a proper way to tell if step A is finished fine. So you can use CountDownLatch to check if A has finished properly. I.e.
private final int count = Runtime.getRuntime().availableProcessors();
private final ExecutorService threadPool = Executors.newFixedThreadPool(count);
// invoke step A
invokeStepA();
// submit step B
final CountDownLatch latch = new CountDownLatch(1);
threadPool.submit(new Runnable() {
#Override
public void run() {
invokeStepB();
latch.countDown();
}
});
// wait for step B
boolean result;
try {
result = latch.await(TIME_OUT_IN_MILLISECONDS, TimeUnit.MILLISECOND);
} catch (InterruptedException e) {
}
// Check result
if (result) {
invokeStepC();
} else {
LOG.error("Timeout waiting for step A.");
}
This assumes your invokeStepA() is a blocking method.

Here's another idea by using an event driven approach. This is just out of my mind and not tested ;)
import org.springframework.context.ApplicationEventPublisher;
#Service
public class JobA {
#Autowired
private ApplicationEventPublisher applicationEventPublisher;
#Scheduled(cron = "0 0 * * * ?")
public void doStepA() {
log.debug("some heavy lifting");
Object someData = ....;
applicationEventPublisher.publishEvent(new JobAEvent("Jo, I'm finished", someData));
}
}
#Service
public class JobB implements ApplicationListener<JobAEvent> {
#Autowired
private ApplicationEventPublisher applicationEventPublisher;
#Override
public void onApplicationEvent(final JobAEvent event) {
log.debug("do something more based on the event data");
Object someMoreData = ....;
applicationEventPublisher.publishEvent(new JobBEvent("Dude, me too", event.getSomeData(), someMoreData));
}
}
#Service
public class JobC implements ApplicationListener<JobBEvent> {
#Autowired
private ApplicationEventPublisher applicationEventPublisher;
#Override
public void onApplicationEvent(final JobBEvent event) {
log.debug("do even more work");
}
}
EDIT:
You can also call the method directly but then it runs synchronosly. Another possibilty is using '#Async'

Related

spring boot and Executor Service

I am using spring boot
public interface StringConsume extends Consumer<String> {
default public void strHandel(String str) {
accept(str);
}
}
Impl
#Component("StrImpl")
public class StringConsumeImpl implements StringConsume {
BlockingQueue<String> queue = new ArrayBlockingQueue<>(500);
final ExecutorService exService = Executors.newSingleThreadExecutor();
Future<?> future = CompletableFuture.completedFuture(true);
#Override
public void accept(String t) {
try {
queue.put(t);
} catch (InterruptedException e) {
e.printStackTrace();
}
while (null != queue.peek()) {
if (future.isDone()) {
future = exService.submit(() -> queue.take());
}
}
}
}
Class
#Component
public class Test {
#Resource(name="StrImpl")
private #Autowired StringConsume handler;
public void insertIntoQueue(String str) {
handler.accept(str);
}
}
In StringConsumeImpl , do I need synchronized while loop? and suppose five time StringConsumeImpl class called, then do while loop will create 5 process or only 1 process ? and what is the best replacement of while loop in StringConsumeImpl , if any ?
There are some problems with that code.
First of all, the consumer doesn't really "consume" anything, it just adds the string to the queue then takes it back out. Let's say for the sake of the argument that it also "consumes" it by printing it to console or something.
Secondly, the consumer will only get called once due to the loop unless it is running in a thread of its own. Eg if you do
public static void main(String[]args) {
StringConsume consumer = new StringConsumeImpl();
consumer.accept("hello");
}
The consumer will put "hello" into the queue, take it out immediately and then stay in the loop, waiting for more elements to take out; however, no one is there to actually add any.
The usual concept of doing what it looks like you're trying to do is "producer/consumer". This means that there is a "producer" that puts items into a queue and a "consumer" taking them out and doing stuff with them.
So in your case what your class does is "consume" the string by putting it into the queue, making it a "producer", then "consuming" the string by taking it back out of the queue. Of course, there's also the "actual" producer of the string, ie the class calling this.
So in general you'd do something like this:
/** Produces random Strings */
class RandomStringProducer {
Random random = new Random();
public String produceString() {
return Double.toString(random.nextDouble());
}
}
/** Prints a String */
class PrintConsumer implements StringConsume {
public void accept(String s) { System.out.println(s); }
}
/** Consumes String by putting it into a queue */
class QueueProducer implements StringConsume {
BlockingQueue<String> queue;
public QueueProducer(BlockingQueue<String> q) { queue = q; }
public void accept(String s) {
queue.put(s);
}
}
public static void main(String[] args) {
// the producer
RandomStringProducer producer = new RandomStringProducer();
// the end consumer
StringConsume printConsumer = new PrintConsumer();
// the queue that links producer and consumer
BlockingQueue<String> queue = new ArrayBlockingQueue<>();
// the consumer putting strings into the queue
QueueProducer queuePutter = new QueueProducer(queue);
// now, let's tie them together
// one thread to produce strings and put them into the queue
ScheduledExecutorService producerService = Executors.newScheduledThreadPool(1);
Runnable createStringAndPutIntoQueue = () -> {
String created = producer.createString();
queuePutter.consume(created);
};
// put string into queue every 100ms
producerService.scheduleAtFixedRate(createStringAndPutIntoQueue, 100, TimeUnit.MILLISECONDS);
// one thread to consume strings
Runnable takeStringFromQueueAndPrint = () -> {
while(true) {
String takenFromQueue = queue.take(); // this will block until a string is available
printConsumer.consume(takenFromQueue);
}
};
// let it run in a different thread
ExecutorService consumerService = Executors.newSingleThreadExecutor();
consumerService.submit(takeStringFromQueueAndPrint);
// this will be printed; we are in the main thread and code is still being executed
System.out.println("the produce/consume has started");
}
So when you run this, there will be three threads: the main thread, the producer thread and the consumer thread. The producer and consumer will be doing their thing concurrently, and the main thread will also continue to run (as exemplified by the System.out.println in the last line).

Call function every x time without blocking the GUI Java

I have a class caled ItemGUI which is handling everything related with the user interface. The user, is able to add some links, which are the items, so when he inserts a link and clicks on the ADD button, it should create a new object of the class Item and start running a function called getPrice(), something like that:
Item newItem = new Item(newItemField.getText());
// should also be added to a list of items which should be in the ItemGUI class
newItem.getPrice()
This should be done after clicking the add button. Then I print the item to the table. The problem is that the method getPrice() should run every 5 seconds without blocking my GUI, so I should implement Threads.
My question is: how can I be able to implement a thread that runs that function (for each item in the list) every 5 seconds until I click on a stop button? I was thinking about using the observer-observable classes with a clock that notifies its observers every 5 seconds. Will this be the best option?
Also, will I be able to retrieve the item variables from the ItemGUI class?
Thanks!
Update
The clearest solution by MadProgrammer's suggestion is to use swing Timers, like this:
protected javax.swing.Timer refresherTimer = null;
protected void stopRefreshing() {
if (refresherTimer != null) {
refresherTimer.stop();
refresherTimer = null;
}
}
protected void startRefreshing() {
stopRefreshing();
refresherTimer = new Timer(500, e -> {
newItem.getPrice()
});
refresherTimer.start();
}
public void onStartButtonClicked() {
Item newItem = new Item(newItemField.getText());
// here newItem should be added to a list of items which should be in the ItemGUI class
startRefreshing();
}
public void onStopButtonClicked() {
stopRefreshing();
}
Original answer
It would be nice to have some utility named e.g. GuiTimer which would make your task as easy as:
protected GuiThread.Task refresherTask = null;
protected void cancelRefreshing() {
if (refresherTask != null) {
refresherTask.cancel();
refresherTask = null;
}
}
public void onStartButtonClicked() {
Item newItem = new Item(newItemField.getText());
// should also be added to a list of items which should be in the ItemGUI class
cancelRefreshing();
refresherTask = GuiThread.scheduleAtFixedRate(() -> {
newItem.getPrice()
}, 0, 5, TimeUnit.SECONDS);
}
public void onStopButtonClicked() {
cancelRefreshing();
}
The problem with regular timers is that they invoke the callback function on their own thread, not on the gui thread, so it requires the developer to ensure proper threading. Unfortunately the builtin java EventQueue does not support dispatching delayed tasks.
For this reason I like to have the following utility called GuiTimer, which will act as a pure gui-threaded timer:
public class GuiTimer {
public static final ScheduledThreadPoolExecutor executor =
new ScheduledThreadPoolExecutor(1);
public static interface Task {
public void cancel();
}
private static class CancelStateTask implements Task {
public volatile boolean canceled = false;
#Override
public void cancel() {
this.canceled = true;
}
}
public static Task schedule(final Runnable action) {
CancelStateTask task = new CancelStateTask();
EventQueue.invokeLater(() -> {
if (!task.canceled)
action.run();
});
return task;
}
public static Task schedule(final Runnable command, long delay,
TimeUnit unit) {
ScheduledFuture<?> future = executor.schedule(
() -> EventQueue.invokeLater(command), delay, unit);
return () -> future.cancel(false);
}
public static Task scheduleAtFixedRate(Runnable command,
long initialDelay, long period, TimeUnit unit) {
ScheduledFuture<?> future = executor.scheduleAtFixedRate(
() -> EventQueue.invokeLater(command), initialDelay,
period, unit);
return () -> future.cancel(false);
}
public static Task scheduleWithFixedDelay(Runnable command,
long initialDelay, long delay, TimeUnit unit) {
ScheduledFuture<?> future = executor.scheduleAtFixedRate(
() -> EventQueue.invokeLater(command), initialDelay, delay,
unit);
return () -> future.cancel(false);
}
public static void shutdown() {
executor.shutdown();
}
}
First off, I'm in no way sure how this should be done, but I think I have a suggestion for how you could make it work.
In a project I'm working on at the moment I use ExecutorService to handle my thread pool. I have to mention here, that I started working with threads in Java a little over a week ago myself, so I apologize if what I'm suggestion is too basic or wrong.
My suggestion is, that you could create a static boolean variable that you could use for controlling your loop. Code could look something like this:
public static boolean isRunning;
public static void main(String[] args) {
ExecutorService executerService = Executors.newCachedThreadPool();
executerService.execute(new Runnable() {
#Override
public void run() {
isRunning=true;
while (isRunning){
System.out.println("hello world"); //Your loop here
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
});
Whenever you set the isRunning boolean to false the loop will stop, and the Thread will finish after a little while.
As for observer-observable classes I don't know this and I'm not sure I understand your last question.
You can use a Timer to schedule a repeating task.
Corresponding to each Timer object is a single background thread that is used to execute all of the timer's tasks, sequentially
Something along the lines of this code should do the trick.
Timer timer = new Timer();
TimerTask task = new TimerTask(){
public void run() {
getPrice(); //your task
}
};
timer.schedule(task, 0, 5000); //first is delay, second is repeat period
...
// on button click, simple cancel the task
task.cancel()

how to "break" in parts a method that takes too much time to finish the process?

I am working in a method (using spring) that will manage a lot of data and information, consulting to the database and generate some files.
I am trying to avoid a timeout exception, so, I decided I should use the #Async annotation.
Not quite sure if it works as I think or not, but I also realized that I will need the method who calls Async to wait until it is finished...so, could be the same problem, couldn't it?
Is there any way I can have a sort of listener that will read the Async information that is being processed at my bean without have to wait for all the Async process to finish??
Right now is somehow like this
private Long myFIrstMethod(){
// DO A LOT OF THINGS AND CALL TO MY ASYNC METHOD
// evaluate if the Async method will have something or not... and based on it make the return
if (myOtherMethod()){
return soemvalue;
}else{
return someOtherValue
}
#Async Future<Boolean> myOtherMethod() {
//do something
new AsyncResult<Boolean>(true); //or false...
}
}
So, I was thinking, I might get a timeout exception on myFirstMethod is there any way to handle long time processing methods and avoiding this exception?
Thanks.
You could use a Timeout
http://sourceforge.net/p/tus/code/HEAD/tree/tjacobs/io/TimeOut.java
Set your timeout length to the length you want to wait. In the meantime, should your method return in a timely manner, you can cancel the TimeOut.
package tjacobs.io;
public class TimeOut implements Runnable {
private long mWaitTime;
private boolean mRunning = true;
private Thread mMyThread;
private TimeOutCmd mTimeOutCmd;
public static final int DEFAULT_URL_WAIT_TIME = 30 * 1000; // 30 Seconds
public static final int NO_TIMEOUT = -1;
public static final int DEFAULT_WAIT_TIME = NO_TIMEOUT;
public static interface TimeOutCmd {
public void timeOut();
}
public TimeOut(TimeOutCmd cmd) {
this(cmd, DEFAULT_WAIT_TIME);
}
public TimeOut(TimeOutCmd cmd, int timeToWait) {
mWaitTime = timeToWait;
mTimeOutCmd = cmd;
}
public void stop() {
mRunning = false;
mTimeOutCmd.timeOut();
if (mMyThread != null) mMyThread.interrupt();
}
/**
* reset the TimeOut
*
*/
public void tick() {
if (mMyThread != null)
mMyThread.interrupt();
}
public void run () {
mMyThread = Thread.currentThread();
while (true) {
try {
Thread.sleep(mWaitTime);
stop();
}
catch (InterruptedException ex) {
if (!mRunning) {
return;
}
}
}
}
}

How to restart scheduled task on runtime with EnableScheduling annotation in spring?

I have been investigating how to change the frequency of a job on runtime with Java 8 and spring. This question was very useful but it did not totally solve my issue.
I can now configure the date when to job should be executed next. But If set the delay to 1 year, then I need to wait 1 year before the new configuration in taken into account.
My idea would be to stop the scheduled task if the configuration value is changed (so from another class). Then recalculate the next time the task should be executed. Perhaps there is an easier way of doing this.
Here is the code I have so far.
#Configuration
#EnableScheduling
public class RequestSchedulerConfig implements SchedulingConfigurer {
#Autowired
SchedulerConfigService schedulerConfigService;
#Bean
public RequestScheduler myBean() {
return new RequestScheduler();
}
#Bean(destroyMethod = "shutdown")
public Executor taskExecutor() {
return Executors.newScheduledThreadPool(100);
}
#Override
public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
taskRegistrar.setScheduler(taskExecutor());
taskRegistrar.addTriggerTask(
new Runnable() {
#Override public void run() {
myBean().startReplenishmentComputation();
}
},
new Trigger() {
#Override public Date nextExecutionTime(TriggerContext triggerContext) {
Duration d = schedulerConfigService.getIntervalFromDB();
return DateTime.now().plus(d).toDate();
}
}
);
}
}
This would be what I would like to do.
#RestController
#RequestMapping("/api/config/scheduler")
public class RequestSchedulerController {
#Autowired
ApplicationConfigWrapper applicationConfigWrapper;
#RequestMapping("/set/")
#ResponseBody
public String setRequestSchedulerConfig(#RequestParam(value = "frequency", defaultValue = "") final String frequencyInSeconds){
changeValueInDb(frequencyInSeconds);
myJob.restart();
return "Yeah";
}
}
Create a singleton bean that gets an injected TaskScheduler. This will hold as state variables all ScheduledFutures, like private ScheduledFuture job1;
On deployment, load from databases all schedule data and start the jobs, filling in all state variables like job1.
On change of scheduling data, cancel the corresponding Future (e.g job1) and then start it again with the new scheduling data.
The key idea here is to get control on the Futures as they are created, so to save them in some state variables, so that when something in scheduling data changes, you can cancel them.
Here is the working code:
applicationContext.xml
<task:annotation-driven />
<task:scheduler id="infScheduler" pool-size="10"/>
The singleton bean, that holds the Futures
#Component
public class SchedulerServiceImpl implements SchedulerService {
private static final Logger logger = LoggerFactory.getLogger(SchedulerServiceImpl.class);
#Autowired
#Qualifier(value="infScheduler")
private TaskScheduler taskScheduler;
#Autowired
private MyService myService;
private ScheduledFuture job1;//for other jobs you can add new private state variables
//Call this on deployment from the ScheduleDataRepository and everytime when schedule data changes.
#Override
public synchronized void scheduleJob(int jobNr, long newRate) {//you are free to change/add new scheduling data, but suppose for now you only want to change the rate
if (jobNr == 1) {//instead of if/else you could use a map with all job data
if (job1 != null) {//job was already scheduled, we have to cancel it
job1.cancel(true);
}
//reschedule the same method with a new rate
job1 = taskScheduler.scheduleAtFixedRate(new ScheduledMethodRunnable(myService, "methodInMyServiceToReschedule"), newRate);
}
}
}
What about using Set<ScheduledTask> ScheduledTaskRegistrar.getScheduledTasks() to get all schedules tasks and calling ScheduledTask::cancel() ?
or maybe executing ThreadPoolTaskScheduler::shutdown()
and recreating ThreadPoolTaskScheduler and setting it again in ScheduledTaskRegistrar ?
The following, an improved version of this code, seems a working POC based on Spring Boot. You can start and stop the scheduled tasks any number of times based on a table configuration. But you can't start a stopped job from where it was stopped.
1) In the main class, make sure scheduling is enabled, and perhaps configure a ThreadPoolTaskScheduler with size more than one so scheduled tasks may run in parallel.
#SpringBootApplication
#EnableScheduling
#Bean
public TaskScheduler poolScheduler() {
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setThreadNamePrefix("ThreadPoolTaskScheduler");
scheduler.setPoolSize(10);
scheduler.initialize();
return scheduler;
}
2) an object that contains the schedule configuration, e.g. a cron like configuration in this case:
public class ScheduleConfigVo {
//some constructors, getter/setters
private String taskName;
private String configValue; // like */10 * * * * * for cron
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
ScheduleConfigVo that = (ScheduleConfigVo) o;
return taskName.equals(that.taskName) &&
configValue.equals(that.configValue) ;
}
#Override
public int hashCode() {
return Objects.hash(taskName, configValue);
}
}
equals and hashCode are needed since object comparison will be conducted.
3) I use mybatis, so the sheduled selection is something like:
#Mapper
public interface ScheduleConfigMapper {
List<ScheduleConfigVo> getAllConfigure();
}
and
public class ScheduleConfigMapperImpl implements ScheduleConfigMapper {
#Override
public List<ScheduleConfigVo>getAllConfigure() {
return getAllConfigure();
}
}
with a simple companion mybatis xml configuration (not shown here but can find it anywhere in the internet).
4) create a table and populate it with a record
CREATE TABLE "SCHEDULER"
( "CLASS_NAME" VARCHAR2(100), --PK
"VALUE" VARCHAR2(20 BYTE) --not null
)
and populated it with a record class_name=Task1, value=*/10 * * * * * etc. => run like a cron every ten seconds
5) the scheduler part:
#Service
public class DynamicScheduler implements SchedulingConfigurer {
#Autowired
private ScheduleConfigMapper repo;
#Autowired
private Runnable [] tsks;
#Autowired
private TaskScheduler tsch;
private ScheduledTaskRegistrar scheduledTaskRegistrar;
private ScheduledFuture future;
private Map<String, ScheduledFuture> futureMap = new ConcurrentHashMap<>(); // for the moment it has only class name
List<ScheduleConfigVo> oldList = new ArrayList<>();
List<ScheduleConfigVo> newList;
List<ScheduleConfigVo> addList = new ArrayList<>();
List<ScheduleConfigVo> removeList = new ArrayList<>();
#Override
public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
if (scheduledTaskRegistrar == null) {
scheduledTaskRegistrar = taskRegistrar;
}
if (taskRegistrar.getScheduler() == null) {
taskRegistrar.setScheduler(tsch);
}
updateJobList();
}
#Scheduled(fixedDelay = 5000)
public void updateJobList() {
newList = repo.getConfigure()== null ? new ArrayList<>() : repo.getConfigure();
addList.clear();
removeList.clear();
if (!newList.isEmpty()) {
//compare new List with oldList
if (!oldList.isEmpty()) {
addList = newList.stream().filter(e -> !oldList.contains(e)).collect(Collectors.toList());
removeList = oldList.stream().filter(e -> !newList.contains(e)).collect(Collectors.toList());
} else {
addList = new ArrayList<>(newList); // nothing to remove
}
} else { // nothing to add
if (!oldList.isEmpty()) {
removeList = new ArrayList<>(oldList);
} // else removeList = 0
}
log.info("addList="+ addList.toString());
log.info("removeList="+ removeList.toString());
//re-schedule here
for ( ScheduleConfigVo conf : removeList ) {
if ( !futureMap.isEmpty()){
future = futureMap.get(conf.getTaskName());
if (future != null) {
log.info("cancelling task "+conf.getTaskName() +" ...");
future.cancel(true);
log.info(conf.getTaskName() + " isCancelled = " + future.isCancelled());
futureMap.remove(conf.getTaskName());
}
}
}
for ( ScheduleConfigVo conf : addList ) {
for (Runnable o: tsks) {
if (o.getClass().getName().contains(conf.getTaskName())) { // o has fqn whereas conf has class name only
log.info("find " + o.getClass().getName() + " to add to scheduler");
future = scheduledTaskRegistrar.getScheduler().schedule(o, (TriggerContext a) -> {
CronTrigger crontrigger = new CronTrigger(conf.getConfigValue());
return crontrigger.nextExecutionTime(a);
});
futureMap.put(o.getClass().getName().substring(o.getClass().getName().lastIndexOf('.')+1), future);
}
}
}
oldList.clear();
oldList= newList;
}
6) one or more Runnable tasks that actually does the cron work, for instance:
#Slf4j
#Service
public class Task1 implements Runnable {
#Override
public void run() {
log.info("Task1 is running...");
}
}
Once the application is started, the cron job will run. The running interval changes as the value in the table changes, and the job stops as the table entry is removed.
Note that if the job runs longer than the cron interval, the next run is after the previous job finishes. You can simulate this situation by adding, for instance, sleep 15 seconds in Task1 above to test it. Sometimes after being cancelled, a job maybe still run till it's done.
***Just edit to add that if folks like lambda to save some lines, the above removeList and addList can be modified as:
removeList.stream().filter(conf -> {
future = futureMap.get(conf.getTaskName());
return future != null;
}).forEach((conf) -> {
log.info("cancelling task " + conf.getTaskName() + " ...");
future.cancel(true);
log.info(conf.getTaskName() + " isCancelled = " + future.isCancelled());
});
and
Arrays.stream(tsks).forEach(task -> {
addList.stream().filter(conf -> task.getClass().getName().contains(conf.getTaskName())).forEach(conf -> {
log.info("find " + task.getClass().getName() + " to add to scheduler");
future = scheduledTaskRegistrar.getScheduler().schedule(task, (TriggerContext a) -> {
CronTrigger crontrigger = new CronTrigger(conf.getConfigValue());
return crontrigger.nextExecutionTime(a);
});
futureMap.put(task.getClass().getName().substring(task.getClass().getName().lastIndexOf('.') + 1), future);
});
});
One simple approach is to only ever add new tasks, not to try and cancel or restart the scheduler.
Each time the configuration changes, just add a new task with its new configuration.
Then, whenever a task runs, it must first check some state (by querying database, or lookup up in a concurrent map, or whatever) to decide if it is the latest version. If it is, then it should proceed. Otherwise, it should end immediately.
The only downside is that if you are changing job configuration frequently compared to how often they run, then of course the list of scheduled tasks will keep growing in memory.

How to use LeaderElection recipe efficiently using Curator for Zookeeper?

I am using Apache Curator library for doing leadership election on the Zookeeper. I have my application code deployed in various machines and I need to execute my code from one machine only so that's why I am doing leadership election on the zookeeper so that I can check if I am the leader, then execute this code.
Below is my LeaderElectionExecutor class which makes sure I am having one Curator instance per application
public class LeaderElectionExecutor {
private ZookeeperClient zookClient;
private static final String LEADER_NODE = "/testleader";
private static class Holder {
static final LeaderElectionExecutor INSTANCE = new LeaderElectionExecutor();
}
public static LeaderElectionExecutor getInstance() {
return Holder.INSTANCE;
}
private LeaderElectionExecutor() {
try {
String hostname = Utils.getHostName();
String nodes = "host1:2181,host2:2181;
zookClient = new ZookeeperClient(nodes, LEADER_NODE, hostname);
zookClient.start();
// added sleep specifically for the leader to get selected
// since I cannot call isLeader method immediately after starting the latch
TimeUnit.MINUTES.sleep(1);
} catch (Exception ex) {
// logging error
System.exit(1);
}
}
public ZookeeperClient getZookClient() {
return zookClient;
}
}
And below is my ZookeeperClient code -
// can this class be improved in any ways?
public class ZookeeperClient {
private CuratorFramework client;
private String latchPath;
private String id;
private LeaderLatch leaderLatch;
public ZookeeperClient(String connString, String latchPath, String id) {
client = CuratorFrameworkFactory.newClient(connString, new ExponentialBackoffRetry(1000, Integer.MAX_VALUE));
this.id = id;
this.latchPath = latchPath;
}
public void start() throws Exception {
client.start();
leaderLatch = new LeaderLatch(client, latchPath, id);
leaderLatch.start();
}
public boolean isLeader() {
return leaderLatch.hasLeadership();
}
public Participant currentLeader() throws Exception {
return leaderLatch.getLeader();
}
public void close() throws IOException {
leaderLatch.close();
client.close();
}
public CuratorFramework getClient() {
return client;
}
public String getLatchPath() {
return latchPath;
}
public String getId() {
return id;
}
public LeaderLatch getLeaderLatch() {
return leaderLatch;
}
}
Now in my application, I am using the code like this -
public void method01() {
ZookeeperClient zookClient = LeaderElectionExecutor.getInstance().getZookClient();
if (zookClient.isLeader()) {
// do something
}
}
public void method02() {
ZookeeperClient zookClient = LeaderElectionExecutor.getInstance().getZookClient();
if (zookClient.isLeader()) {
// do something
}
}
Problem Statement:-
In the Curator library - Calling isLeader() immediately after starting the latch will not work. It takes time for the leader to get selected. And because of this reason only, I have added a sleep of 1 minute in my LeaderElectionExecutor code which works fine but I guess is not the right way to do this.
Is there any better way of doing this? Keeping this in mind, I need a way to check whether I am the leader then execute this piece of code. I cannot do everything in a single method so I need to call isLeader method from different classes and methods to check if I am the leader then execute this piece of code only.
I am using Zookeeper 3.4.5 and Curator 1.7.1 version.
Once I solved a problem very similar to yours. This is how I did it.
First, I had my objects managed by Spring. So, I had a LeaderLatch that was injectable through the container. One of the components that used the LeaderLatch was a LeadershipWatcher, an implementation of Runnable interface that would dispatch the leadership event to other components. These last components were implementations of an interface that I named LeadershipObserver. The implementation of the LeadershipWatcher was mostly like the following code:
#Component
public class LeadershipWatcher implements Runnable {
private final LeaderLatch leaderLatch;
private final Collection<LeadershipObserver> leadershipObservers;
/* constructor with #Inject */
#Override
public void run() {
try {
leaderLatch.await();
for (LeadershipObserver observer : leadershipObservers) {
observer.granted();
}
} catch (InterruptedException e) {
for (LeadershipObserver observer : leadershipObservers) {
observer.interrupted();
}
}
}
}
As this is just a sketch-up, I recommend you to enhance this code, maybe applying the command pattern for calling the observers, or even submitting the observers to thread pools, if their job are blocking or long-running CPU intensive tasks.
I've not worked with zookeeper or curator before, so take my answer with a grain of salt.
Set a flag.
Boolean isLeaderSelected = false;
At the beginning of the Latch, set the flag to false.
When the leader has been selected, set the flag to true.
In the isLeader() function:
isLeader(){
while(!isLeaderSelected){} //waits until leader is selected
//do the rest of the function
}
This is also a relatively hacky workaround, but it should allow the isLeader method to execute as soon as it can. In the case that they are in different classes, a getter should be able to provide isLeaderSelected.
leaderLatch = new LeaderLatch(curatorClient, zkPath, String.valueOf(new Random().nextInt()));
leaderLatch.start();
Participant participant;
while(true) {
participant = leaderLatch.getLeader();
// Leader election happens asynchronously after calling start, this is a hack to wait until election happens
if (!(participant.getId().isEmpty() || participant.getId().equalsIgnoreCase(""))) {
break;
}
}
if(leaderLatch.hasLeadership()) {
...
}
Note that getLeader returns a dummy participant with id "" until it elects a leader.
Here's to reviving an old question...
This is similar to the answer srav gave, but I would caution against using that code because it utilizes a busy-wait and can cause certain callbacks that are issued in-thread to never be called, possibly blocking forever. Furthermore, it could retry forever if there are real issues.
This was my solution, which utilizes the CuratorClient's retry policy to attempt waiting on leadership election if necessary.
RetryPolicy retryPolicy = _client.getZookeeperClient().getRetryPolicy();
RetrySleeper awaitLeadership = _leaderLatch::await;
final long start = System.currentTimeMillis();
int count = 0;
do {
try {
// curator will return a dummy leader in the case when a leader has
// not yet actually been elected. This dummy leader will have isLeader
// set to false, so we need to check that we got a true leader
if (_leaderLatch.getLeader().isLeader()) {
return;
}
} catch (KeeperException.NoNodeException e) {
// this is the case when the leader node has not yet been created
// by any client - this is fine because we are still waiting for
// the algorithm to start up so we ignore the error
}
} while (retryPolicy.allowRetry(count++, System.currentTimeMillis() - start, awaitLeadership));
// we have exhausted the retry policy and still have not elected a leader
throw new IOException("No leader was elected within the specified retry policy!");
Though taking a look at your CuratorFramework initialization I'd caution against using Integer.MAX_VALUE when specifying the retry policy...
I hope this helps!

Categories

Resources