Develop Reference

how to avoid starting duplicate spring batch jobs?

I have a spring batch process which does data reading from the database and writing to file. Basically, the scenario is, that the user can send a request and the job will start and execute the process. But the issue is if the user sends the request 5 times there will be 5 different spring jobs started and running. But those are duplicates. So is there a way that we can avoid or block creating duplicate spring jobs?

You can create a JobExecutionListener that stops the current job execution if another one is already running and configure your job with that listener...
public class SingleExecutionJobListener implements JobExecutionListener {
private static String MATCH_ALL_PATTERN = ".*";
#Autowired
private JobExplorer jobExplorer;
#Autowired
private JobRegistry jobRegistry;
private String jobNamePattern = MATCH_ALL_PATTERN;
#Override
public void beforeJob(JobExecution jobExecution) {
Collection<String> jobNames = jobRegistry.getJobNames();
for (String jobName : jobNames) {
if (jobName.matches(StringUtils.defaultIfBlank(jobNamePattern, MATCH_ALL_PATTERN))) {
Set<JobExecution> jobExecutions = jobExplorer.findRunningJobExecutions(jobName);
if (CollectionUtils.isNotEmpty(jobExecutions)) {
for (JobExecution execution : jobExecutions) {
if (execution.getJobInstance().getId().compareTo(jobExecution.getJobInstance().getId()) != 0) {
jobExecution.stop();
throw new IllegalStateException(jobName + " instance " + execution.getJobInstance().getId()
+ " is currently running. Please restart this job when " + jobName + " has finished.");
}
}
}
}
}
}
#Override
public void afterJob(JobExecution jobExecution) {}
public String getJobNamePattern() {
return jobNamePattern;
}
public void setJobNamePattern(String jobNamePattern) {
this.jobNamePattern = jobNamePattern;
}
}

Quartz Scheduler scheduled but not Fire

i try to schedule a scheduler and only print (only here) when my sheduler fire, when i schedule it cheduled in QRTZ_TRIGGERS table and also in QRTZ_JOB_DETAILS table but not fire on time.
where my Quartz Scheuler Configuration like,
#Configuration
public class QuartzSchedulerConfiguration {
#Bean
public Scheduler getScheduler() throws SchedulerException {
return StdSchedulerFactory.getDefaultScheduler();
}
}
and quartz.properties file like,
## QuartzProperties
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount=10
org.quartz.threadPool.
threadsInheritContextClassLoaderOfInitializingThread=true
org.quartz.jobStore.tablePrefix=QRTZ_
#The datasource for the jobstore that is to be used
org.quartz.jobStore.dataSource=myDS
org.quartz.dataSource.myDS.driver=com.mysql.cj.jdbc.Driver
org.quartz.dataSource.myDS.URL=jdbc:mysql://localhost:3306/my_db?
useSSL=false
org.quartz.dataSource.myDS.user=root
org.quartz.dataSource.myDS.password=1111
org.quartz.dataSource.myDS.maxConnections=20
and Service class have,
#Autowired
private Scheduler scheduler;
public String addFTP(FTPDomain ftpDomain) {
String msg = "";
//Start schedule =======================================================
String str = "2019-01-16 13:10:01";
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
LocalDateTime localeDateTime = LocalDateTime.parse(str, formatter);
System.out.println("Date time is "+localeDateTime);
try {
ZonedDateTime dateTime = ZonedDateTime.of(localeDateTime, ZoneId.of("Asia/Kolkata"));
if(dateTime.isBefore(ZonedDateTime.now())) {
msg = "You are schedule a expired Date and time";
System.out.println(msg);
}
JobDetail jobDetail = buildJobDetail(ftpDomain);
Trigger trigger = buildJobTrigger(jobDetail, dateTime);
scheduler.scheduleJob(jobDetail, trigger); // ## here i schedule
System.out.println(jobDetail.getKey().getName()+" "+ jobDetail.getKey().getGroup()+""+ "Scheduller Scheduled Successfully!");
} catch (SchedulerException ex) {
System.out.println( "Error scheduling email. Please try later!");
}
//Stop schedulling=========================================================================
return ftpTransferDaoInterface.addFTP(ftpDomain);
}
private JobDetail buildJobDetail(FTPDomain ftpDamain) {
JobDataMap jobDataMap = new JobDataMap();
jobDataMap.put("destPath", ftpDamain.getServerDestinationPath());
jobDataMap.put("ip", ftpDamain.getServerIP());
jobDataMap.put("user", ftpDamain.getServerUser());
return JobBuilder.newJob(FTPTransferJob.class)
.withIdentity(UUID.randomUUID().toString(), "schedule-jobs")
.withDescription("Send schedule Job")
.usingJobData(jobDataMap)
.storeDurably().build();
}
private Trigger buildJobTrigger(JobDetail jobDetail, ZonedDateTime startAt) {
return TriggerBuilder.newTrigger()
.forJob(jobDetail)
.withIdentity(jobDetail.getKey().getName(), "email-triggers")
.withDescription("Send Email Trigger")
.startAt(Date.from(startAt.toInstant()))
.withSchedule(SimpleScheduleBuilder
.simpleSchedule()
.withMisfireHandlingInstructionFireNow()
).build();
}
and my Quartz JOB class like,
#Component
public class FTPTransferJob extends QuartzJobBean {
protected void executeInternal(JobExecutionContext context)
throws JobExecutionException {
System.err.println("Executing Job with key {}"+
context.getJobDetail().getKey());
JobDataMap jobDataMap = context.getMergedJobDataMap();
String destination = jobDataMap.getString("destPath");
String ip = jobDataMap.getString("ip");
String user = jobDataMap.getString("user");
System.err.println("Schedulled Job Executed Here "+destination+", ip
is "+ip+", user is "+user);
}
}
can any one find the solution ? Thanks...

Spring batch - restart FAILED job that uses TaskExecutor

What is the proper way to restart FAILED spring batch job that is using TaskExecutor?
I have a job that is loading data from HTTP and sometimes there is 500 error - making this job fail). I would like to restart this job until it is successful.
If I make JobExecutionListener and implement logic inside afterJob() method, I get error message that this job is actually running. If I use RetryTemplate from Spring, this also doesn't work since this is running inside TaskExecutor.
Any code sample would be of a great help.

Finally I solved the issue by re-implementing JobLauncher:
public class FaultTolerantJobLauncher implements JobLauncher, InitializingBean {
protected static final Log logger = LogFactory.getLog(FaultTolerantJobLauncher.class);
private JobRepository jobRepository;
private RetryTemplate retryTemplate;
private TaskExecutor taskExecutor;
/**
* Run the provided job with the given {#link JobParameters}. The
* {#link JobParameters} will be used to determine if this is an execution
* of an existing job instance, or if a new one should be created.
*
* #param job the job to be run.
* #param jobParameters the {#link JobParameters} for this particular
* execution.
* #return JobExecutionAlreadyRunningException if the JobInstance already
* exists and has an execution already running.
* #throws JobRestartException if the execution would be a re-start, but a
* re-start is either not allowed or not needed.
* #throws JobInstanceAlreadyCompleteException if this instance has already
* completed successfully
* #throws JobParametersInvalidException
*/
#Override
public JobExecution run(final Job job, final JobParameters jobParameters)
throws JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException,
JobParametersInvalidException {
Assert.notNull(job, "The Job must not be null.");
Assert.notNull(jobParameters, "The JobParameters must not be null.");
final AtomicReference<JobExecution> executionReference = new AtomicReference<>();
JobExecution lastExecution = jobRepository.getLastJobExecution(job.getName(), jobParameters);
if (lastExecution != null) {
if (!job.isRestartable()) {
throw new JobRestartException("JobInstance already exists and is not restartable");
}
/*
* validate here if it has stepExecutions that are UNKNOWN, STARTING, STARTED and STOPPING
* retrieve the previous execution and check
*/
for (StepExecution execution : lastExecution.getStepExecutions()) {
BatchStatus status = execution.getStatus();
if (status.isRunning() || status == BatchStatus.STOPPING) {
throw new JobExecutionAlreadyRunningException("A job execution for this job is already running: "
+ lastExecution);
} else if (status == BatchStatus.UNKNOWN) {
throw new JobRestartException(
"Cannot restart step [" + execution.getStepName() + "] from UNKNOWN status. "
+ "The last execution ended with a failure that could not be rolled back, "
+ "so it may be dangerous to proceed. Manual intervention is probably necessary.");
}
}
}
// Check the validity of the parameters before doing creating anything
// in the repository...
job.getJobParametersValidator().validate(jobParameters);
taskExecutor.execute(new Runnable() {
#Override
public void run() {
try {
retryTemplate.execute(new FaultTolerantJobRetryCallback(executionReference, job, jobParameters));
} catch (TaskRejectedException e) {
executionReference.get().upgradeStatus(BatchStatus.FAILED);
if (executionReference.get().getExitStatus().equals(ExitStatus.UNKNOWN)) {
executionReference.get().setExitStatus(ExitStatus.FAILED.addExitDescription(e));
}
jobRepository.update(executionReference.get());
}
}
});
return executionReference.get();
}
/**
* Set the JobRepsitory.
*
* #param jobRepository
*/
public void setJobRepository(JobRepository jobRepository) {
this.jobRepository = jobRepository;
}
/**
* Set the retryTemplate
*
* #param retryTemplate
*/
public void setRetryTemplate(RetryTemplate retryTemplate) {
this.retryTemplate = retryTemplate;
}
/**
* Set the TaskExecutor. (Optional)
*
* #param taskExecutor
*/
public void setTaskExecutor(TaskExecutor taskExecutor) {
this.taskExecutor = taskExecutor;
}
/**
* Ensure the required dependencies of a {#link JobRepository} have been
* set.
*/
#Override
public void afterPropertiesSet() throws Exception {
Assert.state(jobRepository != null, "A JobRepository has not been set.");
Assert.state(retryTemplate != null, "A RetryTemplate has not been set.");
if (taskExecutor == null) {
logger.info("No TaskExecutor has been set, defaulting to synchronous executor.");
taskExecutor = new SyncTaskExecutor();
}
}
private class FaultTolerantJobRetryCallback implements RetryCallback<Object, RuntimeException> {
private final AtomicReference<JobExecution> executionReference;
private final Job job;
private final JobParameters jobParameters;
FaultTolerantJobRetryCallback(AtomicReference<JobExecution> executionReference, Job job, JobParameters jobParameters){
this.executionReference = executionReference;
this.job = job;
this.jobParameters = jobParameters;
}
#Override
public Object doWithRetry(RetryContext retryContext) {
if(!job.isRestartable()){
//will be set only once and in case that job can not be restarted we don't retry
retryContext.setExhaustedOnly();
}
if(retryContext.getRetryCount() > 0){
logger.info("Job: [" + job + "] retrying/restarting with the following parameters: [" + jobParameters
+ "]");
}
try {
/*
* There is a very small probability that a non-restartable job can be
* restarted, but only if another process or thread manages to launch
* <i>and</i> fail a job execution for this instance between the last
* assertion and the next method returning successfully.
*/
executionReference.set(jobRepository.createJobExecution(job.getName(), jobParameters));
logger.info("Job: [" + job + "] launched with the following parameters: [" + jobParameters
+ "]");
job.execute(executionReference.get());
logger.info("Job: [" + job + "] completed with the following parameters: [" + jobParameters
+ "] and the following status: [" + executionReference.get().getStatus() + "]");
}
catch (JobInstanceAlreadyCompleteException | JobExecutionAlreadyRunningException e){
retryContext.setExhaustedOnly(); //don't repeat if instance already complete or running
rethrow(e);
}
catch (Throwable t) {
logger.info("Job: [" + job
+ "] failed unexpectedly and fatally with the following parameters: [" + jobParameters
+ "]", t);
rethrow(t);
}
if(job.isRestartable() && executionReference.get().getStatus() == BatchStatus.FAILED){
//if job failed and can be restarted, use retry template to restart the job
throw new TaskRejectedException("RetryTemplate failed too many times");
}
return null;
}
private void rethrow(Throwable t) {
if (t instanceof RuntimeException) {
throw (RuntimeException) t;
}
else if (t instanceof Error) {
throw (Error) t;
}
throw new IllegalStateException(t);
}
}
}

How to make StatefulJobs run consecutively (not concurrently)?

I have a scheduler:
SchedulerFactory schedFact = new org.quartz.impl.StdSchedulerFactory("quartz.properties");
sched = schedFact.getScheduler();
sched.start();
JobDetail jobDetail;
CronTrigger trigger;
for (ReportDetails report : reports) {
jobDetail = new JobDetail(report.getName() + _REPORT, GRP, ReportJob.class);
jobDetail.getJobDataMap().put(ReportJob.DATA_REPORT, report);
sched.addJob(jobDetail, true);
if (report.getCronExp() != null && report.getCronExp().length() > 0) {
trigger = new CronTrigger(report.getName() + _TRIGGER, GRP);
trigger.setCronExpression(report.getCronExp());
trigger.setJobGroup(GRP);
trigger.setJobName(report.getName() + _REPORT);
sched.scheduleJob(trigger);
}
}
And there is my job instance:
public class ReportJob implements StatefulJob {
private static final Logger logger = Logger.getLogger(ReportJob.class);
public void execute(JobExecutionContext context) throws JobExecutionException {
ReportDetails report = (ReportDetails) context.getJobDetail().getJobDataMap().get(DATA_REPORT);
report.getLogger().info("job for report started");
...
report.getLogger().info("Job for report ended");
}
}
The thing is that though ReportJob implements StatefulJob indeed jobs run concurrently.
QuartzScheduler_Worker-1 | job for report started
QuartzScheduler_Worker-2 | job for report started
QuartzScheduler_Worker-2 | job for report ended
QuartzScheduler_Worker-1 | job for report ended
I want them to run consecutively, one by one. How to resolve this issue?

I understood my error: StatefulJob prevents multiple instances of a job WITH THE SAME KEY from running at the same time. While I'd created the jobs with different keys.

Quartz #DisallowConcurrentExecution not working as expected

I am using java quartz schedular. I am able to schedule jobs perfectly, though what i want is wait for job to finish before runing the second round because the time it takes to run each job varies.
I used #DisallowConcurrentExecution, what it did is only make the job to run once and never again. From job listener shows that the job finished successfully once.
Job
=============================================================
#DisallowConcurrentExecution
public class SalesJob implements Job{
List<Transaction> unsentTransaction = new ArrayList<Transaction>();
List<Sale> sales = new ArrayList<Sale>();
public void execute(JobExecutionContext jec) throws JobExecutionException {
System.out.println("Sales Job. . .");
}
}
Job Listener:
public class SalesJobListener implements JobListener{
public static final String LISTENER_NAME = "dummyJobListenerName";
public String getName() {
return LISTENER_NAME;
}
public void jobToBeExecuted(JobExecutionContext context) {
String jobName = context.getJobDetail().getKey().toString();
System.out.println("jobToBeExecuted");
System.out.println("Job : " + jobName + " is going to start...");
}
public void jobExecutionVetoed(JobExecutionContext jec) {
System.out.println("jobExecutionVetoed");
}
public void jobWasExecuted(JobExecutionContext context, JobExecutionException jobException) {
System.out.println("jobWasExecuted");
String jobName = context.getJobDetail().getKey().toString();
System.out.println("Job : " + jobName + " is finished...");
System.out.println("=====================================");
System.out.println("==========" + new Date() + "===========");
if (!jobException.getMessage().equals("")) {
System.out.println(
"Exception thrown by: " + jobName + " Exception: " + jobException.getMessage());
}
}
}
This is the schedular
JobKey salesJobKey = new JobKey("salesJob", "group1");
JobDetail salesJob = JobBuilder.newJob(SalesJob.class)
.withIdentity(salesJobKey).build();
Trigger salesTrigger = TriggerBuilder
.newTrigger()
.withIdentity("salesTrigger", "group1")
.withSchedule(
CronScheduleBuilder.cronSchedule("0/5 * * * * ?"))
.build();
Scheduler scheduler = new StdSchedulerFactory().getScheduler();
scheduler.getListenerManager().addJobListener(
new SalesJobListener(), KeyMatcher.keyEquals(salesJobKey)
);
scheduler.start();
scheduler.scheduleJob(salesJob, salesTrigger);

PROBLEM
it executed this time, Wed Nov 25 12:01:15 EAT 2015 and now is Wed Nov 25 12:32 2015, so basically i have waited > 30 mins. . . and there is no another job
That is saying the Scheduler is not working.
WHY?
you cannot execute a job at second 15 of a minute because the pattern: 0/5 * * * * ? makes scheduler to run ONLY at seconds 0 and 5 of each minute.
Using #DisallowConcurrentExecution will prevent execution a Job if another one of same type is already running.
SOLUTION:
The mistake is in the order of your code, you execute then scheduler (scheduler.start();) before tell that it must schedule a job (scheduler.scheduleJob(salesJob, salesTrigger);):
scheduler.start();
scheduler.scheduleJob(salesJob, salesTrigger);
Check this example and swap your lines:
scheduler.scheduleJob(salesJob, salesTrigger);
scheduler.start();
That's all...

Just Add the below null check
if(null != jobException) {
if (!jobException.getMessage().equals("")) {
logger.debug("Exception thrown by: " + jobName
+ " Exception: " + jobException.getMessage());
}
}
Your code will work