We want to start a job in the background so it doesn't use all resources within the application and can influence the 'normal' tasks of the application. It should start from within the a running Java app and not executed from the command line.
Does anybody know how to start a Spring Batch job as a background task/daemon using Spring scheduling?
After some more investigation I found out you can configure a TaskExecuter for a JobLauncher. You can then use a SimpleAsyncTaskExector and configure it as a deamon and set the thread priority.
#Bean
public JobLauncher jobLauncher(final JobRepository jobRepository, final TaskExecutor taskExecutor) {
final SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
jobLauncher.setTaskExecutor(taskExecutor);
return jobLauncher;
}
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setDaemon(true);
taskExecutor.setThreadPriority(Thread.MIN_PRIORITY);
return taskExecutor;
}
Related
I am having one job in the spring batch application which got successfully completed but after completion, its thread goes to waiting state. As more instances of job get executed, number of threads of tomcat keeps on increasing.
Here is the Job Configuration :
#Bean
#Scope("singleton")
#Qualifier(value = "job1")
public Job job() {
return jobBuilderFactory.get("job1")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.preventRestart()
.build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<Buylead, Buylead>chunk(10000)
.reader(itemReader())
.writer(itemWriter())
.build();
}
public JdbcCursorItemReader<Buylead> itemReader() {
JdbcCursorItemReader<Buylead> reader = new JdbcCursorItemReader<>();
reader.setDataSource(dataSource);
reader.setSql(" --- WHatever Query ----");
reader.setRowMapper(new MapperBL());
return reader;
}
public ItemWriter<Buylead> itemWriter(){
FlatFileItemWriter<Buylead> itemWriter = new FlatFileItemWriter() ;
itemWriter.setResource(new FileSystemResource("output/buyleadSolrDoc.xls"));
itemWriter.setLineAggregator(new DelimitedLineAggregator<Buylead>() { {
setFieldExtractor(new BeanWrapperFieldExtractor<Buylead>() { {
setNames(new String[] {*************** }); } }); } });
itemWriter.setShouldDeleteIfEmpty(true);
return itemWriter;
}
And data Source is kept in another configuration class.
And here is the Main class
#SpringBootApplication
#EnableBatchProcessing
public class Application extends DefaultBatchConfigurer{
public static void main(String[] args) {
SpringApplication.run(SearchApplication.class, args);
}
#Override
public void setDataSource(DataSource dataSource) {
// override to do not set datasource even if a datasource exist.
// initialize will use a Map based JobRepository (instead of database)
}
}
The job is getting executed successfully. But its thread goes to wait state.
And here is the portion of ThreadDump for the application
http-nio-8080-exec-2" #22 daemon prio=5 os_prio=0 cpu=0.11ms elapsed=29.43s tid=0x00007fcb6d6ddd60 nid=0xc724 waiting on condition [0x00007fcae3ffe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base#15.0.1/Native Method)
- parking to wait for <0x0000000713c00710> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base#15.0.1/LockSupport.java:341)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base#15.0.1/AbstractQueuedSynchronizer.java:505)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base#15.0.1/ForkJoinPool.java:3137)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base#15.0.1/AbstractQueuedSynchronizer.java:1614)
at java.util.concurrent.LinkedBlockingQueue.take(java.base#15.0.1/LinkedBlockingQueue.java:435)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base#15.0.1/ThreadPoolExecutor.java:1056)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base#15.0.1/ThreadPoolExecutor.java:1116)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base#15.0.1/ThreadPoolExecutor.java:630)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(java.base#15.0.1/Thread.java:832)*
Please suggest the way to close the thread which is in Park state.
Spring Batch does not create or manage threads directly. It delegates that to a TaskExecutor implementation in different places:
The JobLauncher delegates to a task executor to launch jobs
The StepBuilder delegates to a task executor to create multi-threaded steps
The FlowBuilder delegates to a task executor to create split flows and run steps in parallel
The AsyncItemProcessor delegates to a task executor to process items asynchronously in separate threads
The TaskExecutorPartitionHandler delegates to a task executor to handle partitions in different threads
etc
In all these cases, threads life cycle is managed by the underlying TaskExecutor implementation. If you use a task executor with Spring Batch, you need to make sure that it is correctly stopped/shutdown once it is not needed anymore. From what you shared, you do not seem to use a task executor in your Spring Batch configuration, so you should have a task executor defined somewhere in your application context which is not stopped correctly.
I have this service Scheduling a task:
#ApplicationScoped
public class PaymentService {
#Transactional
public Payment scheduleNewPayment(Payment payment) throws ParseException, SchedulerException {
Payment.persist(payment);
JobDetail job = JobBuilder.newJob(PaymentJob.class)
.withIdentity(String.format("job%d", payment.id), "payment-job-group")
.build();
Date parsed = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse(payment.dueDate);
SimpleTrigger trigger = (SimpleTrigger) TriggerBuilder.newTrigger()
.withIdentity(String.format("trigger%d", payment.id), "trigger-group")
.startAt(parsed)
.forJob(job)
.build();
SchedulerFactory schedulerFactory = new StdSchedulerFactory();
Scheduler scheduler = schedulerFactory.getScheduler();
scheduler.scheduleJob(job, trigger);
scheduler.start();
return payment;
}
}
And this job:
#ApplicationScoped
public class PaymentJob implements Job {
#Override
public void execute(JobExecutionContext jobExecutionContext) throws JobExecutionException {
System.out.println(Payment.count());
}
}
But I can not perform a DB operation inside the Job context (jobExecutionContext.getScheduler().getContext() is null by the way).
I'm running my app with quarkus, the hibernate operation comes from Hibernate Panache and the Scheduler is quartz.
First of all, you should use the underlying managed Quartz Scheduler instance : #Inject org.quartz.Scheduler (I suppose you're using the quarkus-quartz extension).
The other "problem" is that the default Quartz job factory simply calls new PaymentJob() and so no injection/initialization is performed. Quarkus is only using a custom factory for the jobs generated for methods annotated with #Scheduled. If you don't need injection then simply remove the superfluous#ApplicationScoped from the PaymentJob class.
Finally, you need to activate all the necessary CDI contexts manually. It's very likely that the request context is needed. You can copy the following snippet: https://github.com/quarkusio/quarkus/blob/master/extensions/arc/runtime/src/main/java/io/quarkus/arc/runtime/BeanInvoker.java#L14-L24 into your execute() method.
jobExecutionContext.getScheduler().getContext() is null by the way
This is really odd. What exception/error do you actually get?
I have a method in a Spring Boot application that listens to a queue in ActiveMQ. I want to schedule the method so that it does not start listening to the queue on application startup and runs every X minutes.
Here is the method that I wrote to accomplish the task. I have disabled the JMSListener auto startup so that it does not start listening on when the application is started.
#Scheduled(fixedDelay = 1000, initialDelay = 1000)
#JmsListener(destination = "queueName")
public void receiveMessage(final Message jsonMessage) throws JMSException {
System.out.println("Received message " + jsonMessage);
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
factory.setConcurrency("1-1");
factory.setAutoStartup(false);
return factory;
}
But when I run the application I get an exception which says that the scheduled method cannot have arguments:
Encountered invalid #Scheduled method 'receiveMessage': Only no-arg methods may be annotated with #Scheduled
Is there a way I can schedule the JMSListener so that it starts after a delay on the application startup and is scheduled to run every X minutes and read messages from the queue?
You can't use #Scheduled there.
Use the JmsListenerEndpointRegistry bean to start and stop the listener when needed.
#JmsListener(id = "foo" ...)
registry.getListenerContainer("foo").start();
...
registry.getListenerContainer("foo").stop();
I have a multithreaded step configured with chunk commit-interval of 10 and throttle limit of 5. When I run a batch, ItemReader reads first fifty jobs(I think it's based on commit-interval and throttle limit) and process 5 jobs at a time(based on throttle-limit) then write's it. Till now it works fine.
After this, it does not read the next fifty jobs at a time and only one or two job(s) processing at a time until the end of remaining jobs.
Based on the throttle limit 5 should have been running at a time. I see no socket or any other exceptions.
I am new to spring batch, correct me if I am wrong at anything.
Here is how I have configured:
#Bean
public Job executeConcurrentJob() {
return jobBuilderFactory.get("executeConcurrentJob")
.listener(listener())
.start(step1())
.build();
}
#Bean
public TaskExecutor taskExecutor() {
return new SimpleAsyncTaskExecutor();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1").allowStartIfComplete(true)
.<BatchJob,BatchJob>chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.taskExecutor(taskExecutor())
.throttleLimit(5)
.build();
}
Thanks in Advance!
We have multiple spring batch job.But Each of them needs to be started individually.
Is there any way to create a Master Job or any controller in spring which will be responsible for executing all other batch jobs? So that we just have to execute the master job only,and all other started automatically.
I just explained on this question how you can start spring application with all jobs loaded in separate context. We have restart job which is scheduled each 10min and it checks for latest failed execution and attempts to restart couple of more times.
Your use case is pretty much the same, you can define all jobs in separate contexts with own configuration files, in config you can tell spring batch not to run them on startup by setting spring.batch.job.enabled: false and you can write your own launcher which uses JobExplorer to start jobs for you and you can schedule it or something.
Something like:
#Component
#EnableScheduling
public class AllJobLauncher {
#Autowired
JobExplorer jobExplorer;
#Autowired
private JobLauncher jobLauncher;
#Autowired
private JobRegistry jobRegistry;
#Scheduled(cron = "${some.cron:0 0/10 * * * ?}")
public void launchAllJobs() throws JobExecutionException {
final List<String> jobNames = jobExplorer.getJobNames();
for (final String jobName : jobNames) {
final Job job = jobRegistry.getJob(getJobName(organizationId));
final JobParameters jobParameters = new JobParametersBuilder() //build parameters
jobLauncher.run(job, jobParameters);
}
}
Just pay attention that JobLauncher in spring batch is by default sync so launcher will wait until job finishes. If you want to start jobs async you must place this configuration somewhere:
#Bean
public JobLauncher jobLauncher() {
final SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
final SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
jobLauncher.setTaskExecutor(simpleAsyncTaskExecutor);
return jobLauncher;
}