I have a multithreaded step configured with chunk commit-interval of 10 and throttle limit of 5. When I run a batch, ItemReader reads first fifty jobs(I think it's based on commit-interval and throttle limit) and process 5 jobs at a time(based on throttle-limit) then write's it. Till now it works fine.
After this, it does not read the next fifty jobs at a time and only one or two job(s) processing at a time until the end of remaining jobs.
Based on the throttle limit 5 should have been running at a time. I see no socket or any other exceptions.
I am new to spring batch, correct me if I am wrong at anything.
Here is how I have configured:
#Bean
public Job executeConcurrentJob() {
return jobBuilderFactory.get("executeConcurrentJob")
.listener(listener())
.start(step1())
.build();
}
#Bean
public TaskExecutor taskExecutor() {
return new SimpleAsyncTaskExecutor();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1").allowStartIfComplete(true)
.<BatchJob,BatchJob>chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.taskExecutor(taskExecutor())
.throttleLimit(5)
.build();
}
Thanks in Advance!
Related
I am having one job in the spring batch application which got successfully completed but after completion, its thread goes to waiting state. As more instances of job get executed, number of threads of tomcat keeps on increasing.
Here is the Job Configuration :
#Bean
#Scope("singleton")
#Qualifier(value = "job1")
public Job job() {
return jobBuilderFactory.get("job1")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.preventRestart()
.build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<Buylead, Buylead>chunk(10000)
.reader(itemReader())
.writer(itemWriter())
.build();
}
public JdbcCursorItemReader<Buylead> itemReader() {
JdbcCursorItemReader<Buylead> reader = new JdbcCursorItemReader<>();
reader.setDataSource(dataSource);
reader.setSql(" --- WHatever Query ----");
reader.setRowMapper(new MapperBL());
return reader;
}
public ItemWriter<Buylead> itemWriter(){
FlatFileItemWriter<Buylead> itemWriter = new FlatFileItemWriter() ;
itemWriter.setResource(new FileSystemResource("output/buyleadSolrDoc.xls"));
itemWriter.setLineAggregator(new DelimitedLineAggregator<Buylead>() { {
setFieldExtractor(new BeanWrapperFieldExtractor<Buylead>() { {
setNames(new String[] {*************** }); } }); } });
itemWriter.setShouldDeleteIfEmpty(true);
return itemWriter;
}
And data Source is kept in another configuration class.
And here is the Main class
#SpringBootApplication
#EnableBatchProcessing
public class Application extends DefaultBatchConfigurer{
public static void main(String[] args) {
SpringApplication.run(SearchApplication.class, args);
}
#Override
public void setDataSource(DataSource dataSource) {
// override to do not set datasource even if a datasource exist.
// initialize will use a Map based JobRepository (instead of database)
}
}
The job is getting executed successfully. But its thread goes to wait state.
And here is the portion of ThreadDump for the application
http-nio-8080-exec-2" #22 daemon prio=5 os_prio=0 cpu=0.11ms elapsed=29.43s tid=0x00007fcb6d6ddd60 nid=0xc724 waiting on condition [0x00007fcae3ffe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base#15.0.1/Native Method)
- parking to wait for <0x0000000713c00710> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base#15.0.1/LockSupport.java:341)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base#15.0.1/AbstractQueuedSynchronizer.java:505)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base#15.0.1/ForkJoinPool.java:3137)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base#15.0.1/AbstractQueuedSynchronizer.java:1614)
at java.util.concurrent.LinkedBlockingQueue.take(java.base#15.0.1/LinkedBlockingQueue.java:435)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base#15.0.1/ThreadPoolExecutor.java:1056)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base#15.0.1/ThreadPoolExecutor.java:1116)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base#15.0.1/ThreadPoolExecutor.java:630)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(java.base#15.0.1/Thread.java:832)*
Please suggest the way to close the thread which is in Park state.
Spring Batch does not create or manage threads directly. It delegates that to a TaskExecutor implementation in different places:
The JobLauncher delegates to a task executor to launch jobs
The StepBuilder delegates to a task executor to create multi-threaded steps
The FlowBuilder delegates to a task executor to create split flows and run steps in parallel
The AsyncItemProcessor delegates to a task executor to process items asynchronously in separate threads
The TaskExecutorPartitionHandler delegates to a task executor to handle partitions in different threads
etc
In all these cases, threads life cycle is managed by the underlying TaskExecutor implementation. If you use a task executor with Spring Batch, you need to make sure that it is correctly stopped/shutdown once it is not needed anymore. From what you shared, you do not seem to use a task executor in your Spring Batch configuration, so you should have a task executor defined somewhere in your application context which is not stopped correctly.
I don't know if I've used the correct title, but I will try to explain the problem below.
I have the following method to check the payment status from external service:
#Autowired
PaymentService paymentService;
public void checkPaymentStatus(Payment p){
// sending
String status = paymentService.getPaymentStatus(p.getId());
}
It works in main thread and there might be hundreds of requests per second, so I decided to use CompletableFuture to run the tasks asynchronously in separate threads.
#Autowired
PaymentService paymentService;
public void checkPaymentStatus(Payment p){
// sending
CompletableFuture<Response> status = paymentService.getPaymentStatus(p.getId());
}
PaymentService.class
#Service
public class PaymentService{
private final RestTemplate restTemplate;
#Async
public CompletableFuture<Response> getPaymentStatus(long id){
Response results = restTemplate.getForObject(url, Response.class);
return CompletableFuture.completedFuture(results);
}
}
Configuration
#Bean
public Executor taskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(2);
executor.setMaxPoolSize(2);
executor.setQueueCapacity(500);
executor.setThreadNamePrefix("payment-service-");
executor.initialize();
return executor;
}
It works perfectly, but I have another task here. Every request must be wait 30 seconds before sending to the external service.
How to solve this problem?
Update:
Update:
In this methods, I might use Thread sleep, but it is not the correct solution as it blocks the Thread for 30 seconds and next task might run after 60 secs, etc.
#Async
public CompletableFuture<Response> getPaymentStatus(long id){
// I might use Thread sleep here, but it is not the correct solution as it blocks the Thread for 30 seconds and next task might run after 60 secs, etc.
Response results = restTemplate.getForObject(url, Response.class);
return CompletableFuture.completedFuture(results);
}
Every 5 minutes, 2 tasks run, one that updates, one that calculates.
The one that updates should always run first, but doesn't have to know anything about the calculator and visa versa.
Is there any way to manage this in a clean way without calling the other task from within.
After advice, this is my solution.
#PostConstruct
public void scheduleRunnableWithTrigger() {
logger.info("init scheduler");
scheduleTasks.schedule(() -> {
if(init.getIsInitialzed()) {
databaseUpdater.updatePrices();
for (ITechnicalAnalysis technicalAnalysis: adviser.getITechnicalAnalysis()) {
try {
CandleStick lastAddedCandleStickByCurrencyPair = candleStickService.getLatestByCurrencyPair(technicalAnalysis.getCurrencyPair());
technicalAnalysis.updateAlgorithms(lastAddedCandleStickByCurrencyPair);
logger.info(technicalAnalysis.getCurrencyPair() + " has candlesticks");
}catch (NullPointerException e) {
logger.warn(technicalAnalysis.getCurrencyPair() + " has no candlesticks");
}
}
adviser.calculateAnalyses();
}
}, periodicTrigger);
}
#Bean
public PeriodicTrigger periodicTrigger() {
PeriodicTrigger periodicTrigger = new PeriodicTrigger(TIME_TO_WAIT_FOR_DATABASE_TO_FILL, TimeUnit.MINUTES);
periodicTrigger.setInitialDelay(1);
return periodicTrigger;
}
According to the documentation. Be default there will be only one thread for scheduler.
So what you can do is you can schedule the tasks one after the other may be a second apart. This way second task will always run after first task.
Even if first task is not completed and schedule time for second task has reached, second task will have to wait for the thread from first task to be released.
You can also explicitly set the thread count for the scheduler with a bean declaration like this
#Bean
public ThreadPoolTaskScheduler threadPoolTaskScheduler() {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
threadPoolTaskScheduler.setPoolSize(THREAD_COUNT);// use 1 for this use case
return threadPoolTaskScheduler;
}
We want to start a job in the background so it doesn't use all resources within the application and can influence the 'normal' tasks of the application. It should start from within the a running Java app and not executed from the command line.
Does anybody know how to start a Spring Batch job as a background task/daemon using Spring scheduling?
After some more investigation I found out you can configure a TaskExecuter for a JobLauncher. You can then use a SimpleAsyncTaskExector and configure it as a deamon and set the thread priority.
#Bean
public JobLauncher jobLauncher(final JobRepository jobRepository, final TaskExecutor taskExecutor) {
final SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
jobLauncher.setTaskExecutor(taskExecutor);
return jobLauncher;
}
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setDaemon(true);
taskExecutor.setThreadPriority(Thread.MIN_PRIORITY);
return taskExecutor;
}
We have multiple spring batch job.But Each of them needs to be started individually.
Is there any way to create a Master Job or any controller in spring which will be responsible for executing all other batch jobs? So that we just have to execute the master job only,and all other started automatically.
I just explained on this question how you can start spring application with all jobs loaded in separate context. We have restart job which is scheduled each 10min and it checks for latest failed execution and attempts to restart couple of more times.
Your use case is pretty much the same, you can define all jobs in separate contexts with own configuration files, in config you can tell spring batch not to run them on startup by setting spring.batch.job.enabled: false and you can write your own launcher which uses JobExplorer to start jobs for you and you can schedule it or something.
Something like:
#Component
#EnableScheduling
public class AllJobLauncher {
#Autowired
JobExplorer jobExplorer;
#Autowired
private JobLauncher jobLauncher;
#Autowired
private JobRegistry jobRegistry;
#Scheduled(cron = "${some.cron:0 0/10 * * * ?}")
public void launchAllJobs() throws JobExecutionException {
final List<String> jobNames = jobExplorer.getJobNames();
for (final String jobName : jobNames) {
final Job job = jobRegistry.getJob(getJobName(organizationId));
final JobParameters jobParameters = new JobParametersBuilder() //build parameters
jobLauncher.run(job, jobParameters);
}
}
Just pay attention that JobLauncher in spring batch is by default sync so launcher will wait until job finishes. If you want to start jobs async you must place this configuration somewhere:
#Bean
public JobLauncher jobLauncher() {
final SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
final SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
jobLauncher.setTaskExecutor(simpleAsyncTaskExecutor);
return jobLauncher;
}