I am having one job in the spring batch application which got successfully completed but after completion, its thread goes to waiting state. As more instances of job get executed, number of threads of tomcat keeps on increasing.
Here is the Job Configuration :
#Bean
#Scope("singleton")
#Qualifier(value = "job1")
public Job job() {
return jobBuilderFactory.get("job1")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.preventRestart()
.build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<Buylead, Buylead>chunk(10000)
.reader(itemReader())
.writer(itemWriter())
.build();
}
public JdbcCursorItemReader<Buylead> itemReader() {
JdbcCursorItemReader<Buylead> reader = new JdbcCursorItemReader<>();
reader.setDataSource(dataSource);
reader.setSql(" --- WHatever Query ----");
reader.setRowMapper(new MapperBL());
return reader;
}
public ItemWriter<Buylead> itemWriter(){
FlatFileItemWriter<Buylead> itemWriter = new FlatFileItemWriter() ;
itemWriter.setResource(new FileSystemResource("output/buyleadSolrDoc.xls"));
itemWriter.setLineAggregator(new DelimitedLineAggregator<Buylead>() { {
setFieldExtractor(new BeanWrapperFieldExtractor<Buylead>() { {
setNames(new String[] {*************** }); } }); } });
itemWriter.setShouldDeleteIfEmpty(true);
return itemWriter;
}
And data Source is kept in another configuration class.
And here is the Main class
#SpringBootApplication
#EnableBatchProcessing
public class Application extends DefaultBatchConfigurer{
public static void main(String[] args) {
SpringApplication.run(SearchApplication.class, args);
}
#Override
public void setDataSource(DataSource dataSource) {
// override to do not set datasource even if a datasource exist.
// initialize will use a Map based JobRepository (instead of database)
}
}
The job is getting executed successfully. But its thread goes to wait state.
And here is the portion of ThreadDump for the application
http-nio-8080-exec-2" #22 daemon prio=5 os_prio=0 cpu=0.11ms elapsed=29.43s tid=0x00007fcb6d6ddd60 nid=0xc724 waiting on condition [0x00007fcae3ffe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base#15.0.1/Native Method)
- parking to wait for <0x0000000713c00710> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base#15.0.1/LockSupport.java:341)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base#15.0.1/AbstractQueuedSynchronizer.java:505)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base#15.0.1/ForkJoinPool.java:3137)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base#15.0.1/AbstractQueuedSynchronizer.java:1614)
at java.util.concurrent.LinkedBlockingQueue.take(java.base#15.0.1/LinkedBlockingQueue.java:435)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base#15.0.1/ThreadPoolExecutor.java:1056)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base#15.0.1/ThreadPoolExecutor.java:1116)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base#15.0.1/ThreadPoolExecutor.java:630)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(java.base#15.0.1/Thread.java:832)*
Please suggest the way to close the thread which is in Park state.
Spring Batch does not create or manage threads directly. It delegates that to a TaskExecutor implementation in different places:
The JobLauncher delegates to a task executor to launch jobs
The StepBuilder delegates to a task executor to create multi-threaded steps
The FlowBuilder delegates to a task executor to create split flows and run steps in parallel
The AsyncItemProcessor delegates to a task executor to process items asynchronously in separate threads
The TaskExecutorPartitionHandler delegates to a task executor to handle partitions in different threads
etc
In all these cases, threads life cycle is managed by the underlying TaskExecutor implementation. If you use a task executor with Spring Batch, you need to make sure that it is correctly stopped/shutdown once it is not needed anymore. From what you shared, you do not seem to use a task executor in your Spring Batch configuration, so you should have a task executor defined somewhere in your application context which is not stopped correctly.
Related
I'm using rabbitmq receiver. it gets multiple requests at the same time so I'm creating threads to handle multiple requests concurrently.
problem is either I create newSingleThreadExecutor or newFixedThreadPool they remain running
executor.shutdown(); or executor.shutdownNow(); will not close threads below is sample
public void receiver(String message) {
ExecutorService executor = Executors.newFixedThreadPool(10);
:
executor.execute(c);
:
executor.shutdown();
try {
executor.awaitTermination(30, TimeUnit.NANOSECONDS);
executor.shutdownNow();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
below is my rabbitmq config
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(queueName);
container.setMessageListener(listenerAdapter);
return container;
}
#Bean
MessageListenerAdapter listenerAdapter(receiver recv) {
return new MessageListenerAdapter(recv, "receiver");
}
and threads remain in like
Thread[pool-1-thread-"number"]
threads are created as many requests i get in the receiver and never shuts down.
how to avoid this condition or is it best to remove the multi-thread concept entirely ??
thanks in advance
You should not use an executor within your listener in the first place or you will risk message loss.
Increase the container concurrentConsumers and the framework will manage concurrency for you.
I've the problem, that I want to create a scheduled task during runtime. The scheduled task should be triggered with a fixed rate. But now I'm having the problem that the manual setup schedules are not triggered in an async way.
The main problem is, that we do not have any fix point were we can start the scheduler. It should get created when I read a specific value (1) and gets destroyed when the value changes back (0). Otherwise we could use the annotation configuration described in test 1 below.
What I have tried so far:
1. Schedule with #Scheduled(fixedRate = 500L) and #Async
Code
#Async
#Scheduled(fixedRate = 500L)
public void annotationTest() {
UUID id = UUID.randomUUID();
log.warn("Hello from Thread {} going to sleep", id);
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
log.warn("Finished Thread {}", id);
}
Also having the #EnableAsync and #EnableScheduling annotations on class level.
Result
09:56:24.855 [task-5] : Hello from Thread 3b5514b2-3b80-4641-bf12-2cd320c4b6e5 going to sleep
09:56:25.355 [task-6] : Hello from Thread e98514a7-e193-422b-9569-f7635deb33f8 going to sleep
09:56:25.356 [task-4] : Finished Thread d86f5f24-bffb-4ddd-93fe-2334ed48cf91
09:56:25.854 [task-7] : Hello from Thread cfc2ab03-4e7e-4a4a-aa08-41d696cb6df7 going to sleep
09:56:25.855 [task-5] : Finished Thread 3b5514b2-3b80-4641-bf12-2cd320c4b6e5
09:56:26.355 [task-6] : Finished Thread e98514a7-e193-422b-9569-f7635deb33f8
Comment
This works as expected, but we are not able to use it, because we have to create the scheduler during runtime and destroy it after a specific time/input.
2. Setting up a ScheduledTaskRegistrar
Code
//#Configuration
#Bean
public ScheduledTaskRegistrar scheduledTaskRegistrar() {
ScheduledTaskRegistrar scheduledTaskRegistrar = new ScheduledTaskRegistrar();
scheduledTaskRegistrar.setScheduler(threadPoolTaskScheduler());
return scheduledTaskRegistrar;
}
#Bean
public TaskScheduler threadPoolTaskScheduler() {
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setPoolSize(20);
return scheduler;
}
//#Component
public void printMessages() {
scheduledTaskRegistrar.scheduleFixedRateTask(new FixedRateTask(new OwnRunnable(), 500L, 0L));
}
The OwnRunnable will also sleep 1 second and print the finish Text afterwards
Result
10:13:56.983 [TaskScheduler-1] : Finished Thread 73f70de9-35d9-47f0-801b-fb2857ab1c34
10:13:56.984 [TaskScheduler-3] : Hello from Thread 7ab16380-8dba-49e1-bf0d-de8235f81195 going to sleep
10:13:57.984 [TaskScheduler-3] : Finished Thread 7ab16380-8dba-49e1-bf0d-de8235f81195
10:13:57.984 [TaskScheduler-2] : Hello from Thread cc152d2e-f93b-4770-ac55-853a4dd6be97 going to sleep
10:13:58.985 [TaskScheduler-2] : Finished Thread cc152d2e-f93b-4770-ac55-853a4dd6be97
10:13:58.985 [TaskScheduler-4] : Hello from Thread 8d4510a4-773d-49f3-b51b-e58e425b0b68 going to sleep
Comment
As we can see the tasks run in a synchronous way and will not fit to our requirement.
3. Other tests
All other tests are similar to the test described in 2 but will use some other configurations of the ScheduledTaskRegistrar. The results are the same as in test 2.
ConcurrentTaskScheduler instead of ThreadPoolTaskScheduler
ConcurrentTaskScheduler with SimpleAsyncTaskExecutor as ConcurrentExecutor
ConcurrentTaskScheduler with ThreadPoolTaskExecutor as ConcurrentExecutor
Question(s)
How can I use the configuration described in test 2 but get the result of test 1? Is there a way to use the #Async annotation with solution described in test 2? Or does anyone have a better/ another solution for my problem?
Yes, it is possible. Assume that your class that implemented SchedulingConfigurer has a method, doMyJob(). You can annotate that method with Async and use the reference in FixedRateTask. Also notice the class level annotation
#Configuration
#EnableAsync
public class MyJobConfig implements SchedulingConfigurer {
#Override
public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
taskRegistrar.scheduleFixedRateTask(new FixedRateTask(this::doMyJob, 500L, 0L));
}
#Async
public void doMyJob() {
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
Hope it helps
EDIT
I provided the code without testing. Recently when I tried to recreate this scenario, I noticed that if doMyJob is within SchedulingConfigurer, it will not be truly async (if delay is 5seconds and job takes 10seconds, next job runs only after 10seconds). But moving the method to a service class helped.
I have an web app(with Spring/Spring boot) running on tomcat 7.There are some ExecutorService defined like:
public static final ExecutorService TEST_SERVICE = new ThreadPoolExecutor(10, 100, 60L,
TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(1000), new ThreadPoolExecutor.CallerRunsPolicy());
The tasks are important and must complete properly. I catch the exceptions and save them to db for retry, like this:
try {
ThreadPoolHolder.TEST_SERVICE.submit(new Runnable() {
#Override
public void run() {
try {
boolean isSuccess = false;
int tryCount = 0;
while (++tryCount < CAS_COUNT_LIMIT) {
isSuccess = doWork(param);
if (isSuccess) {
break;
}
Thread.sleep(1000);
}
if (!isSuccess) {
saveFail(param);
}
} catch (Exception e) {
log.error("test error! param : {}", param, e);
saveFail(param);
}
}
});
} catch (Exception e) {
log.error("test error! param:{}", param, e);
saveFail(param);
}
So, when tomcat shutting down, what will happen to the threads of the pool(running or waiting in the queue)? how to make sure that all the tasks either completed properly before shutdown or saved to db for retry?
Tomcat has built in Thread Leak detection, so you should get an error when the application is undeployed. As a developer it is your responsibility to link any object you create to the web applications lifecycle, this means You should never ever have static state which are not constants
If you are using Spring Boot, your Spring context is already linked to the applications lifecycle, so the best way is to create you executor as a Spring bean, and let Spring shut it down when the application stops. Here is an example you can put in any #Configuration class.
#Bean(destroyMethod = "shutdownNow", name = "MyExecutorService")
public ThreadPoolExecutor executor() {
ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(10, 100, 60L,
TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(1000),
new ThreadPoolExecutor.CallerRunsPolicy());
return threadPoolExecutor;
}
As you can see the #Bean annotation allows you to specify a destroy method which will be executed when the Spring context is closed. In addition I have added the name property, this is because Spring typically creates a number of ExecutorServices for stuff like async web processing. When you need to use the executor, just Autowire it as any other spring bean.
#Autowired
#Qualifier(value = "MyExecutorService")
ThreadPoolExecutor executor;
Remember static is EVIL, you should only use static for constants, and potentially immutable obbjects.
EDIT
If you need to block the Tomcats shutdown procedure until the tasks have been processed, you need to wrap the Executor in a Component for more control, like this.
#Component
public class ExecutorWrapper implements DisposableBean {
private final ThreadPoolExecutor threadPoolExecutor;
public ExecutorWrapper() {
threadPoolExecutor = new ThreadPoolExecutor(10, 100, 60L,
TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(1000), new ThreadPoolExecutor.CallerRunsPolicy());
}
public <T> Future<T> submit(Callable<T> task) {
return threadPoolExecutor.submit(task);
}
public void submit(Runnable runnable) {
threadPoolExecutor.submit(runnable);
}
#Override
public void destroy() throws Exception {
threadPoolExecutor.shutdown();
boolean terminated = threadPoolExecutor.awaitTermination(1, TimeUnit.MINUTES);
if (!terminated) {
List<Runnable> runnables = threadPoolExecutor.shutdownNow();
// log the runnables that were not executed
}
}
}
With this code you call shutdown first so no new tasks can be submitted, then wait some time for the executor finish the current task and queue. If it does not finish in time you call shutdownNow to interrupt the running task, and get the list of unprocessed tasks.
Note: DisposableBean does the trick, but the best solution is actually to implement the SmartLifecycle interface. You have to implement a few more methods, but you get greater control, because no threads are started until all bean have been instantiated and the entire bean hierarchy is wired together, it even allows you to specify in which orders components should be started.
Tomcat as any Java application will not end untill all non-daeon threads will end. ThreadPoolExecutor in above example uses default thread factory and will create non-daemon threads.
I Have a Spring rest controller which is calling an asynchronous method using Spring's #Async methodology and return immediately an http 202 code (Accepted) to the client.(The asynchronous job is heavy and could lead to a timeout).
So actually, at the end of the asynchronous task, i'm sending an email to the client telling him the status of his request.
Everything works just fine but I'm asking myself what can I do if my server/jvm crashes or if it is shut down? My client would receive a 202 code and will never receive a the status email.
Is there a way to synchronize (in real time) a ThreadPoolTaskExecutor in a database or even in a file to let the server recover at startup without managing this on my own with complex rules and evolution status?
Here is my Executor configuration
#Configuration
#EnableAsync
public class AsyncConfig implements AsyncConfigurer {
#Override
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(4);
executor.setMaxPoolSize(8);
executor.setQueueCapacity(100);
executor.setThreadNamePrefix("asyncTaskExecutor-");
executor.setAwaitTerminationSeconds(120);
executor.setKeepAliveSeconds(30);
executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
executor.initialize();
return executor;
}
#Override
public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
return new SimpleAsyncUncaughtExceptionHandler();
}
}
The controller that launch the async task
#RequestMapping(value = "/testAsync", method = RequestMethod.GET)
public void testAsync() throws InterruptedException{
businessService.doHeavyThings();
}
The async method called:
#Async
public void doHeavyThings() throws InterruptedException {
LOGGER.error("Start doHeavyThings with configured executor - " + Thread.currentThread().getName() + " at " + new Date());
Thread.sleep(5000L);
LOGGER.error("Stop doHeavyThings with configured executor - " + Thread.currentThread().getName() + " at " + new Date());
}
}
Thx
For a web server shutdown the application lifecycle in a java web application will notifiy a ServletContextListener. If you provide an implementation of a ServletContextListener you can put your 'what is already processed' logic in the contextDestroyed method.
When the web server or the application is started again the listener can be used to recover an re-process the unprocessed items of your job using the contextInitialized method.
Another option would be using Spring destruction callbacks and place the logic here.
HTH
My aim is to create a queue system where I can specify a maximum amount of concurrent jobs for each group, i.e. for group A maximum 3 jobs should run at the same time, for group B max Y jobs etc. The jobs can be executed both on cron schedule and only once with SimpleTrigger, therefor I can't check the queue when scheduling the job, I have to check it before or during execution. I'm implementing a joblistener and I'm trying to prevent execution in the jobToBeExecuted() method. I've tried scheduler.interrupt() but it doesn't work when the job hasn't started yet. scheduler.deletejob() and scheduler.unschedule() didn't stop it from executing either.
Any ideas?
public class JobQueueListener implements JobListener {
#Override
public void jobToBeExecuted(JobExecutionContext context) {
JobKey currentJobKey = context.getJobDetail().getKey();
JobDetail jobDetail = context.getJobDetail();
Scheduler scheduler = context.getScheduler();
if (shouldBePutInQueue(currentJobKey)) {
/// Prevent execution and put in queue here, but how?
}
}
#Override
public void jobWasExecuted(JobExecutionContext context, JobExecutionException jobException) {
//Check queue and execute next in queue
}
}
Can you look at TriggerListener
You should implement TriggerListener and have your abort logic within "vetoJobExecution" method.
boolean vetoJobExecution(Trigger trigger,
JobExecutionContext context)
Its Called by the Scheduler when a Trigger has fired, and it's associated JobDetail is about to be executed. If the implementation vetos the execution (via returning true), the job's execute method will not be called.