Develop Reference

Implement fault tolerance in Spring Batch

I have following batch job implemented in Spring Batch config:
#Bean
public Job myJob(Step step1, Step step2, Step step3) {
return jobs.get("myJob").start(step1).next(step2).next(step3).build();
}
#Bean
public Step step1(ItemReader<String> myReader,
ItemProcessor<String, String> myProcessor,
ItemWriter<String> myWriter) {
return steps.get("step1").<String, String>chunk(1)
.reader(myReader)
.processor(myProcessor)
.writer(myWriter)
.build();
}
I would like to retry the step1 (also step2 and step3 so forth) on certain exception and rollback the job on any failure (also between the retries). I understand that rollback is not going to be automatic and I am clear what to rollback for each step with writing custom code.
What is the best way to implement this?

Spring framework provides #Retryable and #Recover annotations to retry and to recover when something fails. You can check this article. https://www.baeldung.com/spring-retry

Fault tolerance features in Spring Batch are applied to items in chunk-oriented steps, not to the entire step.
What you can try to do is use a flow with a decider where you restart from step1 if an exception occurs in one of the subsequent steps.

JobRestartException: JobInstance already exists and is not restartable

I'm using Spring Batch with Spring cloud tasks. I have the following configuration in my job:
#Bean
public Job jobDemo(
#Value("${jobname}")String jobName,
JobBuilderFactory jobBuilderFactory,
JobCompletionNotificationListener listener
) {
return jobBuilderFactory.get(jobName)
.incrementer(new RunIdIncrementer())
.preventRestart()
.listener(listener)
.flow(stepA())
.end()
.build();
}
I don't want the restart functionality in the job, that's why I have put .preventRestart(). I want to launch a new job every time the task runs, that is, a new instance of the job to run even when the last time the job has failed or stopped or anything. But I'm getting the following error:
org.springframework.batch.core.repository.JobRestartException: JobInstance already exists and is not restartable
This happens only in the scenarios when the job does not finish sucessfully. Any ideas about the solution?

A JobInstance can only be completed once successfully. When you are starting a Spring Batch job via Spring Boot, Spring Batch handles the logic to increment a JobParameter if there is a JobParametersIncrementer provides (as you have). However...when Spring Batch does that incrementing, it only increments if the previous job was successful. In your case, you want it to always increment. Because of that, you're going to need to write your own CommandLineRunner that always increments the JobParameters.
Spring Boot's JobLauncherCommandLineRunner is where the code to launch a job exists. You'll probably want to extend that and override it's execute method to be sure job parameters are always incremented.

How to make a Spring Batch step depends on previous step?

I am using Spring Batch to read some data from CSV files and put it in a database.
My Batch job must be compound of 2 steps :
Check files (names, extension, content ..)
Read lines from CSV and save them in DB (ItemReader, ItemProcessor,
ItemWriter..)
Step 2 must not be executed if Step 1 generated an error (files are not conform, files doesn't exist ...)
FYI, I am using Spring Batch without XML configuration ! Only annotations :
Here's what my job config class looks like :
#Configuration
#EnableBatchProcessing
public class ProductionOutConfig {
#Autowired
private StepBuilderFactory steps;
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private ProductionOutTasklet productionOutTasklet;
#Autowired
private CheckFilesForProdTasklet checkFilesForProdTasklet;
#Bean
public Job productionOutJob(#Qualifier("productionOut")Step productionOutStep,
#Qualifier("checkFilesForProd") Step checkFilesForProd){
return jobBuilderFactory.get("productionOutJob").start(checkFilesForProd).next(productionOutStep).build();
}
#Bean(name="productionOut")
public Step productionOutStep(){
return steps.get("productionOut").
tasklet(productionOutTasklet)
.build();}
#Bean(name = "checkFilesForProd")
public Step checkFilesForProd(){
return steps.get("checkFilesForProd")
.tasklet(checkFilesForProdTasklet)
.build();
}
}

What you are looking for is already the default behavior of Spring Batch i.e. next step wouldn't be executed if previous step has failed. To mark current step as failed step, you need to throw a run time exception which is not caught.
If exception is not handled, spring batch will mark that step as failed and next step wouldn't be executed. So all you need to do is to throw an exception on your failed scenarios.
For complicated job flows , you might like to use - JobExecutionDecider , Programmatic Flow Decisions

As the documentation specifies you can use the method "on" which starts a transition to a new state if the exit status from the previous state matches the given pattern.
Your code could be similar to something like this :
return jobBuilderFactory.get("productionOutJob")
.start(checkFilesForProd)
.on(ExitStatus.FAILED.getExitCode()).end()
.from(checkFilesForProd)
.on("*")
.to(productionOutStep)
.build();

How to create and launch spring batch jobs at runtime

We have a requirement to carry out data movement from 1 database to other and exploring spring batch for the same. User of our application selects source and target datasource along with the list of tables for which the data needs to be moved.
Need help with following:
The information necessary to build a job comes at runtime from our web application - that includes datasource details and list of table names. We would like to create a new job by sending these details to the job builder module and launch it using JobLauncher. How do we write this job builder module?
We may have multiple users raising data movement requests in parallel, so need a way to create multiple jobs and run them in suitable order.
We have used the Java based configuration to create a job and launch it from a web container. The configuration is as follows
#Bean
public Job loadDataJob(JobCompletionNotificationListener listener) {
RunIdIncrementer inc = new RunIdIncrementer();
inc.setKey(new Date().toString());
JobBuilder builder = jobBuilderFactory.get("loadDataJob")
.incrementer(inc)
.listener(listener);
SimpleJobBuilder simpleBuilder = builder.start(preExecute());
for(String s : getTables()){
simpleBuilder.next(etlTable(s));
}
simpleBuilder.next(postExecute());
return simpleBuilder.build();
}
#Bean
#Scope("prototype")
public Step etlTable(String tableName) {
return stepBuilderFactory.get(tableName)
.<Map<String,Object>, Map<String,Object>> chunk(1000)
.reader(dbDataReader(tableName))
.processor(processor())
.writer(dbDataWriter(tableName))
.build();
}
Currently we have hardcoded the source and target datasource details into respective beans. The getTables() returns a list of tables (hardcoded) for which the data needs to be moved.
RestController that launches the job
#RestController
public class MyController {
#Autowired
JobLauncher jobLauncher;
#Autowired
Job job;
#RequestMapping("/launchjob")
public String handle() throws Exception {
try {
JobParameters jobParameters = new JobParametersBuilder().addLong("time", new Date().getTime()).toJobParameters();
jobLauncher.run(job, jobParameters);
} catch (Exception e) {
}
return "Done";
}
}

Concerning your first question, you definitely have to use JavaConfiguration. Moreover, you shouldn't define your steps as spring beans, if you want to create a job with a dynamic number of steps (for instance a step per table you have to copy).
I've written a couple of answers to questions about how to create jobs dynamically. Have a look at them, they might be helpful
Spring batch execute dynamically generated steps in a tasklet
Spring batch repeat step ending up in never ending loop
Spring Batch - How to generate parallel steps based on params created in a previous step
Spring Batch - Looping a reader/processor/writer step
Edited
Some remarks concerning your second question:
Firstly, you are using a normal JobLauncher and I assume your instantiate the SimpleJobLauncher. This means, you can provide a job with jobparameters, as you have shown in your code above. However, the provided "job" does not have to be a "SpringBean"-instance, so you don't have to Autowire it and therefore, you can use create-methodes as I suggested in the answers to the questions mentioned above.
Secondly, if you create your Job instance for every request dynamically, there is no need to pass the whole configuration as jobparameters, since you can pass the "configuration properties" like datasource and tables to be copied directly as parameters to your "createJob" method. You could even create your DataSource-instances "on the fly", if you don't know all possible datasources in advance.
Thirdly, I would consider every request as a "single run", which cannot be "restarted". Hence, I'd just but some "meta information" into the jobparameters like user, date/time, datasource names (urls) and a list of tables to be copied. I would use this kind of information just as a kind of logging/auditing which requests where issued, but I wouldn't use the jobparameter-instances as controlparameters inside the job itself (again, you can pass the values of these parameters during the construction time of the job and steps by passing them to your create-Methods, so the structure of your job is created according to your parameters and hence, during runtime - when you could access your jobparameters - there is nothing to do based on the jobparameters).
Finally, if a request fails (meaning the jobs exits with an error) simply a new request has to be executed in order to retry, but this request would be a complete new request and not a restart of an already executed job launch (since I would add the request time to my jobparameters, every launch would be a unique launch).
Edited 2:
Not creating the Job as a Bean doesn't mean to not use Autowiring. Here is an example, aus I would structure my Beans.
#Component
#EnableBatchProcessing
#Import() // list with imports as neede
public class JobCreatorComponent {
#Autowire
private StepBuilderFactory stepBuilder;
#Autowire
private JobBuilderFactory jobBuilder;
public Job createJob(all the parameters you need) {
return jobBuilder.get(). ....
}
}
#RestController
#Import(JobCreatorComponent.class)
public class MyController {
#Autowired
JobLauncher jobLauncher;
#Autowired
JobCreatorComponent jobCreator;
#RequestMapping("/launchjob")
public String handle() throws Exception {
try {
Job job = jobCreator.createJob(... params ...);
JobParameters jobParameters = new JobParametersBuilder().addLong("time", new Date().getTime()).toJobParameters();
jobLauncher.run(job, jobParameters);
} catch (Exception e) {
}
return "Done";
}
}

by using #JobScope on itemreader no need to do things manually at run time just have to annoted your respective reader with #Jobscope, on each interaction with controller you will get fresh record processing.
This is type of job on demand where you can execute the job for goals like do the db migration or get the specific reporting like that.

Running a spring batch job multiple times

I have created a spring batch job with spring boot using below tutorial:
https://spring.io/guides/gs/batch-processing/
The job is reading a file and writing to a database as expected.
However, now I have a use case to run this job multiple times.
I have an ArrayList of parameters.
What changes should I do to the job so that I can run the job the number of times the size of my ArrayList ?

You can kickstart your batch job manually like this
#Component
Class Someclass{
...............
#Autowired
private JobLauncher jobLauncher;
#Autowired
private Job job;
public void someFunction(){
jobLauncher.run(job, new JobParameters());
}
}
Only thing is you cannot restart a batch job if it is already completed, It throws an error saying the status is COMPLETED. For this to work you have to set allowStartIfComplete property to true. This has to be done in your batch step configuration, something like this
stepBuilderFactory.get("step1")
.<Person, Person> chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.allowStartIfComplete(true)
.build();