Develop Reference

How can I call a method of the writer class when my Spring Batch Job complete?

I am working on a Spring Batch application and I have the following problem.
I have the definition of a single step Job containing this step:
#Bean
public Step readNotaryListStep(){
return steps.get("readNotaryListStep").
<Integer,Integer>chunk(1)
.reader(serviceItemReader())
.processor(new NotaryDetailsEnrichProcessor(notaryService))
.writer(new NotaryWriter(wpService))
.build();
}
#Bean(name="updateNotaryListInfoJob")
public Job updateNotaryListInfoJob(){
return jobs.get("updateNotaryListInfoJob")
.incrementer(new RunIdIncrementer())
.start(readNotaryListStep())
.build();
}
It works fine. Basically it is reading from a reader, the output is processed and finnally it is written by the new NotaryWriter(wpService) that I create for the writer.
Ok...the following need has now emerged: after that the Job complete I need to call a method called resetCounter() defined into my NotaryWriter class (the class set for the writer section of my step).
Is it possible to implement a behavior like this? In case how can I implement it?

You can implement a JobExecutionListener or even StepExecutionListener which allow you to add a callback when a job and a step is completed. Both of them can be configured by the listener() on the JobBuilder and StepBuilder .
To allow the JobExecutionListener to call a method on the same instance of NotaryWriter that is configured in the step , you have to make NotaryWriter as a spring bean rather than create it manually , and make sure both the JobExecutionListener and the step refer to it.
Something likes :
#Bean
public NotaryWriter notaryWriter(){
return new NotaryWriter(wpService);
}
And inject it into both JobExecutionListener and the Step :
#Component
public MyJobExecutionListener implements JobExecutionListener {
#Autowired
private NotaryWriter notaryWriter;
public void beforeJob(JobExecution jobExecution){
}
public void afterJob(JobExecution jobExecution){
notaryWriter.resetCounter();
}
}
#Bean
public Step readNotaryListStep(NotaryWriter notaryWriter){
return steps.get("readNotaryListStep").
.........
.writer(notaryWriter)
.build();
}
I just show you the general idea . You have to consider the scope of the JobExecutionListener and the NotaryWriter based on the things you want to do. It supports job scope , step scope or just application scope out of the box.
Because it seems to me that you actually want each step has their own instance of NotaryWriter such that its counter will not mess up with each other. If yes , you could simply define it as the #StepScope and do not need any listener :
#Bean
#StepScope
public NotaryWriter notaryWriter(){
return new NotaryWriter(wpService);
}
#Bean
public Step readNotaryListStep(NotaryWriter notaryWriter){
return steps.get("readNotaryListStep").
.........
.writer(notaryWriter)
.build();
}

Batch Tasklet to read from database with select query

How can I create a tasklet class to make a custom select query from DB and pass the data to the next tasklet? I have to use tasklet (no jdbcReader or any reader)
Code Sample:
public class Taskletreader implements Tasklet, StepExecutionListener{
private final Logger logger = LoggerFactory.getLogger(Taskletreader.class);
#Autowired
private DataSource dataSource;
private List<FichierEclate> FichierEclates;
private String query="select * from FicherEclate where .......some conditions...."
#Override
public void beforeStep(StepExecution stepExecution) {
FichierEclates = new ArrayList<FichierEclate>();
logger.debug("Reader initialized.");
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
new JdbcTemplate(dataSource)
.execute(query);
return RepeatStatus.FINISHED;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
// fu.closeReader();
stepExecution
.getJobExecution()
.getExecutionContext()
.put("FichierEclates", this.FichierEclates);
logger.debug("Reader ended.");
return ExitStatus.COMPLETED;
}
}
Can't understand where is the result of the select and how to pass it to next tasklet for processing ?

Can't understand where is the result of the select
If you want to consume the result of the query, you can use the query method on JdbcTemplate:
List<FichierEclate> fichierEclates = jdbcTemplate.query(query, new BeanPropertyRowMapper<>(FichierEclate.class));
This method accepts a RowMapper that maps each row from the database to an instance of your domain object. The sample uses BeanPropertyRowMapper, but you can provide a custom mapper if needed.
how to pass it to next tasklet for processing ?
You can use the execution context for that as you did. However, you should not be passing the entire items list. The execution context is persisted and it is not recommended to pass a lot of data between steps like this. A list of item IDs could be fine, but the list of entire items objects is not a good idea. Please refer to the Passing Data to Future Steps section of the reference documentation for more details.
That said, I really recommend using a chunk-oriented tasklet for that. I know you said I have to use tasklet (no jdbcReader or any reader), but I don't understand this constraint.

How can setConsumerRebalanceListener get the calling ConsumerAwareRebalanceListener instance?

Maybe this is a naive question but it somhow stuck me there for some time. Please bear with me.
I have a class DataConsumer.java that implements ConsumerAwareRebalanceListener:
#Component
public class DataConsumer implements ConsumerAwareRebalanceListener {
#Override
public void onPartitionsAssigned(Consumer<?, ?> consumer, Collection<TopicPartition> partitions) {
// seek offsets based on a given timestamp
}
#KafkaListener(topics = "dataTopic", containerFactory = "kafkaListenerContainerFactory")
receive(ConsumerRecord payload) {}
}
So in order for onPartitionsAssigned to work, i need to call setConsumerRebalanceListener in the kafkaListenerContainerFactory method which is defined in another class like this:
#Bean
public ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> factory = new ConcurrentKafkaListenerContainerFactory();
factory.getContainerProperties().setConsumerRebalanceListener(____________);
// rest part omitted
}
My question is about this ____________ part above. What shall i put there?
In my understanding, method kafkaListenerContainerFactory is called when we initialize the #KafkaListener container in DataConsumer class, so meaning there is already an existing DataConsumer instance to hold the #kafkaLister. How can i pass that already existing DataConsumer instance to setConsumerRebalanceListener function?
All the sample code snippets I can search out are like below:
setConsumerRebalanceListener(new ConsumerRebalanceListener() {
//override the functions
})
But isn't this creating a new instance? If I put new DataConsumer() it will lose some status in the existing instance (e.g. the timestamp to seek offsets) so this can't work.

You can declare DataConsumer as a #Bean (instead of using #Component) then you can inject your bean there.
However, this is the wrong mechanism to use in this case.
Implement ConsumerSeekAware instead - the container will automatically detect that your listener implements that and will call its onPartitionsAssigned.
See Seeking to a Specific Offset in the documentation.

Maintain pre-job object through Spring Batch job steps

For my batch application, I have a handful of steps I need to take prior to the execution of the Spring Batch job. For instance, I need to do a specific query and store data in a property - a List with a complex type (List<ComplexType>) - so that it can be used and manipulated throughout the Spring Batch job (primarily in the ItemReader).
I've tried autowiring in my list and accessing it in the step, but I can't access the data in the list that way. I get a null ComplexType in my List, no matter what values have been added to my autowired List property prior to the job.
I have also tried passing data using ExecutionContext, but I don't think this is supposed to work outside the Spring Batch job execution.
What I want to know, is what is the best way to populate an item prior to executing a Spring Batch job and maintain that object throughout the lifecycle of the job.
If the best way is one of the previous attempts I've made, any guidance on common mistakes with those approaches are appreciated, thanks.

Thanks Luca Basso Ricci for the JobExecutionListener pointer. I ended up creating my own StepExecutionListener where my pre-step processing would happen.
I followed this example from Mkyong which goes over different types of Spring Batch Listeners.
I created a custom listener like this one in the Java code:
public class CustomStepListener implements StepExecutionListener {
#Autowired
private CustomObject customObject;
#Override
public void beforeStep(StepExecution stepExecution) {
// initialize customObject and do other pre set setup
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return null;
}
And I initialized the autowired CustomObject class here. The CustomObject class is a custom object that simply contained my List of type ComplexType.
#Component
public class CustomObject {
private List<ComplexType> customObjectList;
public List<ComplexType> getCustomObjectList() {
return customObjectList;
}
public void setCustomObjectList(List<ComplexType> customObjectList) {
this.customObjectList= customObjectList;
}
}
Finally, in my job configuration 'batch-job-context.xml' I added my new listener:
<!-- ... -->
<beans:bean id="customStepListener"
class="com.robotsquidward.CustomStepListener"/>
<job id="robotsquidwardJob"
job-repository="jobRepository"
incrementer="runIdIncrementer">
<step id="robotsquidwardStep">
<tasklet task-executor="taskExecutor" throttle-limit="1">
<chunk
reader="robotsquidwardReader"
processor="robotsquidwardProcessor"
writer="robotsquidwardWriter"
commit-interval="1"/>
</tasklet>
<listeners>
<listener ref="customStepListener"/>
</listeners>
</step>
</job>
When I followed these steps I was able to initialize my ComplexObject List within the beforeJob function and access the values of the ComplexObject List within my job's Reader class:
#Component
#Scope(value = "step")
public class RobotsquidwardReader implements ItemReader<ComplexType> {
#Autowired
private CustomObject customObject;
#Override
public ComplexType read() throws Exception, UnexpectedInputException,
ParseException, NonTransientResourceException {
if(customObject.getCustomObjectList() != null) {
return customObject.getCustomObjectList.remove(0);
} else {
return null;
}
}
}
Easy as that. All it took is two new classes, a config change, and a major headache :)

You can do this :)
try to do this in job parameters incrementer trick :
<j:job id="Your_job" incrementer="incrementerBean">
and
<bean id="incrementerBean" class="com.whatever.IncrementerClass"/>
incrementer class :
class IncrementerClass implements JobParametersIncrementer {
#Override
JobParameters getNext(JobParameters parameters) {
Map<String, JobParameter> map = new HashMap<String, JobParameter>(
parameters.getParameters());
...
//you can put here your list, if it can be :
// Domain representation of a parameter to a batch job. Only the following types
// can be parameters: String, Long, Date, and Double.
//make some query
List<String> listStrings = Query.getYourQuery();
//Join your query into string to have something like this below
map.put("listOfSomething", new JobParameter("abc, abc, abc"));
...
return new JobParameters(map);
}
}
And thats all,
then you can use this parameter for example in some processing bean :
#Value("#{jobParameters['listOfSomething']}")
private String yourList
You can build your list from string, and thats all :)
good luck

Best way to access StepExecution/JobExecution in ItemReader/Processor/Writer

I've been using SpringBatch for a few months now..
I used to store execution-related variables(like page count, item count, current position of a batch and so on) into Beans. Then those beans are mounted onto ItemReader, ItemProcessor, ItemWriter by using setVar(), getVar()-setters and getters. Also those beans are shared among threads with manual synchronization.
But now I found out this could be a wrong way of doing batch jobs. Beans mounted to ItemReaders can't be persistent in JobRepository and therefore unable to record states for stopping and restarting of a Job. So I still need to go back and use StepExecution/JobExecution.
Those examples I found online are all based on either XML config, or the worse SpEL autowired to a setter method..
I use purely Java Config..Is there a Java config or Java code-oriented way of accessing StepExecution? What's the best practice for accessing various sorts of ExecutionContext?

To get access to the StepExecution and the JobExecution your ItemReader, ItemProcessor, or ItemWriter will have to implement the StepExecutionListener.
For instance:
public class MyCustomItemWriter implements ItemWriter<Object>, StepExecutionListener {
private StepExecution stepExecution;
#Override
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return stepExecution.getExitStatus();
}
#Override
public void write(List<? extends Object> list) throws Exception {
if (null == list || list.isEmpty()) {
throw new Exception("Cannot write null or empty list");
}
ExecutionContext stepExecContext = this.stepExecution.getExecutionContext()
ExecutionContext jobExecContext = this.stepExecution.getJobExecution().getExecutionContext();
// TODO: Write your code here
}
}

To get access to StepExecution, JobExecution, you can use methods with annotations from package org.springframework.batch.core.annotation or implementing iterfaces like JobExecutionListener, StepExecutionListener depending on your needs