I've been using SpringBatch for a few months now..
I used to store execution-related variables(like page count, item count, current position of a batch and so on) into Beans. Then those beans are mounted onto ItemReader, ItemProcessor, ItemWriter by using setVar(), getVar()-setters and getters. Also those beans are shared among threads with manual synchronization.
But now I found out this could be a wrong way of doing batch jobs. Beans mounted to ItemReaders can't be persistent in JobRepository and therefore unable to record states for stopping and restarting of a Job. So I still need to go back and use StepExecution/JobExecution.
Those examples I found online are all based on either XML config, or the worse SpEL autowired to a setter method..
I use purely Java Config..Is there a Java config or Java code-oriented way of accessing StepExecution? What's the best practice for accessing various sorts of ExecutionContext?
To get access to the StepExecution and the JobExecution your ItemReader, ItemProcessor, or ItemWriter will have to implement the StepExecutionListener.
For instance:
public class MyCustomItemWriter implements ItemWriter<Object>, StepExecutionListener {
private StepExecution stepExecution;
#Override
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return stepExecution.getExitStatus();
}
#Override
public void write(List<? extends Object> list) throws Exception {
if (null == list || list.isEmpty()) {
throw new Exception("Cannot write null or empty list");
}
ExecutionContext stepExecContext = this.stepExecution.getExecutionContext()
ExecutionContext jobExecContext = this.stepExecution.getJobExecution().getExecutionContext();
// TODO: Write your code here
}
}
To get access to StepExecution, JobExecution, you can use methods with annotations from package org.springframework.batch.core.annotation or implementing iterfaces like JobExecutionListener, StepExecutionListener depending on your needs
Related
I am thinking what would be best solution for following case. Suppose we have at start CRUD app - using Spring Boot. I would like to add read only state for this application - which allows only data read and blocks create, update, delete data operations for admin role. I think about adding aspect (#Aspect) which checks current app state (which is saved in db) and starts if create, update, update operations are invoked. If app is in read-only state - exception will be thrown (handled by #ControllerAdvice)
I wonder if adding aspect is the best option - I dont want modify existing code. Whats your take on that? Moreover - how would you write integration tests for #aspect - testing if aspect starts properly? How could be aspects tested for this case? What are good practises for testing #aspects (integration tests #springboottest)
This honestly seems like an inconvenient way of doing this. Why not just add an Interceptor that checks for this? I did something similar before
#Component
#RequiredArgsConstructor
public class ReadOnlyModeInterceptor implements HandlerInterceptor {
private final ServerProperties serverProperties;
#Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
if (serverProperties.isReadOnlyMode()) {
String method = request.getMethod();
boolean isReadOnlyMethod = "GET".equals(method) || "OPTIONS".equals(method);
String servletPath = request.getServletPath();
boolean isReadOnlyPath = isReadOnlyPath(servletPath);
if (!isReadOnlyMethod && isReadOnlyPath) {
throw new ServiceUnavailableException("Server is in read-only mode.");
}
}
return true;
}
private boolean isReadOnlyPath(String servletPath) {
if (serverProperties.isFullyReadOnly()) {
return true; // wildcard option, entire server is read-only
}
return serverProperties.getReadOnlyPaths().stream().anyMatch(servletPath::contains);
}
}
You also need to register it
#RequiredArgsConstructor
#Configuration
public class WebMvcConfig implements WebMvcConfigurer {
private final ReadOnlyModeInterceptor readOnlyModeInterceptor;
#Override
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(readOnlyModeInterceptor).order(0);
}
}
How can I create a tasklet class to make a custom select query from DB and pass the data to the next tasklet? I have to use tasklet (no jdbcReader or any reader)
Code Sample:
public class Taskletreader implements Tasklet, StepExecutionListener{
private final Logger logger = LoggerFactory.getLogger(Taskletreader.class);
#Autowired
private DataSource dataSource;
private List<FichierEclate> FichierEclates;
private String query="select * from FicherEclate where .......some conditions...."
#Override
public void beforeStep(StepExecution stepExecution) {
FichierEclates = new ArrayList<FichierEclate>();
logger.debug("Reader initialized.");
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
new JdbcTemplate(dataSource)
.execute(query);
return RepeatStatus.FINISHED;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
// fu.closeReader();
stepExecution
.getJobExecution()
.getExecutionContext()
.put("FichierEclates", this.FichierEclates);
logger.debug("Reader ended.");
return ExitStatus.COMPLETED;
}
}
Can't understand where is the result of the select and how to pass it to next tasklet for processing ?
Can't understand where is the result of the select
If you want to consume the result of the query, you can use the query method on JdbcTemplate:
List<FichierEclate> fichierEclates = jdbcTemplate.query(query, new BeanPropertyRowMapper<>(FichierEclate.class));
This method accepts a RowMapper that maps each row from the database to an instance of your domain object. The sample uses BeanPropertyRowMapper, but you can provide a custom mapper if needed.
how to pass it to next tasklet for processing ?
You can use the execution context for that as you did. However, you should not be passing the entire items list. The execution context is persisted and it is not recommended to pass a lot of data between steps like this. A list of item IDs could be fine, but the list of entire items objects is not a good idea. Please refer to the Passing Data to Future Steps section of the reference documentation for more details.
That said, I really recommend using a chunk-oriented tasklet for that. I know you said I have to use tasklet (no jdbcReader or any reader), but I don't understand this constraint.
I have a job with a multi-threaded chunk-oriented step and I need to count how many written items satisfy some business rules. (PS: For legacy reasons, I'm using Spring Batch 3.0.x)
I have to keep in mind that if a rollback happens, then previous already counted items within the same transaction (i.e. same chunk) must be ignored. So I can't just update JobExecutionContext straight from Writer, rather I update an attribute in ChunkContext and use a CustomChunkListener to only update the JobExecutionContext after the chunk succeeds (as you can see in code below).
Before making the step multi-threaded, I had following implementation that worked as expected (I simplified the code as much as I could to focus on the issue):
CustomItemWriter
public class CustomItemWriter implements ItemWriter<String[]> {
private ChunkContext chunkContext;
#Override
public void write(List<? extends String[]> items) throws Exception {
for (String[] item : items) {
((AtomicLong)this.chunkContext.getAttribute("chunkCounter")).incrementAndGet();
}
}
#BeforeChunk
private void beforeChunk(ChunkContext chunkContext) {
this.chunkContext = chunkContext;
}
}
CustomChunkListener
public class CustomChunkListener extends ChunkListenerSupport {
#Override
public void beforeChunk(ChunkContext context) {
context.setAttribute("chunkCounter", new AtomicLong());
}
#Override
public void afterChunk(ChunkContext context) {
((AtomicLong)context.getStepContext().getJobExecutionContext().get("jobCounter")).addAndGet(((AtomicLong)context.getAttribute("chunkCounter")).get());
}
}
CustomJobListener
public class CustomJobListener extends JobExecutionListenerSupport {
#Override
public void beforeJob(JobExecution jobExecution) {
jobExecution.getExecutionContext().put("jobCounter", new AtomicLong());
}
#Override
public void afterJob(JobExecution jobExecution) {
System.out.println("jobCounter = " + jobExecution.getExecutionContext().get("jobCounter"));
}
}
However, when I configured the job to run the step in a multi-threaded fashion, the counter wasn't being updated properly and I know that it was because of the way I was getting access to the ChunkContext in the CustomItemWriter.
The bean CustomItemWriter is of "step scope" (as far as I know, there is no "chunk scope" available), so each time a thread started a new ChunkContext, the method beforeChunk in CustomItemWriter was overwriting the previous ChunkContext and was messing everything up (previously counted would then be gone, since I had lost reference to previous ChunkContext instances).
So, I managed to fix the issue by using ThreadLocal, like below:
CustomItemWriter (v2)
public class CustomItemWriter implements ItemWriter<String[]> {
private ThreadLocal<ChunkContext> chunkContext = new ThreadLocal<ChunkContext>();
#Override
public void write(List<? extends String[]> items) throws Exception {
for (String[] item : items) {
((AtomicLong)this.chunkContext.get().getAttribute("chunkCounter")).incrementAndGet();
}
}
#BeforeChunk
private void beforeChunk(ChunkContext chunkContext) {
this.chunkContext.set(chunkContext);
}
}
Although I have managed to solve the problem, I'm wondering if there is a better way to access the current ChunkContext (for the current thread) from within the CustomItemWriter. Is there a way to get it programmaticaly? To do it "the Spring way", perhaps a [new] Chunk Scope should be implemented in newer versions of Spring Batch?
PS: Also, although the problem is solved, I thought it would be helpful to write this question so it can help someone with the same needs.
Although I have managed to solve the problem, I'm wondering if there is a better way to access the current ChunkContext (for the current thread) from within the CustomItemWriter
No, I see no other way to do that. However, it is not recommended to play with the chunk context in a multi-threaded step. This context is a mutable state which is shared between all threads processing the step (with all the nasty bugs of using shared mutable state in a multi-threaded environment).
The documentation will be updated in that regard, see https://github.com/spring-projects/spring-batch/pull/591/files#diff-177ad333794c9242aaa9ec2d0bec1842R147-R150.
For my batch application, I have a handful of steps I need to take prior to the execution of the Spring Batch job. For instance, I need to do a specific query and store data in a property - a List with a complex type (List<ComplexType>) - so that it can be used and manipulated throughout the Spring Batch job (primarily in the ItemReader).
I've tried autowiring in my list and accessing it in the step, but I can't access the data in the list that way. I get a null ComplexType in my List, no matter what values have been added to my autowired List property prior to the job.
I have also tried passing data using ExecutionContext, but I don't think this is supposed to work outside the Spring Batch job execution.
What I want to know, is what is the best way to populate an item prior to executing a Spring Batch job and maintain that object throughout the lifecycle of the job.
If the best way is one of the previous attempts I've made, any guidance on common mistakes with those approaches are appreciated, thanks.
Thanks Luca Basso Ricci for the JobExecutionListener pointer. I ended up creating my own StepExecutionListener where my pre-step processing would happen.
I followed this example from Mkyong which goes over different types of Spring Batch Listeners.
I created a custom listener like this one in the Java code:
public class CustomStepListener implements StepExecutionListener {
#Autowired
private CustomObject customObject;
#Override
public void beforeStep(StepExecution stepExecution) {
// initialize customObject and do other pre set setup
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return null;
}
And I initialized the autowired CustomObject class here. The CustomObject class is a custom object that simply contained my List of type ComplexType.
#Component
public class CustomObject {
private List<ComplexType> customObjectList;
public List<ComplexType> getCustomObjectList() {
return customObjectList;
}
public void setCustomObjectList(List<ComplexType> customObjectList) {
this.customObjectList= customObjectList;
}
}
Finally, in my job configuration 'batch-job-context.xml' I added my new listener:
<!-- ... -->
<beans:bean id="customStepListener"
class="com.robotsquidward.CustomStepListener"/>
<job id="robotsquidwardJob"
job-repository="jobRepository"
incrementer="runIdIncrementer">
<step id="robotsquidwardStep">
<tasklet task-executor="taskExecutor" throttle-limit="1">
<chunk
reader="robotsquidwardReader"
processor="robotsquidwardProcessor"
writer="robotsquidwardWriter"
commit-interval="1"/>
</tasklet>
<listeners>
<listener ref="customStepListener"/>
</listeners>
</step>
</job>
When I followed these steps I was able to initialize my ComplexObject List within the beforeJob function and access the values of the ComplexObject List within my job's Reader class:
#Component
#Scope(value = "step")
public class RobotsquidwardReader implements ItemReader<ComplexType> {
#Autowired
private CustomObject customObject;
#Override
public ComplexType read() throws Exception, UnexpectedInputException,
ParseException, NonTransientResourceException {
if(customObject.getCustomObjectList() != null) {
return customObject.getCustomObjectList.remove(0);
} else {
return null;
}
}
}
Easy as that. All it took is two new classes, a config change, and a major headache :)
You can do this :)
try to do this in job parameters incrementer trick :
<j:job id="Your_job" incrementer="incrementerBean">
and
<bean id="incrementerBean" class="com.whatever.IncrementerClass"/>
incrementer class :
class IncrementerClass implements JobParametersIncrementer {
#Override
JobParameters getNext(JobParameters parameters) {
Map<String, JobParameter> map = new HashMap<String, JobParameter>(
parameters.getParameters());
...
//you can put here your list, if it can be :
// Domain representation of a parameter to a batch job. Only the following types
// can be parameters: String, Long, Date, and Double.
//make some query
List<String> listStrings = Query.getYourQuery();
//Join your query into string to have something like this below
map.put("listOfSomething", new JobParameter("abc, abc, abc"));
...
return new JobParameters(map);
}
}
And thats all,
then you can use this parameter for example in some processing bean :
#Value("#{jobParameters['listOfSomething']}")
private String yourList
You can build your list from string, and thats all :)
good luck
I want to retrieve JobParameter and JobExecutionContext object in my ItemWriter class.
How to proceed?
I tried implementing StepExecutionListener through which I am just calling the parent class methods. But it is not succeeding.
Thanks in advance.
Implementing StepExecutionListener is one way. In fact that's the only way in Spring Batch 1.x.
Starting from Spring Batch 2, you have another choice: You can inject whatever entries in Job Parameters and Job Execution Context to your item writer. Make your item writer with step scope, then make use of expression like #{jobParameters['theKeyYouWant']} or #{jobExecutionContext['someOtherKey']} for value injecting to you item writer.
Use the #BeforeStep annotation to call a method before step processing.
//From the StepExecution get the current running JobExecution object.
public class MyDataProcessor implements ItemProcessor<MyDataRow, MyDataRow> {
private JobExecution jobExecution;
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
jobExecution = stepExecution.getJobExecution();
}
}
To add to Adrian Shum's answer, if you want to avoid each job parameter to be injected as a class property, you can directly inject the Map of JobParameters as follows:
#Value("#{jobParameters}")
private Map<String, JobParameter> jobParameters;
If you are using Spring Configuration file, you can access the StepExecution object with:
<bean id="aaaReader" class="com.AAAReader" scope="step">
<property name="stepExecution" value="#{stepExecution}" />
</bean>
In AAAReader class you need to create the proper field and a setter:
private StepExecution stepExecution;
public void setStepExecution(final StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
Same for Processor and Writer classes.