I am using spring batch, but due to job instance already exist error I need to add current time in my job parameter. I am unable to figure out where to add job parameters. Here is my code:
<step id="myStep">
<tasklet>
<chunk reader="myReader" processor="myProcessor" writer="myWriter" commit-interval="6000" skip-limit="9000">
//some more code.
</chunk>
</tasklet>
</step>
<bean id="myReader" class="org.springframework,batch.item.database.StoredProcedueItemReader" scope="step">
//define property for datasource , procedurename , rowmapper, parameters
<property name="preparedStatementSetter" ref="myPreparedStatmentSetter">
</bean>
<bean id="myPreparedStatmentSetter" class="com.mypackage.MyPreparedStatementSetter" scope="step">
<property name="kId" value="#{jobParameters[kId]}">
</bean>
When I try to run the job for same kId multiple times I get The job already exist error, so I need to add current timestamp to my job parameter.
Would adding current time stamp as a property in the bean myPreparedStatmentSetter be sufficient, or do I need to add jobparameter somewhere else too? From where exactly are jobparameters picked from in spring file?
In case I need to add timestamp to the bean here is a questions -My stored procedure takes only kID as paramter, I dont need to pass current time stamp to stored procedure, then why I need to add the same in myPreparedStatmentSetter.
Also how would I add current timestamp in an xml file without java code?
EDIT
Here is my jobLauncher bean
<bean Id= "jobLauncher "class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" value="myJobRepo">
</bean>
Adding a "random" job parameter by hand, while it can work, isn't the most ideal way to get around the job instance already exists error. Instead, you should consider adding a JobParametersIncrementer to your job. Spring provides the RunIdIncrementer as an implementation of this out of the box. A job configured with it would look something like the following:
#Bean
public Job myJob() {
return jobBuilderFactory.get("myJob")
.incrementer(runIdIncrementer())
.start(step1())
.build();
}
#Bean
public JobParametersIncrementer runIdIncrementer() {
return new RunIdIncrementer();
}
I am guessing that you already adding KId to your job parameters. Add following to your joblaucher.run() method.
new JobParametersBuilder()
.addLong("time",System.currentTimeMillis())
.addLong("KId",<your KID>)
.toJobParameters();
Related
We have created a simple spring batch job with single step. There are custom implemented ItemReader and ItemWriter. The ItemReader gets the initial data from job parameter. The batch runs perfectly when run as a standalone java process. But what we want is to host the batch on some server. Therefore, we have created REST service to initialize the batch. The service calls the job URL and passes some parameter. This parameter is passed as job parameter to the batch. The service and job run fine when it is called for one parameter.
But when we call the service more than once (twice for testing purpose), the batch behaves strangely. We are passing different job parameters. But when the execution starts for second job initialization, the job parameter value which is received by the ItemReader is the same as the one for the first execution. And both execution interfere with each other, sharing database connection, interfering with data retrieved etc.
We have tried setting the restartable parameter to false but it didn't work. We have also tried the following solution:
Can we create multiple instances of a same java(spring) batch job?
The above solution started giving "Interrupted attempting lock" error in JBoss.
On further investigation we found that ItemReader is getting initialized only once. That is why it is getting same job parameter value and is interfering with the previous execution.
EDIT
Following is the job configuration:
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
<job id="jobid" restartable="false">
<step id="step1">
<tasklet>
<chunk reader="reader" writer="writer"
commit-interval="2">
</chunk>
</tasklet>
</step>
</job>
Following is the code snippet to launch the job:
JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher");
Job job = (Job) context.getBean("jobid");
try {
JobParameters param = new JobParametersBuilder().addString("key","value").toJobParameters();
JobExecution execution = jobLauncher.run(job, param);
} catch (Exception e) {
e.printStackTrace();
}
Can anyone please suggest some solution? Am I missing some configuration for the step?
Thanks in advance.
I found that if we create the Context and JobLauncher objects statically, that is, if there is only one instance of these two objects, the above thing can work. In this way, we can launch the same job multiple times, but with different parameters.
Class MyClass{
private static ConfigurableApplicationContext context = null;
private static JobLauncher jobLauncher = null;
static{
String[] springConfig = {BatchTokeniserConstants.SPRING_CONFIG_FILE_NAME};
try {
context = new ClassPathXmlApplicationContext(springConfig);
jobLauncher = (JobLauncher) context.getBean("jobLauncher");
BatchTokeniserUtils.loadSystemVaiables();
} catch (BeansException e) {
}
}
}
Now the jobLauncher can be used to launch any job any number of time.
I hope it helps others.
How can I implement a Spring Batch job which has to read a list and then repeat one or more steps for each item in the list?
I am currently reading the list in the one step and then I put it in the job context. But the job context is persisted in the DB and if it gets too big, a CLOB has to be used and I do not have access to one.
So I am looking for a solution that doesn't involve storing the whole list on the job context.
Of course, I could simply put the list in a local variable. But I am curious whether there is a more Spring Batch-like option.
Aside from the comments above about structuring the job in the first place (which I tend to agree with), if you use the latest 3.0.0.M3, you can create a JobScope'ed container that could hold the collection as you loop through the various steps. From there you can read/process/write to that container instead of an external source.
As discussed on Spring Batch-Repeat step for each item in a data list question, Partitioning seems to be the solution.
Configuration for PartitionHandler:
<batch:step id="anyNameJobMaster">
<batch:partition step="nameOfActualStepToBeIterated" partitioner="partitioner">
<batch:handler grid-size="10" task-executor="taskExecutor" />
</batch:partition>
</batch:step>
<bean id="partitioner" class="org.example.PartitionerThatImplementsPartitioner" lazy-init="true" scope="step"/>
PartitionHandler Sample Code:
public class PartitionerThatImplementsPartitioner implements Partitioner{
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> map = new HashMap<String, ExecutionContext>(gridSize);
//Iteration goes here
//For Each Iteration, create ExecutionContext and add it to map.
map.put("somePartionKeyString", new ExecutionContext().put("ContextIdentifiere.g.FileName", "IteratedVale.g.FilePath"))
}
}
Job Configuration:
<batch:step id="nameOfActualStepToBeIterated">
<batch:tasklet transaction-manager="someTxManager">
<batch:chunk processor="someProcessor"
writer="itemWriter" commit-interval="${commitInterval}">
<batch:reader>
<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="#{stepExecutionContext['ContextIdentifiere.g.FileName']}" />
</bean>
</batch:reader>
</batch:chunk>
</batch:tasklet>
We are using Quartz 2.1.5; we have the following properties set:
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.CloudscapeDelegate
org.quartz.jobStore.useProperties = true
org.quartz.jobStore.tablePrefix=QRTZ_
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval=20000
and the following beans configuration:
<bean name="abcRequestsJob" class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
<property name="jobClass" value="com.hsc.correspondence.job.AbcRequestsJob" />
<property name="group" value="sftpTransfers"/>
</bean>
<bean id="abcRequestsJobTrigger" class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
<property name="jobDetail" ref="abcRequestsJob" />
<property name="group" value="sftpTransfers"/>
<property name="cronExpression" value="${quartz.abcRequests.cronExpression}" />
</bean>
When we run, we are getting an error saying that
nested exception is org.quartz.JobPersistenceException: Couldn't store trigger 'sftpTransfers.abcRequestsJobTrigger' for 'sftpTransfers.abcRequestsJob'
job:JobDataMap values must be Strings when the 'useProperties' property is set.
Key of offending value: jobDetail
[See nested exception: java.io.IOException: JobDataMap values must be Strings when the 'useProperties' property is set. Key of offending value: jobDetail]
Is there another way to configure a CronTriggerFactoryBean than using a reference to the JobDetailFactoryBean reference, or a different trigger factory bean that only takes strings as properties? This all worked before we wanted to use clustering, but now that the job is going to be written to a blob they want only strings to be persisted. That's fine, how do I get it done?
Please refer:
http://site.trimplement.com/using-spring-and-quartz-with-jobstore-properties/
http://forum.springsource.org/archive/index.php/t-130984.html
Problem:
This happens with Spring Framework and Quartz together when using org.quartz.jobStore.useProperties=true, meaning that all Job data is stored in the database as properties instead of serialized Java objects.
Error is because of Spring class CronTriggerFactoryBean that stores a reference to the JobDetail in the JobDataMap, which cannot be represented as a set of properties.
CronTriggerFactoryBean is setting the jobDetail into the trigger's jobDataMap.
Workaround:
Extend CronTriggerFactoryBean and remove JobDetail from jobDataMap.
import org.springframework.scheduling.quartz.CronTriggerFactoryBean;
import org.springframework.scheduling.quartz.JobDetailAwareTrigger;
public class PersistableCronTriggerFactoryBean extends CronTriggerFactoryBean {
#Override
public void afterPropertiesSet() {
super.afterPropertiesSet();
//Remove the JobDetail element
getJobDataMap().remove(JobDetailAwareTrigger.JOB_DETAIL_KEY);
}
}
Is there any possibility to find out, If a job is restarted in Spring Batch?
We do provide some Tasklets without restart-support from spring-batch and has to implement our own proceeding, if job is restarted.
Can't find any possibility in JobRepository, JobOperator, JobExplorer, etc.
Define a JobExplorer bean with required properties
<bean id="jobExplorer"
class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="lobHandler" ref="lobHandler"/>
</bean>
Query it with your jobName
List<JobInstance> jobInstances= jobExplorer.getJobInstances(jobName);
for (JobInstance jobInstance : jobInstances) {
List<JobExecution> jobExecutions = jobExplorer.getJobExecutions(jobInstance);
for (JobExecution jobExecution : jobExecutions) {
if (jobExecution.getExitStatus().equals(ExitStatus.COMPLETED)) {
//You found a completed job, possible candidate for a restart
//You may check if the job is restarted comparing jobParameters
JobParameters jobParameters = jobInstance.getParameters();
//Check your running job if it has the same jobParameters
}
}
}
Did not compile this but I hope it gives an idea
Another way using jobExplorer is execute the following command:
jobExplorer.getJobExecutions(jobExplorer.getJobInstance(currentJobExecution.getJobInstance().getId())).size() > 1;
This statement verifies if another execution of the the same job (same id) exists. In environments with minimum control, does not exist possibility that the other execution be not a failed or stopped execution.
Potentially you can find this information in spring-batch's database tables, can't remeber the exact table's name, but you can figure out quickly because there are only few tables. I guess there is some information regarding restarting.
I have a Spring-Batch job that I launch from a Spring MVC controller. The controller gets an uploaded file from the user and the job is supposed to process the file:
#RequestMapping(value = "/upload")
public ModelAndView uploadInventory(UploadFile uploadFile, BindingResult bindingResult) {
// code for saving the uploaded file to disk goes here...
// now I want to launch the job of reading the file line by line and saving it to the database,
// but I want to launch this job in a new thread, not in the HTTP request thread,
// since I so not want the user to wait until the job ends.
jobLauncher.run(
jobRegistry.getJob(JOB_NAME),
new JobParametersBuilder().addString("targetDirectory", folderPath).addString("targetFile", fileName).toJobParameters()
);
return mav;
}
I've tried the following XML config:
<job id="writeProductsJob" xmlns="http://www.springframework.org/schema/batch">
<step id="readWrite">
<tasklet task-executor="taskExecutor">
<chunk reader="productItemReader" writer="productItemWriter" commit-interval="10" />
</tasklet>
</step>
</job>
<bean id="taskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="5" />
<property name="maxPoolSize" value="5" />
</bean>
...but it seems like the multithreading happens only within the job boundaries itself. I.e., the controller thread waits until the job ends, and the job execution is handled by multiple threads (which is good but not the main thing I wanted). The main thing I wanted is that the job will be launched on a separate thread (or threads) while the controller thread will continue its execution without waiting for the job threads to end.
Is there a way to achieve this with Spring-batch?
The official documentation describes your exact problem and a solution in 4.5.2. Running Jobs from within a Web Container:
[...] The controller launches a Job using a JobLauncher that has been configured to launch asynchronously, which immediately returns a JobExecution. The Job will likely still be running, however, this nonblocking behaviour allows the controller to return immediately, which is required when handling an HttpRequest.
Spring Batch http://static.springsource.org/spring-batch/reference/html-single/images/launch-from-request.png
So you were pretty close in trying to use TaskExecutor, however it needs to be passed to the JobLauncher instead:
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
<property name="taskExecutor" ref="taskExecutor"/>
</bean>
Disclaimer: I have never used Spring Batch...
The jobLauncher.run() method can be called in a new Thread like so:
#RequestMapping(value = "/upload")
public ModelAndView uploadInventory(UploadFile uploadFile, BindingResult bindingResult) {
[...]
final SomeObject jobLauncher = [...]
Thread thread = new Thread(){
#Override
public void run(){
jobLauncher.run([...]);
}
};
thread.start();
return mav;
}
The thread.start() line will spawn a new thread, and then continue to execute the code below it.
Note that, if jobLauncher is a local variable, it must be declared final in order for it to be used inside of the anonymous Thread class.
If you don't need to show the processing errors to your client, you can start the spring batch job in a seperate thread.