Spring-Batch second step not working properly

Spring-Batch second step not working properly - java

I am configuring a job with two step, Both step need to read different file and dump the data to database. When I am launching the job it executed the first step and save the data successfully and second step is not reading the file. But in database I can see that both Step Status is successful. My Configuration is looks like:
job id="job1" job-repository="jobRepository" xmlns="http://www.springframework.org/schema/batch" restartable="true">
<step id="step1">
<tasklet transaction-manager="transactionManager" task-executor="taskExecutor">
<chunk reader="DomainItemReader" writer="DomainItemWriter" commit-interval="40" skip-limit="10">
<skippable-exception-classes>
<include class="java.lang.Exception"/>
<exclude class="java.io.FileNotFoundException"/>
</skippable-exception-classes>
<listeners>
<listener ref="MyListener"/>
</listeners>
</chunk>
</tasklet>
<next on="COMPLETED" to="step2"/>
</step>
<step id="step2" >
<tasklet transaction-manager="transactionManager" task-executor="taskExecutor" allow-start-if-complete="true">
<chunk reader="DomainItemReader2" writer="DomainItemWriter2" commit-interval="100" skip-limit="10">
<skippable-exception-classes>
<include class="java.lang.Exception"/>
<exclude class="java.io.FileNotFoundException"/>
</skippable-exception-classes>
<listeners>
<listener ref="MyListener2"/>
</listeners>
</chunk>
</tasklet>
</step>
</job>
And my reader is looks like :
<bean id="billDomainItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" value="file:./firstfile.txt" />
<property name="linesToSkip" value="1" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter">
<util:constant static-field="org.springframework.batch.item.file.transform.DelimitedLineTokenizer.DELIMITER_TAB"/>
</property>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.abc.DomainMapper">
<property name="accessJobParam" ref="accessJobParam"/>
</bean>
</property>
</bean>
</property>
</bean>
And my second step's reader is looks like :
<bean id="DomainItemReader2" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="file:./secondFile.txt" />
<property name="linesToSkip" value="1" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<!-- <property name="names" value="PRODUCT_ID,NAME,DESCRIPTION,PRICE" />-->
<property name="delimiter">
<util:constant static-field="org.springframework.batch.item.file.transform.DelimitedLineTokenizer.DELIMITER_TAB"/>
</property>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.abc.DomainMapper2">
<property name="accessJobParam" ref="accessJobParam"/>
</bean>
</property>
</bean>
</property>
</bean>
Can any body please tell me where I am doing wrong and why it is not processing my second step file.
Thanks

I removed
allow-start-if-complete="true"`
from my second step and added
parent="step1"
in my second step and it works fine for me.

In your second reader configuration
<property name="fieldSetMapper">
<bean class="com.DomainMapper2">
<property name="accessJobParam" ref="accessJobParam"/>
</bean>
</property>
check the bean class mapping <bean class="com.DomainMapper2">
does this bean in same package or in different package <bean class="com.abc.DomainMapper2">

Related

How to improve Spring batch performance while working on more than 8 million records

I am trying to generate reports in my project using spring batch. I have more than 10 million records in my database. Complete process is taking more than 44 hours to generate the report.
Earlier I have set commit-interval as 1 but after reading some articles, I have set commit-interval as 10000 and page-size as 10000. I am using Task executor as well. But still, it's taking more than 44 hours to generate the report. In every iteration, It is taking 3 to 4 minutes to get the records, processes those records and writing in my CSV file.
Jobs-Context.xml
<job id="reportJob" xmlns="http://www.springframework.org/schema/batch">
<!-- master step, 10 threads (grid-size) -->
<step id="masterStep">
<partition step="slave" partitioner="rangePartitioner">
<handler grid-size="10" task-executor="taskExecutor" />
</partition>
<next on="*" to="combineStep"/>
</step>
<step id="combineStep">
<tasklet>
<chunk reader="multiResourceReader" writer="combineFlatFileItemWriter"
commit-interval="1" />
</tasklet>
<next on="*" to="deleteFiles"/>
</step>
<step id="deleteFiles">
<tasklet ref="debitfileDeletingTasklet" />
</step>
</job>
<!-- each thread will run this job, with different stepExecutionContext
values. -->
<step id="slave" xmlns="http://www.springframework.org/schema/batch">
<tasklet>
<chunk reader="pagingItemReader" writer="flatFileItemWriter"
processor="itemProcessor" commit-interval="10000" />
</tasklet>
</step>
<bean id="rangePartitioner" class="com.test.RangePartitioner" />
<bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="10" />
<property name="maxPoolSize" value="10" />
<property name="allowCoreThreadTimeOut" value="true" />
</bean>
itemreader bean
<bean id="pagingItemReader"
class="org.springframework.batch.item.database.JdbcPagingItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="queryProvider">
<bean
class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="selectClause" value="SELECT * " />
<property name="fromClause" value="***QUERY****/>
<property name="whereClause" value="where rn between :fromId and :toId" />
<property name="sortKey" value="rn" />
</bean>
</property>
<!-- Inject via the ExecutionContext in rangePartitioner -->
<property name="parameterValues">
<map>
<entry key="fromId" value="#{stepExecutionContext[fromId]}" />
<entry key="toId" value="#{stepExecutionContext[toId]}" />
</map>
</property>
<property name="pageSize" value="10000" />
<property name="rowMapper">
<bean class="com.hello.ItemRowMapper" />
</property>
</bean>
The whole process of read/write takes more than 44 hrs which is not acceptable to the client. How do i scale using spring batch.i want the job to complete this within overnight. Is it possible to generate a report having more than 8 million records using Spring Batch.

How to clear cache using CacheConfiguration in Apache ignite

Not able to destroy the cache after some duration time using Spring CacheConfiguration in Apache Ignite.After 40 seconds cache should be cleared.
Please see the below Code.
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="myCache" />
<property name="atomicityMode" value="ATOMIC" />
<property name="backups" value="1" />
<property name="cacheMode" value="PARTITIONED"/>
<property name="expiryPolicyFactory"ref="createdExpiryPolicyForDay"/>
</bean>
<bean id="createdExpiryPolicyForDay" class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf" >
<constructor-arg type="javax.cache.expiry.Duration" ref="durationForDay"/>
<constructor-arg type="javax.cache.expiry.Duration" >
<util:constant static-field="javax.cache.expiry.Duration.FIVE_MINUTES"/>
</constructor-arg>
</bean>
<bean name="durationForDay" class="javax.cache.expiry.Duration" >
<constructor-arg name="timeUnit" value="SECONDS" />
<constructor-arg name="durationAmount" value="40"/>
</bean>
Please help us, we are stuck with this issue

dynamically replace the value to placeholder in config xml file in Spring Batch

I am using Spring Batch 2 version. I am reading data from database using JdbcCursorItemReader.I have successfully fetched the data and also written it to a file.
Below is itemReader bean defined in Job.xml File::
<bean id="itemReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql"
value="select u.ID, u.USER_LOGIN, u.PASSWORD, u.AGE from USERS u" />
</property>
<property name="rowMapper">
<bean class="com.example.UserRowMapper" />
</property>
</bean>
But the issue is,my query is quite big so I just want to keep that query out of xml file and get that query from other file or property file(.property,yaml or xml).
So that I can write xml code as below::
<bean id="itemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader" scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql" value="$sql_query" />
</property><property name="rowMapper">
<bean class="com.example.UserRowMapper" />
</property>
</bean>
What is best way to achieve this?

<bean id="myProperties" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<list>
<value>path1.properties</value>
<value>path2.properties</value>
.....
</list>
</property>
<property name="ignoreUnresolvablePlaceholders" value="false"/>
</bean>
...
<bean id="itemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader" scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql" value="${sql_query}" />
</property><property name="rowMapper">
<bean class="com.example.UserRowMapper" />
</property>
</bean>
path1.properties:
sql_query=value
PropertySourcesPlaceholderConfigurer is preffered in 3.1 and higher, instead of PropertyPlaceholderConfigurer

You can add sql in jobexecutioncontext by using job listener before step.

Attempt to process next row failed . Closed Resultset error

Getting below error while executing a batch job. Not immediate but after half an hour.
2016-02-17 15:24:25,106 ERROR [AbstractStep ][poolTaskExecutor-2 ] Encountered an error executing step processSubscriptionFile-stepTwo in job subscriptionJob
org.springframework.jdbc.UncategorizedSQLException: Attempt to process next row failed; uncategorized SQLException for SQL [
SELECT DISTINCT PERSON_ID FROM CUST_SUB_INTER where job_id = 77317301 order by PERSON_ID
]; SQL state [99999]; error code [17010]; Closed Resultset: next; nested exception is java.sql.SQLException: Closed Resultset: next
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
at
My batch job files is as below. In first step I am reading the data from a file and adding to an intermediate table. This chunk uses partition and multiple threads. This step uses partitions and threads. In second step I am reading the data from this temporary table as distinct customer ids and passing it to writer. It is failing halfway through. There are 70000 records in intermediate table and 10000 distinct customer ids. Second step does not use partition and threads. Please help
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd"
default-autowire="byName">
<bean id="SubscriptionJob-jobDetail" class="org.springframework.scheduling.quartz.JobDetailBean">
<property name="jobClass" value="uk.co.batch.datamigration.JobLaunchDetails"/>
<property name="jobDataAsMap">
<map>
<entry key="jobName" value="SubscriptionJob" />
<entry key="jobLocator" value-ref="jobRegistry" />
<entry key="jobLauncher" value-ref="jobLauncher" />
</map>
</property>
</bean>
<!-- Job -->
<batch:job id="SubscriptionJob" restartable="false">
<batch:step id="copySubscriptionFile" next="processSubscriptionFile">
<tasklet ref="copySubscriptionFileLocally" />
</batch:step>
<batch:step id="processSubscriptionFile" next="processSubscriptionFile-stepTwo">
<partition step="processSubscriptionFile-stepOne" partitioner="SubscriptionPartitioner">
<handler grid-size="20" task-executor="SubscriptionTaskExecutor"/>
</partition>
</batch:step>
<batch:step id="processSubscriptionFile-stepTwo">
<batch:tasklet>
<batch:chunk reader="SubscriptionItemStepTwoReader" writer="SubscriptionItemStepTwoWriter" commit-interval="200"/>
</batch:tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="SubscriptionJobListener"/>
<batch:listener ref="SubscriptionJobNotifier"/>
</batch:listeners>
</batch:job>
<batch:step id="processSubscriptionFile-stepOne">
<batch:tasklet>
<batch:chunk reader="SubscriptionItemStepOneReader" writer="SubscriptionItemStepOneWriter" commit-interval="200"/>
</batch:tasklet>
</batch:step>
<bean id ="SubscriptionJobNotifier" class="uk.co.and.batch.status.JobExecutionNotifier">
<property name="snsEventType" value="SUBSCRIPTION_JOB_NOTIFICATION"/>
<property name="onlyNotifyFailures" value="true"/>
</bean>
<bean id="SubscriptionPartitioner" class="uk.co.batch.datamigration.FlatFilePartitioner" scope="step">
<property name="resource" value="file:#{jobExecutionContext[tempSubscriptionFile]}" />
</bean>
<bean id="SubscriptionItemStepOneReader" scope="step" autowire-candidate="false" parent="SubscriptionItemStepOneReaderParent">
<property name="resource" value="file:#{jobExecutionContext[tempSubscriptionFile]}" />
<property name="startAt" value="#{stepExecutionContext['startAt']}"/>
<property name="itemsCount" value="#{stepExecutionContext['itemsCount']}"/>
</bean>
<bean id="SubscriptionItemStepOneReaderParent" abstract="true"
class="uk.co.batch.datamigration.MultiThreadedFlatFileItemReader">
<property name="linesToSkip" value="1"/>
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="^"/>
<property name="names">
<list >
<value>city_name</value>
<value>country_name</value>
<value>publication_name</value>
<value>subscription_ref_code</value>
<value>subscription_state</value>
<value>unsubscribed</value>
<value>city_id</value>
<value>person_id</value>
<value>subscription_created_at</value>
<value>subscription_updated_at</value>
<value>end_value</value>
</list>
</property>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="SubscriptionVO"/>
</bean>
</property>
</bean>
</property>
<property name="recordSeparatorPolicy">
<bean class="org.springframework.batch.item.file.separator.DefaultRecordSeparatorPolicy"/>
</property>
</bean>
<bean id="SubscriptionVO" class="uk.co.batch.datamigration.subscription.SubscriptionVO" scope="prototype"/>
<!-- TODO:Reduce number of columns to improve performance -->
<bean id="SubscriptionItemStepOneWriter"
class="org.springframework.batch.item.database.JdbcBatchItemWriter" scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value>
<![CDATA[
insert into CUST_SUB_INTER(ID,CITY_NAME,COUNTRY_NAME,PUBLICATION_NAME,SUBSCRIPTION_REF_CODE,SUBSCRIPTION_STATE,
UNSUBSCRIBED,CITY_ID,PERSON_ID,SUBSCRIPTION_CREATED_AT,SUBSCRIPTION_UPDATED_AT,END_VALUE,JOB_ID,
MIGRATION_FILE_NAME,RECORD_INSERT_DATE)
values (SEQ_CUST_SUB_ERROR.NEXTVAL, :city_name, :country_name, :publication_name,
:subscription_ref_code, :subscription_state, :unsubscribed,
:city_id, :person_id, :subscription_created_at, :subscription_updated_at, :end_value,
#{jobExecutionContext[jobId]}, '#{jobExecutionContext[tempSubscriptionFile]}', SYSTIMESTAMP)
]]>
</value>
</property>
<property name="itemSqlParameterSourceProvider">
<bean class="org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider" />
</property>
</bean>
<!-- scope step is critical here-->
<bean id="SubscriptionItemStepTwoReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader" scope="step">
<property name="dataSource" ref="dataSource"/>
<property name="sql">
<value>
<![CDATA[
SELECT DISTINCT PERSON_ID FROM CUST_SUB_INTER where job_id = #{jobExecutionContext.jobId} order by PERSON_ID
]]>
</value>
</property>
<property name="rowMapper">
<bean class="org.springframework.jdbc.core.SingleColumnRowMapper" p:requiredType="java.lang.Long"/>
</property>
</bean>
<bean id="SubscriptionItemStepTwoWriter" class="uk.co.batch.datamigration.subscription.SubscriptionItemStepTwoWriter" scope="step">
<property name="fileName" value="#{jobExecutionContext[tempSubscriptionFile]}" />
<property name="jobId" value="#{jobExecutionContext.jobId}" />
<property name="errorSaving" value="${Job.errorSaving}"/>
</bean>
<bean id="SubscriptionTaskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="10"/>
</bean>
<bean id="SubscriptionJobListener" class="uk.co.batch.datamigration.JobListener">
<property name="sessionFactory" ref="sftpSessionFactory"/>
<property name="remotePath" value="${Job.subscription.sftp.remoteDirectory}"/>
<property name="remoteExtension" value="${Job.sftp.remoteExtension}"/>
<property name="renameRemoteFile" value="true"/>
<property name="jobContextSingleFileParameterName" value="tempSubscriptionFile"/>
<property name="batchAdministratorUserId" value="${Job.batchAdministratorUserId}"/>
</bean>
<bean id="copySubscriptionFileLocally" class="uk.co.and.batch.core.LocalCopyTasklet" scope="prototype">
<property name="deleteRemoteFile" value="false"/>
<property name="fileNamePattern" value="${Job.subscription.filePattern}"/>
<property name="jobContextSingleFileParameterName" value="tempSubscriptionFile"/>
<property name="localPath" value="${Job.subscription.sftp.localDirectory}"/>
<property name="mustMatch" value="true"/>
<property name="remotePath" value="${Job.subscription.sftp.remoteDirectory}"/>
<property name="remoteSessionFactory" ref="sftpSessionFactory"/>
<property name="maxNumberDownload" value="1"/>
<property name="resolver">
<bean class="uk.co.and.batch.core.remote.FileNameResolvers.SFtpFileNameResolver" />
</property>
</bean>
<bean id="sftpSessionFactory" class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="ftp.co.uk"/>
<property name="port" value="22"/>
<property name="user" value=""/>
<property name="password" value=""/>
<property name="proxy" value="#{ ${proxyRef} }"/>
</bean>
</beans>

Spring Batch: Fill n files but the next step sees empty

I implemented a step in spring batch that reads records from the database and generates output files.
The step is partitioned so that each slave step generates a different file.
Then I implemented a step which is executed after the slaves, and what it does is to read each of these
generated files and consolidate them into a single file (merge).
The problem I am having is that when run the consolidator step, the input files are empty (probably because
no flush was performed), then tried to configure forceSync = true and transactional = false in the slave writer, but without effect.
When they finish the job, the files generated by the slaves contains data, and the merge file is empty.
The problem, then, is that the input files are empty at the time of execution of consolidator step.
Can help me?, needs more information?
(sorry for my english)
thanks
<batch:job id="interfacesJob" xmlns="">
<batch:step id="step.1" parent="readerParent" next="step.2"/>
<batch:step id="step.2" parent="merge"/>
</batch:job>
<batch:step id="readerParent">
<batch:partition step="slave" partitioner="partitioner">
<batch:handler grid-size="50" task-executor="poolTaskExecutor" />
</batch:partition>
</batch:step>
<batch:step id="slave">
<batch:tasklet
transaction-manager="transactionManager">
<batch:chunk
reader="dummyReader"
processor="dummyProcessor"
writer="dummyWriter"
commit-interval="1"
skip-limit="50">
</batch:tasklet>
</batch:step>
<bean id="dummyWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter"
scope="step" >
<property name="resource" value="file:operations_#{stepExecutionContext[fromId]}.txt" />
<property name="forceSync" value="true" />
<property name="transactional" value="false" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value="," /> <!-- default -->
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="id,field1,field2" />
</bean>
</property>
</bean>
</property>
</bean>
<batch:step id="merge">
<tasklet>
<chunk reader="mergeReader" writer="mergeWriter" commit-interval="1" />
</tasklet>
</batch:step>
<bean id="mergeReader"
class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resources" value="file:operations_*.txt" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="," />
<property name="names" value="id,field1,field2" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="dummyPrototypeBean" />
</bean>
</property>
</bean>
</property>
</bean>
<bean id="mergeWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:final.txt" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value="," />
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="id,field1,field2" />
</bean>
</property>
</bean>
</property>
</bean>
I taste with a MultiResourceItemReader for the mergeReader, but still continue with the same problem:
<bean id="mergeReader"
class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="file:operations_*.txt" />
<property name="delegate" ref="mergeReaderSpecific" />
</bean>
<bean id="mergeReaderSpecific"
class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="," />
<property name="names" value="id,field1,field2" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="dummyPrototypeBean" />
</bean>
</property>
</bean>
</property>
</bean>

Probably you have to use a MultiResourceItemReader as mergeReader using a FlatFileItemReader as delegate.
FlatFileItemReader.resource property is intended to be used on a single resource opposite to
MultiResourceItemReader.resources (plurals).

Develop Reference

Java is a programming language and computing platform first released by Sun Microsystems in 1995.

Spring-Batch second step not working properly - java

I removed allow-start-if-complete="true"` from my second step and added parent="step1" in my second step and it works fine for me.

Related

How to improve Spring batch performance while working on more than 8 million records

How to clear cache using CacheConfiguration in Apache ignite

dynamically replace the value to placeholder in config xml file in Spring Batch

Attempt to process next row failed . Closed Resultset error

Spring Batch: Fill n files but the next step sees empty

Categories

Resources