I have the following code
#Bean
public JdbcBatchItemWriter<QuotationDto> writer1() {
return new JdbcBatchItemWriterBuilder<QuotationDto>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO ...")
.dataSource(dataSource)
.build();
}
#Bean
public JdbcBatchItemWriter<QuotationDto> writer2() {
return new JdbcBatchItemWriterBuilder<QuotationDto>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO ...")
.dataSource(dataSource)
.build();
}
#Bean
public CompositeItemWriter<QuotationDto> compositeItemWriter() {
CompositeItemWriter writer = new CompositeItemWriter();
writer.setDelegates(Arrays.asList(writer1(), writer2()));
return writer;
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<QuotationDto, QuotationDto>chunk(5)
.reader(reader())
.processor(processor())
.writer(compositeItemWriter())
.stream(writer1())
.stream(writer2())
.build();
}
I get IntelliJ error in setting writer1 as stream, because id does no implement ItemStream.
What am I doing wrong? Do anyone has solutions? I don't find so much informations about java-based composite writer configuration.
JdbcBatchItemWriter does not implement ItemStream so it cannot be used as a stream in a chunk-oriented step.
If you want to compose two JDBC item writers, you can create a custom item writer that delegates to a JdbcTemplate. Here is a quick example:
class MyItemWriter implements ItemWriter<QuotationDto> {
private JdbcTemplate jdbcTemplate;
public MyItemWriter(DataSource dataSource) {
this.jdbcTemplate = new JdbcTemplate(dataSource);
}
#Override
public void write(List<? extends QuotationDto> items) throws Exception {
for (QuotationDto dto : items) {
// use jdbcTemplate to batch insert items.
// can do multiple inserts here, they will be part of
// the same transaction driven by Spring Batch
}
}
}
Related
How to validate database is pulling data or not in spring batch since ItemWriter is not getting called.
I am following this example where I am reading the data from the table and trying to print the list.
Read from Table
GitHub link for that
GitHub Link
So here is my batch config :
#Configuration
#EnableBatchProcessing
public class BatchConfing {
private static final String QUERY_FIND_STUDENTS =
"select " +
"stu_info " +
"from myTable " +
"where status = '\"+PASS + \"' " ;
#Bean
public ItemReader<StudentDTO> itemReader(DataSource dataSource) {
System.out.println("Reached Reader");
return new JdbcCursorItemReaderBuilder<StudentDTO>()
.name("cursorItemReader")
.dataSource(dataSource)
.sql(QUERY_FIND_STUDENTS)
.rowMapper(new BeanPropertyRowMapper<>(StudentDTO.class))
.build();
}
#Bean
public ItemWriter<StudentDTO> itemWriter(){
System.out.println("I am in writer");
return new LoggingItemWriter();
/* return student -> {
System.out.println("student.toString()");
};*/
}
#Bean
public JobExecutionListener listener() {
return new ScheduelListener();
}
#Bean
public Step stepA(#Qualifier("itemReader") ItemReader<StudentDTO> reader,
#Qualifier("itemWriter") ItemWriter<StudentDTO> writer,
StepBuilderFactory stepBuilderFactory) {
return stepBuilderFactory.get("stepA")
.<StudentDTO,StudentDTO>chunk(2)
.reader(reader)
.writer(writer)
.build()
;
}
#Bean
public Job jobA(#Qualifier("stepA") Step exampleJobStep,
JobBuilderFactory jobBuilderFactory){
return jobBuilderFactory.get("jobA")
.incrementer(new RunIdIncrementer())
.listener(listener())
.flow(exampleJobStep)
.end()
.build()
;
}
}
Here I can see itemreader and itemwriter is getting called but from item writer it si not going to write method of LoggingItemWriter
public class LoggingItemWriter implements ItemWriter<StudentDTO> {
public LoggingItemWriter() {
System.out.println("test LoggingItemWriter ");
}
private static final Logger LOGGER = LoggerFactory.getLogger(LoggingItemWriter.class);
#Override
public void write(List<? extends StudentDTO> list) throws Exception {
LOGGER.info("Writing student info: {}", list);
for (StudentDTO item : list) {
System.out.println("test"+list);
}
}
}
You should to make sure your item reader is correctly defined and returning data from the table. A unit test can validate that.
Once your reader is correctly defined, then if the writer is not called, this means the table is empty (the reader is returning null on the first call).
I have a spring batch application to get a file in samba server
and generate a new file in a different folder on the same server.
However,
only ItemReader is called in the flow.
What is the problem? Thanks.
BatchConfiguration:
#Configuration
#EnableBatchProcessing
public class BatchConfiguration extends BaseConfiguration {
#Bean
public ValeTrocaItemReader reader() {
return new ValeTrocaItemReader();
}
#Bean
public ValeTrocaItemProcessor processor() {
return new ValeTrocaItemProcessor();
}
#Bean
public ValeTrocaItemWriter writer() {
return new ValeTrocaItemWriter();
}
#Bean
public Job importUserJob(JobCompletionNotificationListener listener) throws Exception {
return jobBuilderFactory()
.get("importUserJob")
.incrementer(new RunIdIncrementer())
.repository(getJobRepository())
.listener(listener)
.start(this.step1())
.build();
}
#Bean
public Step step1() throws Exception {
return stepBuilderFactory()
.get("step1")
.<ValeTroca, ValeTroca>chunk(10)
.reader(this.reader())
.processor(this.processor())
.writer(this.writer())
.build();
}
}
BaseConfiguration:
public class BaseConfiguration implements BatchConfigurer {
#Bean
#Override
public PlatformTransactionManager getTransactionManager() {
return new ResourcelessTransactionManager();
}
#Bean
#Override
public SimpleJobLauncher getJobLauncher() throws Exception {
final SimpleJobLauncher simpleJobLauncher = new SimpleJobLauncher();
simpleJobLauncher.setJobRepository(this.getJobRepository());
return simpleJobLauncher;
}
#Bean
#Override
public JobRepository getJobRepository() throws Exception {
return new MapJobRepositoryFactoryBean(this.getTransactionManager()).getObject();
}
#Bean
#Override
public JobExplorer getJobExplorer() {
MapJobRepositoryFactoryBean repositoryFactory = this.getMapJobRepositoryFactoryBean();
return new SimpleJobExplorer(repositoryFactory.getJobInstanceDao(), repositoryFactory.getJobExecutionDao(),
repositoryFactory.getStepExecutionDao(), repositoryFactory.getExecutionContextDao());
}
#Bean
public MapJobRepositoryFactoryBean getMapJobRepositoryFactoryBean() {
return new MapJobRepositoryFactoryBean(this.getTransactionManager());
}
#Bean
public JobBuilderFactory jobBuilderFactory() throws Exception {
return new JobBuilderFactory(this.getJobRepository());
}
#Bean
public StepBuilderFactory stepBuilderFactory() throws Exception {
return new StepBuilderFactory(this.getJobRepository(), this.getTransactionManager());
}
}
ValeTrocaItemReader:
#Configuration
public class ValeTrocaItemReader implements ItemReader<ValeTroca>{
#Value(value = "${url}")
private String url;
#Value(value = "${user}")
private String user;
#Value(value = "${password}")
private String password;
#Value(value = "${domain}")
private String domain;
#Value(value = "${inputDirectory}")
private String inputDirectory;
#Bean
#Override
public ValeTroca read() throws MalformedURLException, SmbException, IOException, Exception {
File tempOutputFile = getInputFile();
DefaultLineMapper<ValeTroca> lineMapper = new DefaultLineMapper<>();
lineMapper.setLineTokenizer(new DelimitedLineTokenizer() {
{
setDelimiter(";");
setNames(new String[]{"id_participante", "cpf", "valor"});
}
});
lineMapper.setFieldSetMapper(
new BeanWrapperFieldSetMapper<ValeTroca>() {
{
setTargetType(ValeTroca.class);
}
});
FlatFileItemReader<ValeTroca> itemReader = new FlatFileItemReader<>();
itemReader.setLinesToSkip(1);
itemReader.setResource(new FileUrlResource(tempOutputFile.getCanonicalPath()));
itemReader.setLineMapper(lineMapper);
itemReader.open(new ExecutionContext());
tempOutputFile.deleteOnExit();
return itemReader.read();
}
Sample of ItemProcessor:
public class ValeTrocaItemProcessor implements ItemProcessor<ValeTroca, ValeTroca> {
#Override
public ValeTroca process(ValeTroca item) {
//Do anything
ValeTroca item2 = item;
System.out.println(item2.getCpf());
return item2;
}
EDIT:
- Spring boot 2.1.2.RELEASE - Spring batch 4.1.1.RELEASE
Looking at your configuration, here are a couple of notes:
BatchConfiguration looks good. That's a typical job with a single chunk-oriented step.
BaseConfiguration is actually the default configuration you get when using #EnableBatchProcessing without providing a datasource. So this class can be removed
Adding #Configuration on ValeTrocaItemReader and marking the method read() with #Bean is not correct. This means your are declaring a bean named read of type ValeTroca in your application context. Moreover, your custom reader uses a FlatFileItemReader but has no added value compared to a FlatFileItemReader. You can declare your reader as a FlatFileItemReader and configure it as needed (resource, line mapper, etc ). This will also avoid the mistake of opening the execution context in the read method, which should be done when initializaing the reader or in the ItemStream#open method if the reader implements ItemStream
Other than that, I don't see from what you shared why the processor and writer are not called.
SOLVED: The problem was that even though I'm not using any databases, the spring batch, although configured to have the JobRepository in memory, needs a database (usually H2) to save the configuration tables, jobs, etc.
In this case, the dependencies of JDBC and without H2 in pom.xml were disabled. Just added them to the project and the problem was solved!
I have a Spring Boot project that use Mongodb. So, in my pom i have that dependence:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
So i'm able to access to the database with this repository class:
package it.de.marini.server.dal;
import org.springframework.data.mongodb.repository.MongoRepository;
import it.de.marini.server.model.Role;
public interface RoleRepository extends MongoRepository<Role, String> {
}
I need to inizialize my data in Mongodb database putting default Role for example. What is the best way in Spring Boot framework?
There are several ways to do this, I will suggest you with CommandlineRunner
try:
#Bean
public CommandLineRunner initConfig(MyRepo repo) {
if (data not exist) {
repo.save(...);
}
}
Otherwise you can use #PostConstruct to initiate it..
if you need something like liquibase for RDBMS, checkout mongobee: https://github.com/mongobee/mongobee
As #Jaiwo99 say i understand that there is not a standard for do that. So i decided to make the work with Spring Batch. I realized a batch for load from CSV files Roles and Permissions of my application.
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public MongoTemplate mongoTemplate;
#Bean
public PlatformTransactionManager transactionManager() {
return new ResourcelessTransactionManager();
}
#Bean
public Tasklet defaultRolePermissionTasklet() {
return new DefaultRolePermissionTasklet();
}
public <T> FlatFileItemReader<T> readerFile(String fileName,String[] fields,Class<T> type) {
FlatFileItemReader<T> reader = new FlatFileItemReader<T>();
reader.setStrict(false);
reader.setResource(new ClassPathResource(fileName));
reader.setLineMapper(new DefaultLineMapper<T>() {
{
setLineTokenizer(new DelimitedLineTokenizer() {
{
setNames(fields);
setStrict(false);
}
});
setFieldSetMapper(new BeanWrapperFieldSetMapper<T>() {
{
setTargetType(type);
}
});
}
});
return reader;
}
#Bean
public PermissionItemProcessor permissionProcessor() {
return new PermissionItemProcessor();
}
#Bean
public RoleItemProcessor roleProcessor() {
return new RoleItemProcessor();
}
#Bean
public Job initAuthenticationData(JobCompletionNotificationListener listener) {
return jobBuilderFactory.get("initAuthenticationData").incrementer(new RunIdIncrementer()).listener(listener)
.start(stepPermission())
.next(stepRole())
.next(stepDefaultRolePermissions())
.build();
}
#Bean
public Step stepDefaultRolePermissions() {
return stepBuilderFactory.get("stepDefaultRolePermissions").tasklet(defaultRolePermissionTasklet()).build();
}
#Bean
public Step stepPermission() {
MongoItemWriter<Permission> writer = new MongoItemWriter<Permission>();
writer.setTemplate(mongoTemplate);
return stepBuilderFactory.get("stepPermission").<Permission, Permission>chunk(20)
.reader(readerFile("permission-data.csv",new String[] {"name","description"},Permission.class))
.processor(permissionProcessor())
.writer(writer)
.build();
}
#Bean
public Step stepRole() {
MongoItemWriter<Role> writer = new MongoItemWriter<Role>();
writer.setTemplate(mongoTemplate);
return stepBuilderFactory.get("stepRole").<Role, Role>chunk(20)
.reader(readerFile("role-data.csv",new String[] {"name","description"},Role.class))
.processor(roleProcessor())
.writer(writer)
.build();
}
}
At least, there is one more way how to initialize data in Mongodb using Spring Boot. You can create in your configuration like this:
#Configuration
public class AppConfiguration {
#Autowired
public void prepare(ReactiveMongoOperations mongoOperations,
UserRepository userRepository) {
mongoOperations.createCollection("users",
CollectionOptions.empty()
.maxDocuments(1_000)
.size(1024 * 8)
.capped()).block();
userRepository
.insert(List.of(
User.builder()
.name("Joe Doe")
.build()
))
.blockLast();
}
}
And of course, you must make a check that collection doesn't exist, in order to not create a collection if the database has already been created.
So I have a problem in Spring Batch 3.0.7.RELEASE and Spring 4.3.2.RELEASE where the Listeners are not running in my ItemProcessor class. Regular injection at the #StepScope level is working for #Value("#{jobExecutionContext['" + Constants.SECURITY_TOKEN + "']}") as seen below. But it isn't working for beforeProcess or beforeStep, I have tried both the annotation version and interface version. I'm almost 100% sure this was working at some point, but can't figure out why it's stopped.
Any ideas? Does it look like I have configured it wrong?
AppBatchConfiguration.java
#Configuration
#EnableBatchProcessing
#ComponentScan(basePackages = "our.org.base")
public class AppBatchConfiguration {
private final static SimpleLogger LOGGER = SimpleLogger.getInstance(AppBatchConfiguration.class);
private final static String OUTPUT_XML_FILE_PATH_PLACEHOLDER = null;
private final static String INPUT_XML_FILE_PATH_PLACEHOLDER = null;
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean(name = "cimAppXmlReader")
#StepScope
public <T> ItemStreamReader<T> appXmlReader(#Value("#{jobParameters[inputXmlFilePath]}")
String inputXmlFilePath) {
LOGGER.info("Job Parameter => App XML File Path :" + inputXmlFilePath);
StaxEventItemReader<T> reader = new StaxEventItemReader<T>();
reader.setResource(new FileSystemResource(inputXmlFilePath));
reader.setUnmarshaller(mecaUnMarshaller());
reader.setFragmentRootElementNames(getAppRootElementNames());
reader.setSaveState(false);
// Make the StaxEventItemReader thread-safe
SynchronizedItemStreamReader<T> synchronizedItemStreamReader = new SynchronizedItemStreamReader<T>();
synchronizedItemStreamReader.setDelegate(reader);
return synchronizedItemStreamReader;
}
#Bean
#StepScope
public ItemStreamReader<JAXBElement<AppIBTransactionHeaderType>> appXmlTransactionHeaderReader(#Value("#{jobParameters[inputXmlFilePath]}")
String inputXmlFilePath) {
LOGGER.info("Job Parameter => App XML File Path for Transaction Header :" + inputXmlFilePath);
StaxEventItemReader<JAXBElement<AppIBTransactionHeaderType>> reader = new StaxEventItemReader<>();
reader.setResource(new FileSystemResource(inputXmlFilePath));
reader.setUnmarshaller(mecaUnMarshaller());
String[] fragmentRootElementNames = new String[] {"AppIBTransactionHeader"};
reader.setFragmentRootElementNames(fragmentRootElementNames);
reader.setSaveState(false);
return reader;
}
#Bean
public Unmarshaller mecaUnMarshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan(ObjectFactory.class.getPackage().getName());
return marshaller;
}
#Bean
public Marshaller uberMarshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setClassesToBeBound(ServiceRequestType.class);
marshaller.setSupportJaxbElementClass(true);
return marshaller;
}
#Bean(destroyMethod="") // To stop multiple close calls, see: http://stackoverflow.com/a/23089536
#StepScope
public ResourceAwareItemWriterItemStream<JAXBElement<ServiceRequestType>> writer(#Value("#{jobParameters[outputXmlFilePath]}")
String outputXmlFilePath) {
SyncStaxEventItemWriter<JAXBElement<ServiceRequestType>> writer = new SyncStaxEventItemWriter<JAXBElement<ServiceRequestType>>();
writer.setResource(new FileSystemResource(outputXmlFilePath));
writer.setMarshaller(uberMarshaller());
writer.setSaveState(false);
HashMap<String, String> rootElementAttribs = new HashMap<String, String>();
rootElementAttribs.put("xmlns:ns1", "http://some.org/corporate/message/2010/1");
writer.setRootElementAttributes(rootElementAttribs);
writer.setRootTagName("ns1:SetOfServiceRequests");
return writer;
}
#Bean
#StepScope
public <T> ItemProcessor<T, JAXBElement<ServiceRequestType>> appNotificationProcessor() {
return new AppBatchNotificationItemProcessor<T>();
}
#Bean
public ItemProcessor<JAXBElement<AppIBTransactionHeaderType>, Boolean> appBatchCreationProcessor() {
return new AppBatchCreationItemProcessor();
}
public String[] getAppRootElementNames() {
//get list of App Transaction Element Names
return AppProcessorEnum.getValues();
}
#Bean
public Step AppStep() {
// INPUT_XML_FILE_PATH_PLACEHOLDER and OUTPUT_XML_FILE_PATH_PLACEHOLDER will be overridden
// by injected jobParameters using late binding (StepScope)
return stepBuilderFactory.get("AppStep")
.<Object, JAXBElement<ServiceRequestType>> chunk(10)
.reader(appXmlReader(INPUT_XML_FILE_PATH_PLACEHOLDER))
.processor(appNotificationProcessor())
.writer(writer(OUTPUT_XML_FILE_PATH_PLACEHOLDER))
.taskExecutor(concurrentTaskExecutor())
.throttleLimit(1)
.build();
}
#Bean
public Step BatchCreationStep() {
return stepBuilderFactory.get("BatchCreationStep")
.<JAXBElement<AppIBTransactionHeaderType>, Boolean>chunk(1)
.reader(appXmlTransactionHeaderReader(INPUT_XML_FILE_PATH_PLACEHOLDER))
.processor(appBatchCreationProcessor())
.taskExecutor(concurrentTaskExecutor())
.throttleLimit(1)
.build();
}
#Bean
public Job AppJob() {
return jobBuilderFactory.get("AppJob")
.incrementer(new RunIdIncrementer())
.listener(AppJobCompletionNotificationListener())
.flow(AppStep())
.next(BatchCreationStep())
.end()
.build();
}
#Bean
public JobCompletionNotificationListener AppJobCompletionNotificationListener() {
return new JobCompletionNotificationListener();
}
#Bean
public TaskExecutor concurrentTaskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(1);
return taskExecutor;
}
}
AppBatchNotificationItemProcessor.java
#StepScope
public class AppBatchNotificationItemProcessor<E> extends AppAbstractItemProcessor<E, JAXBElement<ServiceRequestType>> implements ItemProcessor<E, JAXBElement<ServiceRequestType>>, StepExecutionListener {
// This is populated correctly
#Value("#{jobExecutionContext['" + Constants.SECURITY_TOKEN + "']}")
private SecurityToken securityToken;
#Autowired
private AppProcessorService processor;
#Override
public JAXBElement<ServiceRequestType> process(E item) throws BPException {
// Do Stuff
return srRequest;
}
#BeforeProcess
public void beforeProcess(E item) {
System.out.println("Doesn't execute");
}
#Override
public void beforeStep(StepExecution stepExecution) {
// Doesn't execute
System.out.println("Doesn't execute");
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
// Doesn't execute
System.out.println("Doesn't execute");
}
}
This is due to the fact that you are returning interfaces instead of implementations in your #Bean methods. IMHO, you should return the most specific type possible when using java configuration in Spring. Here's why:
When configuring via XML, you provide the class in the XML configuration. This exposes the implementation to Spring so that any interfaces the class implements can be discovered and handled appropriately. When using java configuration, the return type of the #Bean method serves as the replacement for that information. And there is the issue. If your return type is an interface, Spring only knows about that specific interface and not all the interfaces an implementation may implement. By returning the concrete type where you can, you give Spring insight into what you're actually returning so it can better handle the various registration and wiring use cases for you.
For your specific example, since you're returning an ItemProcessor and it's step scoped (therefore proxied), all Spring knows about are the methods/behaviors expected with the ItemProcessor interface. If you return the implementation (AppBatchNotificationItemProcessor), other behaviors can be autoconfigured.
As far as I remember, you have to register a reader, writer, processor directly as listener on the step, if you use StepScope.
StepScope prevents the framework from being able to figure out what kind of interfaces, resp. #annotations (e.g.#BeforeProcess) the proxy actually implements/defines and therefore it is not able to register it as a listener.
So, I assume if add
return stepBuilderFactory.get("AppStep")
.<Object, JAXBElement<ServiceRequestType>> chunk(10)
.reader(appXmlReader(INPUT_XML_FILE_PATH_PLACEHOLDER))
.processor(appNotificationProcessor())
.writer(writer(OUTPUT_XML_FILE_PATH_PLACEHOLDER))
.listener(appNotificationProcessor())
.taskExecutor(concurrentTaskExecutor())
.throttleLimit(1)
.build();
it will work.
I have a simple csv file with ~400,000 line(one column only)
It takes me alot of time to read the records and process them
the processor validating records against couchbase
the writer - writing into remote topic
Takes me around 30 mins. thats insane.
I read that flatfileItemreader is not thread safe. so my chunk value is 1.
I read the Asynchronous processing could assist. but I cant see any improvements.
Thats my code:
#Configuration
#EnableBatchProcessing
public class NotificationFileProcessUploadedFileJob {
#Value("${expected.snid.header}")
public String snidHeader;
#Value("${num.of.processing.chunks.per.file}")
public int numOfProcessingChunksPerFile;
#Autowired
private InfrastructureConfigurationConfig infrastructureConfigurationConfig;
private static final String OVERRIDDEN_BY_EXPRESSION = null;
#Inject
private JobBuilderFactory jobs;
#Inject
private StepBuilderFactory stepBuilderFactory;
#Inject
ExecutionContextPromotionListener executionContextPromotionListener;
#Bean
public Job processUploadedFileJob() throws Exception {
return this.jobs.get("processUploadedFileJob").start((processSnidUploadedFileStep())).build();
}
#Bean
public Step processSnidUploadedFileStep() {
return stepBuilderFactory.get("processSnidFileStep")
.<PushItemDTO, PushItemDTO>chunk(numOfProcessingChunksPerFile)
.reader(snidFileReader(OVERRIDDEN_BY_EXPRESSION))
.processor(asyncItemProcessor())
.writer(asyncItemWriter())
// .throttleLimit(20)
// .taskJobExecutor(infrastructureConfigurationConfig.taskJobExecutor())
// .faultTolerant()
// .skipLimit(10) //default is set to 0
// .skip(MySQLIntegrityConstraintViolationException.class)
.build();
}
#Inject
ItemWriter writer;
#Bean
public AsyncItemWriter asyncItemWriter() {
AsyncItemWriter asyncItemWriter=new AsyncItemWriter();
asyncItemWriter.setDelegate(writer);
return asyncItemWriter;
}
#Bean
#Scope(value = "step", proxyMode = ScopedProxyMode.INTERFACES)
public ItemStreamReader<PushItemDTO> snidFileReader(#Value("#{jobParameters[filePath]}") String filePath) {
FlatFileItemReader<PushItemDTO> itemReader = new FlatFileItemReader<PushItemDTO>();
itemReader.setLineMapper(snidLineMapper());
itemReader.setLinesToSkip(1);
itemReader.setResource(new FileSystemResource(filePath));
return itemReader;
}
#Bean
public AsyncItemProcessor asyncItemProcessor() {
AsyncItemProcessor<PushItemDTO, PushItemDTO> asyncItemProcessor = new AsyncItemProcessor();
asyncItemProcessor.setDelegate(processor(OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION,
OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION, OVERRIDDEN_BY_EXPRESSION));
asyncItemProcessor.setTaskExecutor(infrastructureConfigurationConfig.taskProcessingExecutor());
return asyncItemProcessor;
}
#Scope(value = "step", proxyMode = ScopedProxyMode.INTERFACES)
#Bean
public ItemProcessor<PushItemDTO, PushItemDTO> processor(#Value("#{jobParameters[pushMessage]}") String pushMessage,
#Value("#{jobParameters[jobId]}") String jobId,
#Value("#{jobParameters[taskId]}") String taskId,
#Value("#{jobParameters[refId]}") String refId,
#Value("#{jobParameters[url]}") String url,
#Value("#{jobParameters[targetType]}") String targetType,
#Value("#{jobParameters[gameType]}") String gameType) {
return new PushItemProcessor(pushMessage, jobId, taskId, refId, url, targetType, gameType);
}
#Bean
public LineMapper<PushItemDTO> snidLineMapper() {
DefaultLineMapper<PushItemDTO> lineMapper = new DefaultLineMapper<PushItemDTO>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(true);
lineTokenizer.setStrict(true);
String[] splittedHeader = snidHeader.split(",");
lineTokenizer.setNames(splittedHeader);
BeanWrapperFieldSetMapper<PushItemDTO> fieldSetMapper = new BeanWrapperFieldSetMapper<PushItemDTO>();
fieldSetMapper.setTargetType(PushItemDTO.class);
lineMapper.setLineTokenizer(lineTokenizer);
lineMapper.setFieldSetMapper(new PushItemFieldSetMapper());
return lineMapper;
}
}
#Bean
#Override
public SimpleAsyncTaskExecutor taskProcessingExecutor() {
SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
simpleAsyncTaskExecutor.setConcurrencyLimit(300);
return simpleAsyncTaskExecutor;
}
How do you think I could improve the processing performances and make them faster?
thank you
ItemWriter code:
#Bean
public ItemWriter writer() {
return new KafkaWriter();
}
public class KafkaWriter implements ItemWriter<PushItemDTO> {
private static final Logger logger = LoggerFactory.getLogger(KafkaWriter.class);
#Autowired
KafkaProducer kafkaProducer;
#Override
public void write(List<? extends PushItemDTO> items) throws Exception {
for (PushItemDTO item : items) {
try {
logger.debug("Writing to kafka=" + item);
sendMessageToKafka(item);
} catch (Exception e) {
logger.error("Error writing item=" + item.toString(), e);
}
}
}
Increasing your commit count is where I'd begin. Keep in mind what the commit count means. Since you have it set at 1, you are doing the following for each item:
Start a transaction
Read an item
Process the item
Write the item
Update the job repository
Commit the transaction
Your configuration doesn't show what the delegate ItemWriter is so I can't tell, but at a minimum you are executing multiple SQL statements per item to update the job repository.
You are correct in that the FlatFileItemReader is not thread safe. However, you aren't using multiple threads to read, only process so there is no reason to set the commit count to 1 from what I can see.