I am using an api to call the springbatch job. when i call for the first time it works fine. second time gives the below error.
Caused by: java.lang.IllegalArgumentException: Sheet index (3) is out of range (0..2).
Below is my code.-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Config:
#Configuration
#EnableBatchProcessing
#PropertySource("classpath:batch.properties")
public class ExcelFileToDatabaseJobConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
ItemReader<StudentDTO> excelStudentReader() throws Exception {
System.out.println("inside excelStudentReader");
PoiItemReader<StudentDTO> reader = new PoiItemReader<>();
reader.setLinesToSkip(1);
//reader.setResource(new FileSystemResource("C:/Users/lenovo/IdeaProjects/test/out/production/classes/Price Change.xlsx"));
reader.setResource(new UrlResource("file:///C:/Users/lenovo/IdeaProjects/test/out/production/classes/inserttest.xlsx"));
reader.setRowMapper(excelRowMapper());
return reader;
}
private RowMapper<StudentDTO> excelRowMapper() {
System.out.println("inside excelRowMapper");
BeanWrapperRowMapper<StudentDTO> rowMapper = new BeanWrapperRowMapper<>();
rowMapper.setTargetType(StudentDTO.class);
return rowMapper;
}
#Bean
public JdbcBatchItemWriter<StudentDTO> writer(DataSource dataSource) {
System.out.println("inside writer");
return new JdbcBatchItemWriterBuilder<StudentDTO>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO tbl_item_master (mrp_rate,price,item_code) VALUES (:mrp_rate,:rate,:u_item_code)")
//.sql("update ignore tbl_item_master set mrp_rate=:mrp_rate,price=:rate where item_code=:u_item_code")
.dataSource(dataSource)
.build();
}
#Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
System.out.println("inside importUserJob");
return jobBuilderFactory.get("importUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
}
#Bean
public Step step1(JdbcBatchItemWriter<StudentDTO> writer,CustomChunkListener customChunkListener,
CustomItemReaderListener customItemReaderListener,CustomItemWriterListener customItemWriterListener,
CustomStepListener customStepListener,CustomSkipListener customSkipListener) throws Exception {
System.out.println("inside step1");
return stepBuilderFactory.get("step1")
.<StudentDTO, StudentDTO>chunk(3000)
.reader(excelStudentReader())
//.processor(processor())
.writer(writer)
.faultTolerant()
.skip(Exception.class)
.noRetry(Exception.class)
.noRollback(Exception.class)
.skipLimit(100)
.listener(customItemWriterListener)
.listener(customSkipListener)
.build();
}
#Bean
public CustomStepListener customStepListener() {
return new CustomStepListener();
}
#Bean
public CustomItemWriterListener itemWriterListener() {
return new CustomItemWriterListener();
}
#Bean
public CustomItemReaderListener itemReaderListener() {
return new CustomItemReaderListener();
}
#Bean
public CustomSkipListener customSkipListener() {
return new CustomSkipListener();
}
}
Controller:
#Controller
public class FileController1 {
#Autowired
private JobLauncher jobLauncher;
#Autowired
private Job importUserJob;
#RequestMapping(value = "/echofile", method = RequestMethod.POST, produces = {"application/json"})
public #ResponseBody String echoFile(MultipartHttpServletRequest request,
HttpServletResponse response) throws Exception {
MultipartFile multipartFile = request.getFile("file");
String path = new ClassPathResource("/").getURL().getPath();
File fileToImport = new File(path + multipartFile.getOriginalFilename());
//filePath = fileToImport.getAbsolutePath();
OutputStream outputStream = new FileOutputStream(fileToImport);
IOUtils.copy(multipartFile.getInputStream(), outputStream);
outputStream.flush();
outputStream.close();
//Launch the Batch Job
JobExecution jobExecution = jobLauncher.run(importUserJob,new JobParametersBuilder()
.addString("fullPathFileName", fileToImport.getAbsolutePath()).addLong("time",System.currentTimeMillis()).toJobParameters());
return "something";
}
}
batch.properties:
spring.batch.job.enabled=false
Related
I am trying to load set of csv file into mysql using spring batch.
My code is given below. But when I run with 2 or 3 files the data is loading fine. but when I try with 100 or more Its trowing an execption aftre loading 4 to 8 files data.
Exception:
ERROR 11633 --- [ main] o.s.batch.core.step.AbstractStep : Encountered an error executing step masterStep in job importUserJob
org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:112) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:66) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:136) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:797) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:781) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:137) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
I don't know whats happening behind, Any help appreciated.
Code:
#Configuration
#EnableBatchProcessing
public class ImportJobConfig {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private DataSource dataSource;
#Autowired
private JdbcBatchItemWriter<Person> writer;
#Autowired
private FlatFileItemReader<Person> personItemReader;
#Bean
public DataSource dataSource() {
final DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("URL");
dataSource.setUsername("USERNAME");
dataSource.setPassword("PASSWORD");
return dataSource;
}
#Bean
public ThreadPoolTaskExecutor taskExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(4);
taskExecutor.setCorePoolSize(4);
taskExecutor.setQueueCapacity(4);
taskExecutor.afterPropertiesSet();
taskExecutor.setAllowCoreThreadTimeOut(true);
return taskExecutor;
}
#Bean
#Qualifier("masterStep")
public Step masterStep() {
return stepBuilderFactory.get("masterStep").partitioner("step1", partitioner()).step(step1())
.taskExecutor(taskExecutor()).build();
}
#Bean("partitioner")
#StepScope
public Partitioner partitioner() {
System.out.println("In Partitioner");
MultiResourcePartitioner partitioner = new MultiResourcePartitioner();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
String filePath = "//data//person//*.csv";
Resource[] resources = null;
try {
resources = resolver.getResources("file:" + filePath);
} catch (IOException e) {
e.printStackTrace();
}
partitioner.setResources(resources);
partitioner.partition(4);
return partitioner;
}
#Bean
public APSUploadFileItemProcessor processor() {
return new APSUploadFileItemProcessor();
}
#Bean
public JdbcBatchItemWriter<Person> writer() {
JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<>();
try {
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Person>());
writer.setSql("INSERT INTO person(name,business,phone) VALUES(name, :business, :phone)");
writer.setDataSource(dataSource);
} catch (Exception e) {
e.printStackTrace();
}
return writer;
}
#Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
return jobBuilderFactory.get("importUserJob").incrementer(new RunIdIncrementer()).listener(listener)
.flow(masterStep()).end().build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1").<Person, Person>chunk(10000).processor(processor())
.writer(writer).reader(personItemReader).build();
}
#Bean
#StepScope
#Qualifier("personItemReader")
#DependsOn("partitioner")
public FlatFileItemReader<Person> personItemReader(
#Value("#{stepExecutionContext['fileName']}") String filename) throws MalformedURLException {
return new FlatFileItemReaderBuilder<Person>().name("personItemReader").delimited().delimiter("|")
.names(new String[] { "name","business","phone" })
.fieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {
{
setTargetType(Person.class);
}
}).resource(new UrlResource(filename)).build();
}
}
I have implemented an IntegrationFlow where I want to do to following tasks:
Poll for files from a directory
Transform the file content to a string
Send the string via WebFluxRequestExecutingMessageHandler to a REST-Endpoint and use an AdviceChain to handle success and error responses
Implementation
#Configuration
#Slf4j
public class JsonToRestIntegration {
#Autowired
private LoadBalancerExchangeFilterFunction lbFunction;
#Value("${json_folder}")
private String jsonPath;
#Value("${json_success_folder}")
private String jsonSuccessPath;
#Value("${json_error_folder}")
private String jsonErrorPath;
#Value("${rest-service-url}")
private String restServiceUrl;
#Bean
public DirectChannel httpResponseChannel() {
return new DirectChannel();
}
#Bean
public MessageChannel successChannel() {
return new DirectChannel();
}
#Bean
public MessageChannel failureChannel() {
return new DirectChannel();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedDelay(1000).get();
}
#Bean
public IntegrationFlow jsonFileToRestFlow() {
return IntegrationFlows
.from(fileReadingMessageSource(), e -> e.id("fileReadingEndpoint"))
.transform(org.springframework.integration.file.dsl.Files.toStringTransformer())
.enrichHeaders(s -> s.header("Content-Type", "application/json; charset=utf8"))
.handle(reactiveOutbound())
.log()
.channel(httpResponseChannel())
.get();
}
#Bean
public FileReadingMessageSource fileReadingMessageSource() {
FileReadingMessageSource source = new FileReadingMessageSource();
source.setDirectory(new File(jsonPath));
source.setFilter(new SimplePatternFileListFilter("*.json"));
source.setUseWatchService(true);
source.setWatchEvents(FileReadingMessageSource.WatchEventType.CREATE);
return source;
}
#Bean
public MessageHandler reactiveOutbound() {
WebClient webClient = WebClient.builder()
.baseUrl("http://jsonservice")
.filter(lbFunction)
.build();
WebFluxRequestExecutingMessageHandler handler = new WebFluxRequestExecutingMessageHandler(restServiceUrl, webClient);
handler.setHttpMethod(HttpMethod.POST);
handler.setCharset(StandardCharsets.UTF_8.displayName());
handler.setOutputChannel(httpResponseChannel());
handler.setExpectedResponseType(String.class);
handler.setAdviceChain(singletonList(expressionAdvice()));
return handler;
}
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setTrapException(true);
advice.setSuccessChannel(successChannel());
advice.setOnSuccessExpressionString("payload + ' war erfolgreich'");
advice.setFailureChannel(failureChannel());
advice.setOnFailureExpressionString("payload + ' war nicht erfolgreich'");
return advice;
}
#Bean
public IntegrationFlow loggingFlow() {
return IntegrationFlows.from(httpResponseChannel())
.handle(message -> {
String originalFileName = (String) message.getHeaders().get(FileHeaders.FILENAME);
log.info("some log");
})
.get();
}
#Bean
public IntegrationFlow successFlow() {
return IntegrationFlows.from(successChannel())
.handle(message -> {
MessageHeaders messageHeaders = ((AdviceMessage) message).getInputMessage().getHeaders();
File originalFile = (File) messageHeaders.get(ORIGINAL_FILE);
String originalFileName = (String) messageHeaders.get(FILENAME);
if (originalFile != null && originalFileName != null) {
File jsonSuccessFolder = new File(jsonSuccessPath);
File jsonSuccessFile = new File(jsonSuccessFolder, originalFileName);
try {
Files.move(originalFile.toPath(), jsonSuccessFile.toPath());
} catch (IOException e) {
log.error("some log", e);
}
}
})
.get();
}
#Bean
public IntegrationFlow failureFlow() {
return IntegrationFlows.from(failureChannel())
.handle(message -> {
Message<?> failedMessage = ((MessagingException) message.getPayload()).getFailedMessage();
if (failedMessage != null) {
File originalFile = (File) failedMessage.getHeaders().get(FileHeaders.ORIGINAL_FILE);
String originalFileName = (String) failedMessage.getHeaders().get(FileHeaders.FILENAME);
if (originalFile != null && originalFileName != null) {
File jsonErrorFolder = new File(tonisJsonErrorPath);
File jsonErrorFile = new File(jsonErrorFolder, originalFileName);
try {
Files.move(originalFile.toPath(), jsonErrorFile.toPath());
} catch (IOException e) {
log.error("some log", e);
}
}
}
})
.get();
}
}
So far it seems to work in production. In the test I want to do the following steps:
Copy JSON-Files to the input directory
Start the polling for the json files
Do assertions on the HTTP-Response from the WebFluxRequestExecutingMessageHandler which are routed through my advice chain
But I'm struggling in the test with the following tasks:
Mocking the WebFluxRequestExecutingMessageHandler with the MockIntegrationContext.substituteMessageHandlerFor()-method
Manually start the polling of the json files
Test
#RunWith(SpringRunner.class)
#SpringIntegrationTest()
#Import({JsonToRestIntegration.class})
#JsonTest
public class JsonToRestIntegrationTest {
#Autowired
public DirectChannel httpResponseChannel;
#Value("${json_folder}")
private String jsonPath;
#Value("${json_success_folder}")
private String jsonSuccessPath;
#Value("${json_error_folder}")
private String jsonErrorPath;
#Autowired
private MockIntegrationContext mockIntegrationContext;
#Autowired
private MessageHandler reactiveOutbound;
#Before
public void setUp() throws Exception {
Files.createDirectories(Paths.get(jsonPath));
Files.createDirectories(Paths.get(jsonSuccessPath));
Files.createDirectories(Paths.get(jsonErrorPath));
}
#After
public void tearDown() throws Exception {
FileUtils.deleteDirectory(new File(jsonPath));
FileUtils.deleteDirectory(new File(jsonSuccessPath));
FileUtils.deleteDirectory(new File(jsonErrorPath));
}
#Test
public void shouldSendJsonToRestEndpointAndReceiveOK() throws Exception {
File jsonFile = new ClassPathResource("/test.json").getFile();
Path targetFilePath = Paths.get(jsonPath + "/" + jsonFile.getName());
Files.copy(jsonFile.toPath(), targetFilePath);
httpResponseChannel.subscribe(httpResponseHandler());
this.mockIntegrationContext.substituteMessageHandlerFor("", reactiveOutbound);
}
private MessageHandler httpResponseHandler() {
return message -> Assert.assertThat(message.getPayload(), is(notNullValue()));
}
#Configuration
#Import({JsonToRestIntegration.class})
public static class JsonToRestIntegrationTest {
#Autowired
public MessageChannel httpResponseChannel;
#Bean
public MessageHandler reactiveOutbound() {
ArgumentCaptor<Message<?>> messageArgumentCaptor = ArgumentCaptor.forClass(Message.class);
MockMessageHandler mockMessageHandler = mockMessageHandler(messageArgumentCaptor).handleNextAndReply(m -> m);
mockMessageHandler.setOutputChannel(httpResponseChannel);
return mockMessageHandler;
}
}
}
Updated Working Example with mocked WebFluX web client:
Implementation
public class JsonToRestIntegration {
private final LoadBalancerExchangeFilterFunction lbFunction;
private final BatchConfigurationProperties batchConfigurationProperties;
#Bean
public DirectChannel httpResponseChannel() {
return new DirectChannel();
}
#Bean
public DirectChannel errorChannel() {
return new DirectChannel();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedDelay(100, TimeUnit.MILLISECONDS).get();
}
#Bean
public IntegrationFlow jsonFileToRestFlow() {
return IntegrationFlows
.from(fileReadingMessageSource(), e -> e.id("fileReadingEndpoint"))
.transform(org.springframework.integration.file.dsl.Files.toStringTransformer("UTF-8"))
.enrichHeaders(s -> s.header("Content-Type", "application/json; charset=utf8"))
.handle(reactiveOutbound())
.channel(httpResponseChannel())
.get();
}
#Bean
public FileReadingMessageSource fileReadingMessageSource() {
FileReadingMessageSource source = new FileReadingMessageSource();
source.setDirectory(new File(batchConfigurationProperties.getJsonImportFolder()));
source.setFilter(new SimplePatternFileListFilter("*.json"));
source.setUseWatchService(true);
source.setWatchEvents(FileReadingMessageSource.WatchEventType.CREATE);
return source;
}
#Bean
public WebFluxRequestExecutingMessageHandler reactiveOutbound() {
WebClient webClient = WebClient.builder()
.baseUrl("http://service")
.filter(lbFunction)
.build();
WebFluxRequestExecutingMessageHandler handler = new WebFluxRequestExecutingMessageHandler(batchConfigurationProperties.getServiceUrl(), webClient);
handler.setHttpMethod(HttpMethod.POST);
handler.setCharset(StandardCharsets.UTF_8.displayName());
handler.setOutputChannel(httpResponseChannel());
handler.setExpectedResponseType(String.class);
handler.setAdviceChain(singletonList(expressionAdvice()));
return handler;
}
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setTrapException(true);
advice.setFailureChannel(errorChannel());
return advice;
}
#Bean
public IntegrationFlow responseFlow() {
return IntegrationFlows.from(httpResponseChannel())
.handle(message -> {
MessageHeaders messageHeaders = message.getHeaders();
File originalFile = (File) messageHeaders.get(ORIGINAL_FILE);
String originalFileName = (String) messageHeaders.get(FILENAME);
if (originalFile != null && originalFileName != null) {
File jsonSuccessFolder = new File(batchConfigurationProperties.getJsonSuccessFolder());
File jsonSuccessFile = new File(jsonSuccessFolder, originalFileName);
try {
Files.move(originalFile.toPath(), jsonSuccessFile.toPath());
} catch (IOException e) {
log.error("Could not move file", e);
}
}
})
.get();
}
#Bean
public IntegrationFlow failureFlow() {
return IntegrationFlows.from(errorChannel())
.handle(message -> {
Message<?> failedMessage = ((MessagingException) message.getPayload()).getFailedMessage();
if (failedMessage != null) {
File originalFile = (File) failedMessage.getHeaders().get(ORIGINAL_FILE);
String originalFileName = (String) failedMessage.getHeaders().get(FILENAME);
if (originalFile != null && originalFileName != null) {
File jsonErrorFolder = new File(batchConfigurationProperties.getJsonErrorFolder());
File jsonErrorFile = new File(jsonErrorFolder, originalFileName);
try {
Files.move(originalFile.toPath(), jsonErrorFile.toPath());
} catch (IOException e) {
log.error("Could not move file", originalFileName, e);
}
}
}
})
.get();
}
}
Test
#RunWith(SpringRunner.class)
#SpringIntegrationTest(noAutoStartup = "fileReadingEndpoint")
#Import({JsonToRestIntegration.class, BatchConfigurationProperties.class})
#JsonTest
#DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD)
public class JsonToRestIntegrationIT {
private static final FilenameFilter JSON_FILENAME_FILTER = (dir, name) -> name.endsWith(".json");
#Autowired
private BatchConfigurationProperties batchConfigurationProperties;
#Autowired
private ObjectMapper om;
#Autowired
private MessageHandler reactiveOutbound;
#Autowired
private DirectChannel httpResponseChannel;
#Autowired
private DirectChannel errorChannel;
#Autowired
private FileReadingMessageSource fileReadingMessageSource;
#Autowired
private SourcePollingChannelAdapter fileReadingEndpoint;
#MockBean
private LoadBalancerExchangeFilterFunction lbFunction;
private String jsonImportPath;
private String jsonSuccessPath;
private String jsonErrorPath;
#Before
public void setUp() throws Exception {
jsonImportPath = batchConfigurationProperties.getJsonImportFolder();
jsonSuccessPath = batchConfigurationProperties.getJsonSuccessFolder();
jsonErrorPath = batchConfigurationProperties.getJsonErrorFolder();
Files.createDirectories(Paths.get(jsonImportPath));
Files.createDirectories(Paths.get(jsonSuccessPath));
Files.createDirectories(Paths.get(jsonErrorPath));
}
#After
public void tearDown() throws Exception {
FileUtils.deleteDirectory(new File(jsonImportPath));
FileUtils.deleteDirectory(new File(jsonSuccessPath));
FileUtils.deleteDirectory(new File(jsonErrorPath));
}
#Test
public void shouldMoveJsonFileToSuccessFolderWhenHttpResponseIsOk() throws Exception {
final CountDownLatch latch = new CountDownLatch(1);
httpResponseChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
latch.countDown();
super.postSend(message, channel, sent);
}
});
errorChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
fail();
}
});
ClientHttpConnector httpConnector = new HttpHandlerConnector((request, response) -> {
response.setStatusCode(HttpStatus.OK);
response.getHeaders().setContentType(MediaType.APPLICATION_JSON_UTF8);
DataBufferFactory bufferFactory = response.bufferFactory();
String valueAsString = null;
try {
valueAsString = om.writeValueAsString(new ResponseDto("1"));
} catch (JsonProcessingException e) {
fail();
}
return response.writeWith(Mono.just(bufferFactory.wrap(valueAsString.getBytes())))
.then(Mono.defer(response::setComplete));
});
WebClient webClient = WebClient.builder()
.clientConnector(httpConnector)
.build();
new DirectFieldAccessor(this.reactiveOutbound)
.setPropertyValue("webClient", webClient);
File jsonFile = new ClassPathResource("/test.json").getFile();
Path targetFilePath = Paths.get(jsonImportPath + "/" + jsonFile.getName());
Files.copy(jsonFile.toPath(), targetFilePath);
fileReadingEndpoint.start();
assertThat(latch.await(12, TimeUnit.SECONDS), is(true));
File[] jsonImportFolder = new File(jsonImportPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonImportFolder, is(notNullValue()));
assertThat(filesInJsonImportFolder.length, is(0));
File[] filesInJsonSuccessFolder = new File(jsonSuccessPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonSuccessFolder, is(notNullValue()));
assertThat(filesInJsonSuccessFolder.length, is(1));
File[] filesInJsonErrorFolder = new File(jsonErrorPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonErrorFolder, is(notNullValue()));
assertThat(filesInJsonErrorFolder.length, is(0));
}
#Test
public void shouldMoveJsonFileToErrorFolderWhenHttpResponseIsNotOk() throws Exception {
final CountDownLatch latch = new CountDownLatch(1);
errorChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
latch.countDown();
super.postSend(message, channel, sent);
}
});
httpResponseChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
fail();
}
});
ClientHttpConnector httpConnector = new HttpHandlerConnector((request, response) -> {
response.setStatusCode(HttpStatus.BAD_REQUEST);
response.getHeaders().setContentType(MediaType.APPLICATION_JSON_UTF8);
DataBufferFactory bufferFactory = response.bufferFactory();
return response.writeWith(Mono.just(bufferFactory.wrap("SOME BAD REQUEST".getBytes())))
.then(Mono.defer(response::setComplete));
});
WebClient webClient = WebClient.builder()
.clientConnector(httpConnector)
.build();
new DirectFieldAccessor(this.reactiveOutbound)
.setPropertyValue("webClient", webClient);
File jsonFile = new ClassPathResource("/error.json").getFile();
Path targetFilePath = Paths.get(jsonImportPath + "/" + jsonFile.getName());
Files.copy(jsonFile.toPath(), targetFilePath);
fileReadingEndpoint.start();
assertThat(latch.await(11, TimeUnit.SECONDS), is(true));
File[] filesInJsonImportFolder = new File(jsonImportPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonImportFolder, is(notNullValue()));
assertThat(filesInJsonImportFolder.length, is(0));
File[] filesInJsonSuccessFolder = new File(jsonSuccessPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonSuccessFolder, is(notNullValue()));
assertThat(filesInJsonSuccessFolder.length, is(0));
File[] filesInJsonErrorFolder = new File(jsonErrorPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonErrorFolder, is(notNullValue()));
assertThat(filesInJsonErrorFolder.length, is(1));
}
}
this.mockIntegrationContext.substituteMessageHandlerFor("", reactiveOutbound);
The first parameter of this method is an endpoint id. (I guess we are just missing Javadocs there on those methods...).
So, what you need is something like this:
.handle(reactiveOutbound(), e -> e.id("webFluxEndpoint"))
And then in that test-case you do:
this.mockIntegrationContext.substituteMessageHandlerFor("webFluxEndpoint", reactiveOutbound);
You don't need to override bean in the test class config. The MockMessageHandler can be just used in the test method body.
You poll files via .from(fileReadingMessageSource()). To do a manual control you need to have it stopped in the beginning. For this purpose you add an endpoint id again:
.from(fileReadingMessageSource(), e -> e.id("fileReadingEndpoint"))
And then in the test configuration you do this:
#SpringIntegrationTest(noAutoStartup = "fileReadingEndpoint")
Another approach for the WebFlux would be via customized WebClient to mock request to the server. For example:
ClientHttpConnector httpConnector = new HttpHandlerConnector((request, response) -> {
response.setStatusCode(HttpStatus.OK);
response.getHeaders().setContentType(MediaType.TEXT_PLAIN);
DataBufferFactory bufferFactory = response.bufferFactory();
return response.writeWith(Mono.just(bufferFactory.wrap("FOO\nBAR\n".getBytes())))
.then(Mono.defer(response::setComplete));
});
WebClient webClient = WebClient.builder()
.clientConnector(httpConnector)
.build();
new DirectFieldAccessor(this.reactiveOutbound)
.setPropertyValue("webClient", webClient);
I am using springbatch to do a batch update based on user input(file). But Step is not getting called due to which writer is not getting called. Also not getting any error or exception but the job completed successfully without Below is my code.
Technologies used: mysql,spring-batch,java,spring-boot,intellij idea
#Configuration
public class ExcelFileToDatabaseJobConfig
{
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
#StepScope
ItemReader<StudentDTO> excelStudentReader(#Value("#{jobParameters[fullPathFileName]}") String pathToFile) throws Exception{
PoiItemReader<StudentDTO> reader = new PoiItemReader<>();
try {
System.out.println("inside excelStudentReader");
reader.setLinesToSkip(1);
System.out.println("pathToFile = " + pathToFile);
reader.setResource(new ClassPathResource(pathToFile));
//reader.setResource(new UrlResource("file:///D:/joannes/ee.xlsx"));
reader.setRowMapper(excelRowMapper());
}catch (Exception ex)
{
ex.printStackTrace();
}
return reader;
}
private RowMapper<StudentDTO> excelRowMapper() {
System.out.println("inside excelRowMapper");
BeanWrapperRowMapper<StudentDTO> rowMapper = new BeanWrapperRowMapper<>();
rowMapper.setTargetType(StudentDTO.class);
return rowMapper;
}
#Bean
public JdbcBatchItemWriter<StudentDTO> writer(DataSource dataSource)throws Exception {
System.out.println("inside writer");
return new JdbcBatchItemWriterBuilder<StudentDTO>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO tbl_item_master (mrp_rate,price,item_code) VALUES (:mrp_rate,:rate,:u_item_code)")
//.sql("update tbl_item_master set mrp_rate=:mrp_rate,price=:rate where item_code=:u_item_code")
.dataSource(dataSource)
.build();
}
#Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1)throws Exception {
System.out.println("inside importUserJob");
return jobBuilderFactory.get("importUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
}
#Bean
public Step step1(JdbcBatchItemWriter<StudentDTO> writer,#Qualifier("excelStudentReader") ItemReader<StudentDTO> importReader)throws Exception {
System.out.println("inside step1");
return stepBuilderFactory.get("step1")
.<StudentDTO, StudentDTO> chunk(3000)
.reader(importReader)
//.processor(processor())
.writer(writer)
.build();
}
}
Controller Class:
#Controller
public class FileController1 {
#Autowired
private JobLauncher jobLauncher;
#Autowired
private Job importUserJob;
#RequestMapping(value = "/echofile", method = RequestMethod.POST, produces = {"application/json"})
//public #ResponseBody HashMap<String, Object> echoFile(MultipartHttpServletRequest request,
// HttpServletResponse response) throws Exception {
public #ResponseBody String echoFile(MultipartHttpServletRequest request,
HttpServletResponse response) throws Exception {
MultipartFile multipartFile = request.getFile("file");
/* Long size = multipartFile.getSize();
String contentType = multipartFile.getContentType();
InputStream stream = multipartFile.getInputStream();
byte[] bytes = IOUtils.toByteArray(stream);
FileUtils.writeByteArrayToFile(new File("D:/joannes/wow.xlsx"), bytes);
HashMap<String, Object> map = new HashMap<String, Object>();
map.put("fileoriginalsize", size);
map.put("contenttype", contentType);
map.put("base64", new String(Base64Utils.encode(bytes)));
*/
String path = new ClassPathResource("/").getURL().getPath();//it's assumed you have a folder called tmpuploads in the resources folder
File fileToImport = new File(path + multipartFile.getOriginalFilename());
//filePath = fileToImport.getAbsolutePath();
OutputStream outputStream = new FileOutputStream(fileToImport);
IOUtils.copy(multipartFile.getInputStream(), outputStream);
outputStream.flush();
outputStream.close();
//Launch the Batch Job
JobExecution jobExecution = jobLauncher.run(importUserJob,new JobParametersBuilder()
.addString("fullPathFileName", fileToImport.getAbsolutePath()).addLong("time",System.currentTimeMillis()).toJobParameters());
//return map;
return "something";
}
}
I'm new to the Spring Batch. I have written code for a spring batch job and it is working also. But it is not exiting after finishing the job.Even during debugging I cannot stop it by keyboard shortcut to exit.
If there is no file , it exits. But when the file is available it is ending after execution. I don't prefer to use System.exit.
Any help on this will be great and thanks in advance.
public class OfferRedemptionJobConfig {
#Bean
public Job RedemptionJob(){
Flow flow = new FlowBuilder<SimpleFlow>("RedemptionJobFlow")
.start(jobValidatorStep())
.next(fileDecisionDecider)
.on("COMPLETED")
.end()
.from(fileDecisionDecider)
.on("CONTINUE")
.to(RedemptionFileTaskletStep())
.next(redemptionFileReprocessStep())
.next(fileDecisionDecider)
.on("COMPLETED")
.end().build();
return jobBuilderFactory.get("RedemptionJob")
.incrementer(new RunIdIncrementer())
.listener(new DefaultJobListener())
.start(flow).end().build();
}
#Bean
public Step redemptionFileReprocessStep() {
return stepBuilderFactory.get("redemptionFileReprocessStep")
.<RedemptionVO, RedemptionVO>chunk(100)
.reader(RedemptionFileReader(WILL_BE_INJECTED))
.processor(FileProcessor(WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED))
.listener(RedemptionFileListenerStep(WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED))
.build();
}
#Bean
#StepScope
public FlatFileItemReader<RedemptionVO> RedemptionFileReader(#Value("#{jobExecutionContext['localDownloadedPath']}") String filepath) {
FlatFileItemReader<RedemptionVO> reader = new FlatFileItemReader<>();
logger.info("FILE READER FILE PATH {}",filepath);
reader.setResource(new FileSystemResource(new File(filepath)));
reader.setLineMapper(RedemptionFileLineMapper());
reader.setLinesToSkip(1);
return reader;
}
}
}
#Component
public class OfferRedemptionJob {
private static final Logger logger = LoggerFactory.getLogger(YTBSOfferRedemptionJob.class);
#Autowired
JobRequestLauncher jobRequestLauncher;
#Autowired
BatchJobConfigurationRepo batchJobConfigurationRepo;
#Autowired
CommonJobsHelper commonJobsHelper;
public void run() {
logger.info("Job is runnning");
List<BatchJobConfigurationEntity> batchJobConfigurationRepoList = batchJobConfigurationRepo.getAllActiveJobConfigsByJobName("ytbsRedemptionJob");
Map<String, String> jobConfigs = commonJobsHelper.extractConfigs(batchJobConfigurationRepoList);
if ((!jobConfigs.containsKey("enabled")) || jobConfigs.get("enabled").equalsIgnoreCase("true")) {
JobLaunchRequest jobLaunchRequest = new JobLaunchRequest("RedemptionJob", jobConfigs);
JobExecution jobExecution = jobRequestLauncher.launch(jobLaunchRequest);
logger.info("jobExecution ExitStatus {}", jobExecution.getExitStatus());
}
}
}
public class JobRequestLauncher {
private static final Logger logger = LoggerFactory.getLogger(JobRequestLauncher.class);
#Autowired
JobRegistry jobRegistry;
#Autowired
SimpleLauncher simpleLauncher;
public JobExecution launch(JobLaunchRequest request){
JobExecution jobExecution = null;
try {
Job job = jobRegistry.getJob(request.getJobName());
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
for (Map.Entry<String, String> entry : request.getJobParameters().entrySet()) {
jobParametersBuilder.addString(entry.getKey(),entry.getValue());
}
logger.info(jobParametersBuilder.toString());
jobParametersBuilder.addLong("jobStartTime",new Date().getTime());
jobExecution = simpleLauncher.launchJob(job,jobParametersBuilder.toJobParameters());
} catch (NoSuchJobException | JobInstanceAlreadyCompleteException | JobExecutionAlreadyRunningException | JobRestartException | JobParametersInvalidException e) {
logger.error(e.getClass().getSimpleName() + " occured while launching job: {} with params: {}. {}",request.getJobName(),request.getJobParameters(),e.getMessage(),e);
} catch(Exception e){
logger.error(e.getClass().getSimpleName() + " occured while launching job: {} with params: {}. {}",request.getJobName(),request.getJobParameters(),e.getMessage(),e);
}
return jobExecution;
}
}
#Component
public class SimpleLauncher {
#Autowired
BatchRepoConfig batchRepoConfig;
#Retryable()
public JobExecution launchJob(Job job, JobParameters jobParameters) throws Exception {
System.out.println(job.getName());
return batchRepoConfig.getJobLauncher().run(job,jobParameters);
}
}
Hello I have the follow Tasklet in my spring batch application:
public class FileMovingTasklet implements Tasklet, InitializingBean {
#Value("${positionFile.source-path}")
private String sourcePath;
#Value("${positionFile.local-path}")
private String localDirectory;
#Value("${positionFile.patternName}")
private String fileNamePattern;
#Value("${positionFile.suffix}")
private String suffix;
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
//***code***
List<PathResource> resources = FileManager.getInputFileList(sourcePath, fileNamePattern, suffix, fileDate);
if (!CollectionUtils.isEmpty(resources)) {
copyFiles(resources, localDirectory);
log.info("Copied files to local directory...");
}
} catch (IOException e) {
log.error("Position File not found with sourcePatch={}, fileName={}, suffix={}, filedate={}", sourcePath, fileNamePattern, suffix, fileDate);
throw new TaskletException("Could not move file from source to local " + e);
}
return RepeatStatus.FINISHED;
}
private void copyFiles(List<PathResource> resources, String localDirectory) {
for (Resource resource : resources) {
File source;
File target;
try {
source = resource.getFile();
target = new File(localDirectory + File.separator + source.getName());
try {
FileCopyUtils.copy(source, target);
} catch (IOException e) {
}
} catch (IOException e) {
}
}
}
}
I am moving some files from source to my local destination. The local destination is where I want to to read in files and process and write in my subsequent steps.
I have configured my Job as follows:
#Configuration
public class BatchConfiguration {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private FileMovingTasklet fileMovingTasklet;
#Value("${positionFile.local-path}")
private Resource[] resources;
#Autowired
private PriceDao scpDao;
#Autowired
public PositionRowReader positionRowReader;
#Bean
public MultiResourceItemReader<Pos> multiResourceItemReader() {
MultiResourceItemReader<Pos> resourceItemReader = new MultiResourceItemReader<>();
resourceItemReader.setResources(resources);
resourceItemReader.setDelegate(posRowReader());
return resourceItemReader;
}
#Bean
public FlatFileItemReader<Pos> posRowReader() {
return positionRowReader.getReader();
}
#Bean
public ItemProcessor<Pos, Price> posRowProcessor() {
return new PosRowItemProcessor();
}
#Bean
public JobExecutionListener listener() {
return new JobCompletionNotificationListener(scpDao);
}
#Bean
public Job import() {
return jobBuilderFactory.get("import")
.incrementer(new RunIdIncrementer())
.listener(listener())
.start(getPositionFileStep())
.next(step1())
.build();
}
#Bean
public Step getPositionFileStep() {
return stepBuilderFactory.get("getPositionFileStep")
.tasklet(fileMovingTasklet)
.build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<Pos, Price>chunk(50)
.reader(multiResourceItemReader())
.processor(posRowProcessor())
.writer(new PriceWriter(scpDao))
.build();
}
}
I get failed to initialize the reader when executing step1:
org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
my local-path property is as follows:
positionFile.local-path=C:\\Dev\\workspace\\batch\\src\\main\\resources\\localPath
positionFile.patterName=PositionFile*
my question is how can I access the resources (files) which can be multiple files in step1 after files have been copied over from source folder.
My resources size is 0 even though files are there:
#Value("${positionFile.local-path}")
private String filePath;
#Value("${positionFile.patternName}")
private String filePattern;
#Bean
#StepScope
public MultiResourceItemReader<PosRow> multiResourceItemReader() {
MultiResourceItemReader<PosRow> resourceItemReader = new MultiResourceItemReader<>();
Resource[] resources = new Resource[0];
try {
ClassLoader cl = this.getClass().getClassLoader();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);
resources = resolver.getResources("file:" + filePath + File.separator + filePattern + "*");
} catch (IOException e) {
log.error("Problem with getting resource files");
}
resourceItemReader.setResources(resources);
resourceItemReader.setDelegate(posRowReader());
return resourceItemReader;
}
You need to make your MultiResourceItemReader and FlatFileItemReader lazy by using #StepScope
#Bean
#StepScope
public MultiResourceItemReader<Pos> multiResourceItemReader() {
MultiResourceItemReader<Pos> resourceItemReader = new MultiResourceItemReader<>();
resourceItemReader.setResources(resources);
resourceItemReader.setDelegate(posRowReader());
return resourceItemReader;
}
#Bean
#StepScope
public FlatFileItemReader<Pos> posRowReader() {
return positionRowReader.getReader();
}