I am using springbatch to do a batch update based on user input(file). But Step is not getting called due to which writer is not getting called. Also not getting any error or exception but the job completed successfully without Below is my code.
Technologies used: mysql,spring-batch,java,spring-boot,intellij idea
#Configuration
public class ExcelFileToDatabaseJobConfig
{
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
#StepScope
ItemReader<StudentDTO> excelStudentReader(#Value("#{jobParameters[fullPathFileName]}") String pathToFile) throws Exception{
PoiItemReader<StudentDTO> reader = new PoiItemReader<>();
try {
System.out.println("inside excelStudentReader");
reader.setLinesToSkip(1);
System.out.println("pathToFile = " + pathToFile);
reader.setResource(new ClassPathResource(pathToFile));
//reader.setResource(new UrlResource("file:///D:/joannes/ee.xlsx"));
reader.setRowMapper(excelRowMapper());
}catch (Exception ex)
{
ex.printStackTrace();
}
return reader;
}
private RowMapper<StudentDTO> excelRowMapper() {
System.out.println("inside excelRowMapper");
BeanWrapperRowMapper<StudentDTO> rowMapper = new BeanWrapperRowMapper<>();
rowMapper.setTargetType(StudentDTO.class);
return rowMapper;
}
#Bean
public JdbcBatchItemWriter<StudentDTO> writer(DataSource dataSource)throws Exception {
System.out.println("inside writer");
return new JdbcBatchItemWriterBuilder<StudentDTO>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO tbl_item_master (mrp_rate,price,item_code) VALUES (:mrp_rate,:rate,:u_item_code)")
//.sql("update tbl_item_master set mrp_rate=:mrp_rate,price=:rate where item_code=:u_item_code")
.dataSource(dataSource)
.build();
}
#Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1)throws Exception {
System.out.println("inside importUserJob");
return jobBuilderFactory.get("importUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
}
#Bean
public Step step1(JdbcBatchItemWriter<StudentDTO> writer,#Qualifier("excelStudentReader") ItemReader<StudentDTO> importReader)throws Exception {
System.out.println("inside step1");
return stepBuilderFactory.get("step1")
.<StudentDTO, StudentDTO> chunk(3000)
.reader(importReader)
//.processor(processor())
.writer(writer)
.build();
}
}
Controller Class:
#Controller
public class FileController1 {
#Autowired
private JobLauncher jobLauncher;
#Autowired
private Job importUserJob;
#RequestMapping(value = "/echofile", method = RequestMethod.POST, produces = {"application/json"})
//public #ResponseBody HashMap<String, Object> echoFile(MultipartHttpServletRequest request,
// HttpServletResponse response) throws Exception {
public #ResponseBody String echoFile(MultipartHttpServletRequest request,
HttpServletResponse response) throws Exception {
MultipartFile multipartFile = request.getFile("file");
/* Long size = multipartFile.getSize();
String contentType = multipartFile.getContentType();
InputStream stream = multipartFile.getInputStream();
byte[] bytes = IOUtils.toByteArray(stream);
FileUtils.writeByteArrayToFile(new File("D:/joannes/wow.xlsx"), bytes);
HashMap<String, Object> map = new HashMap<String, Object>();
map.put("fileoriginalsize", size);
map.put("contenttype", contentType);
map.put("base64", new String(Base64Utils.encode(bytes)));
*/
String path = new ClassPathResource("/").getURL().getPath();//it's assumed you have a folder called tmpuploads in the resources folder
File fileToImport = new File(path + multipartFile.getOriginalFilename());
//filePath = fileToImport.getAbsolutePath();
OutputStream outputStream = new FileOutputStream(fileToImport);
IOUtils.copy(multipartFile.getInputStream(), outputStream);
outputStream.flush();
outputStream.close();
//Launch the Batch Job
JobExecution jobExecution = jobLauncher.run(importUserJob,new JobParametersBuilder()
.addString("fullPathFileName", fileToImport.getAbsolutePath()).addLong("time",System.currentTimeMillis()).toJobParameters());
//return map;
return "something";
}
}
Related
I have a requirement to process thousands of Payments..So I used VtdXml instead of StaxEventItemReader in Spring Batch, I have created Custom Item Reader for the same. In order to read huge xml with Multi Threading, I have created partition with 10 Threads. I am splitting huge xml file into 10 files and assigning to each Thread in Partition. Once I read the xml and I will convert into list of Objects and send to Writer. After received in Writer, I will customize the List of Objects and Merge into the final List. Whenever Am returning list of Objects read called again and it is never ending. How do I pass the list of Objects to the Writer and merge into the final List?
public class VtdWholeItemReader<T> implements ResourceAwareItemReaderItemStream<T> {
private Resource resource;
private boolean noInput;
private boolean strict = true;
private InputStream inputStream;
private int index = 0;
#Override
public void open(ExecutionContext executionContext) {
Assert.notNull(resource, "The Resource must not be null.");
noInput = true;
if (!resource.exists()) {
if (strict) {
throw new IllegalStateException("Input resource must exist (reader is in 'strict' mode)");
}
log.warn("Input resource does not exist " + resource.getDescription());
return;
}
if (!resource.isReadable()) {
if (strict) {
throw new IllegalStateException("Input resource must be readable (reader is in 'strict' mode)");
}
log.warn("Input resource is not readable " + resource.getDescription());
return;
}
noInput = false;
}
#Override
public void update(ExecutionContext executionContext) {
}
#Override
public void close() {
try {
if (inputStream != null) {
inputStream.close();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
inputStream = null;
}
}
#Override
public void setResource(Resource resource) {
this.resource = resource;
}
#Override
public T read()
throws java.lang.Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
if (noInput) {
return null;
}
List<Payment> paymentList = new ArrayList<Payment>();
try {
VTDGen vg = new VTDGen();
VTDGen vgHen = new VTDGen();
boolean headercheck = true;
if (vg.parseFile("src/main/resources/input/partitioner/" + resource.getFilename(), false)) {
VTDNav vn = vg.getNav();
AutoPilot ap = new AutoPilot(vn);
ap.selectXPath("/root/Payment");
// flb contains all the offset and length of the segments to be skipped
FastLongBuffer flb = new FastLongBuffer(4);
int i;
byte[] xml = vn.getXML().getBytes();
while ((i = ap.evalXPath()) != -1) {
flb.append(vn.getElementFragment());
}
int size = flb.size();
log.info("Payment Size {}", size);
if (size != 0) {
for (int k = 0; k < size; k++) {
String message = new String(xml, flb.lower32At(k), flb.upper32At(k), StandardCharsets.UTF_8);
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
Payment payment = objectMapper
.readValue(message, Payment.class);
paymentList.add(pcPayment);
index = pcPaymentList.size() + 1;
}
}
log.info("Payment List:: {}", paymentList.size());
log.info("Index::{}", index);
return index > paymentList .size() ? null : (T) paymentList;
}
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
}
SpringBatch ConfigClass
private final Logger logger = LoggerFactory.getLogger(SpringBatchConfig.class);
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
ResourcePatternResolver resoursePatternResolver;
#Bean
public Job job() {
return jobBuilderFactory.get("job").start(readpayment()).build();
}
#Bean
public JobLauncher jobLauncher() throws Exception {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository());
jobLauncher.afterPropertiesSet();
return jobLauncher;
}
#Bean
public JobRepository jobRepository() throws Exception {
MapJobRepositoryFactoryBean factory = new MapJobRepositoryFactoryBean();
factory.setTransactionManager(new ResourcelessTransactionManager());
return (JobRepository) factory.getObject();
}
#Bean
protected Step readpayment() {
return stepBuilderFactory.get("readpayment").partitioner("paymentStep", partitioner(null))
.step(paymentStep()).taskExecutor(taskExecutor()).build();
}
#Bean
protected Step paymentStep() {
return stepBuilderFactory.get("paymentStep")
.<Payment,Payment>chunk(10)
.reader(xmlFileItemReader(null))
.writer(writer()).build();
}
#Bean
public TaskExecutor taskExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(10);
taskExecutor.setCorePoolSize(10);
taskExecutor.setQueueCapacity(10);
taskExecutor.afterPropertiesSet();
return taskExecutor;
}
#Bean
#StepScope
ItemReader<Payment> xmlFileItemReader(#Value("#{stepExecutionContext[fileName]}") String filename) {
VtdWholeItemReader<Payment> xmlFileReader = new VtdWholeItemReader<>();
xmlFileReader.setResource(new ClassPathResource("input/partitioner/" + filename));
return xmlFileReader;
}
#Bean
#StepScope
public CustomMultiResourcePartitioner partitioner(#Value("#{jobParameters['fileName']}") String fileName) {
logger.info("fileName {}", fileName);
CustomMultiResourcePartitioner partitioner = new CustomMultiResourcePartitioner();
Resource[] resources;
try {
resources = resoursePatternResolver.getResources("file:src/main/resources/input/partitioner/*.xml");
} catch (IOException e) {
throw new RuntimeException("I/O problems when resolving the input file pattern.", e);
}
partitioner.setResources(resources);
return partitioner;
}
#Bean
public ItemWriter<Payment> writer() {
return new PaymentItemWriter();
}
PaymentItemWriter
#Override
public void write(List<? extends List<Payment>> items) throws Exception {
log.info("Items {}", items.size());
}
May be try making ItemWriter bean as step scope ["#StepScope"] in Spring batch configuration class
Make your xmlFileItemReader method return the actual type VtdWholeItemReader instead of the interface type ItemReader:
#Bean
#StepScope
VtdWholeItemReader<Payment> xmlFileItemReader(#Value("#{stepExecutionContext[fileName]}") String filename) {
VtdWholeItemReader<Payment> xmlFileReader = new VtdWholeItemReader<>();
xmlFileReader.setResource(new ClassPathResource("input/partitioner/" + filename));
return xmlFileReader;
}
This way, Spring will correctly proxy your reader as an ItemStreamReader (and not ItemReader) and honor the contract of calling open/update/close methods.
I have implemented an IntegrationFlow where I want to do to following tasks:
Poll for files from a directory
Transform the file content to a string
Send the string via WebFluxRequestExecutingMessageHandler to a REST-Endpoint and use an AdviceChain to handle success and error responses
Implementation
#Configuration
#Slf4j
public class JsonToRestIntegration {
#Autowired
private LoadBalancerExchangeFilterFunction lbFunction;
#Value("${json_folder}")
private String jsonPath;
#Value("${json_success_folder}")
private String jsonSuccessPath;
#Value("${json_error_folder}")
private String jsonErrorPath;
#Value("${rest-service-url}")
private String restServiceUrl;
#Bean
public DirectChannel httpResponseChannel() {
return new DirectChannel();
}
#Bean
public MessageChannel successChannel() {
return new DirectChannel();
}
#Bean
public MessageChannel failureChannel() {
return new DirectChannel();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedDelay(1000).get();
}
#Bean
public IntegrationFlow jsonFileToRestFlow() {
return IntegrationFlows
.from(fileReadingMessageSource(), e -> e.id("fileReadingEndpoint"))
.transform(org.springframework.integration.file.dsl.Files.toStringTransformer())
.enrichHeaders(s -> s.header("Content-Type", "application/json; charset=utf8"))
.handle(reactiveOutbound())
.log()
.channel(httpResponseChannel())
.get();
}
#Bean
public FileReadingMessageSource fileReadingMessageSource() {
FileReadingMessageSource source = new FileReadingMessageSource();
source.setDirectory(new File(jsonPath));
source.setFilter(new SimplePatternFileListFilter("*.json"));
source.setUseWatchService(true);
source.setWatchEvents(FileReadingMessageSource.WatchEventType.CREATE);
return source;
}
#Bean
public MessageHandler reactiveOutbound() {
WebClient webClient = WebClient.builder()
.baseUrl("http://jsonservice")
.filter(lbFunction)
.build();
WebFluxRequestExecutingMessageHandler handler = new WebFluxRequestExecutingMessageHandler(restServiceUrl, webClient);
handler.setHttpMethod(HttpMethod.POST);
handler.setCharset(StandardCharsets.UTF_8.displayName());
handler.setOutputChannel(httpResponseChannel());
handler.setExpectedResponseType(String.class);
handler.setAdviceChain(singletonList(expressionAdvice()));
return handler;
}
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setTrapException(true);
advice.setSuccessChannel(successChannel());
advice.setOnSuccessExpressionString("payload + ' war erfolgreich'");
advice.setFailureChannel(failureChannel());
advice.setOnFailureExpressionString("payload + ' war nicht erfolgreich'");
return advice;
}
#Bean
public IntegrationFlow loggingFlow() {
return IntegrationFlows.from(httpResponseChannel())
.handle(message -> {
String originalFileName = (String) message.getHeaders().get(FileHeaders.FILENAME);
log.info("some log");
})
.get();
}
#Bean
public IntegrationFlow successFlow() {
return IntegrationFlows.from(successChannel())
.handle(message -> {
MessageHeaders messageHeaders = ((AdviceMessage) message).getInputMessage().getHeaders();
File originalFile = (File) messageHeaders.get(ORIGINAL_FILE);
String originalFileName = (String) messageHeaders.get(FILENAME);
if (originalFile != null && originalFileName != null) {
File jsonSuccessFolder = new File(jsonSuccessPath);
File jsonSuccessFile = new File(jsonSuccessFolder, originalFileName);
try {
Files.move(originalFile.toPath(), jsonSuccessFile.toPath());
} catch (IOException e) {
log.error("some log", e);
}
}
})
.get();
}
#Bean
public IntegrationFlow failureFlow() {
return IntegrationFlows.from(failureChannel())
.handle(message -> {
Message<?> failedMessage = ((MessagingException) message.getPayload()).getFailedMessage();
if (failedMessage != null) {
File originalFile = (File) failedMessage.getHeaders().get(FileHeaders.ORIGINAL_FILE);
String originalFileName = (String) failedMessage.getHeaders().get(FileHeaders.FILENAME);
if (originalFile != null && originalFileName != null) {
File jsonErrorFolder = new File(tonisJsonErrorPath);
File jsonErrorFile = new File(jsonErrorFolder, originalFileName);
try {
Files.move(originalFile.toPath(), jsonErrorFile.toPath());
} catch (IOException e) {
log.error("some log", e);
}
}
}
})
.get();
}
}
So far it seems to work in production. In the test I want to do the following steps:
Copy JSON-Files to the input directory
Start the polling for the json files
Do assertions on the HTTP-Response from the WebFluxRequestExecutingMessageHandler which are routed through my advice chain
But I'm struggling in the test with the following tasks:
Mocking the WebFluxRequestExecutingMessageHandler with the MockIntegrationContext.substituteMessageHandlerFor()-method
Manually start the polling of the json files
Test
#RunWith(SpringRunner.class)
#SpringIntegrationTest()
#Import({JsonToRestIntegration.class})
#JsonTest
public class JsonToRestIntegrationTest {
#Autowired
public DirectChannel httpResponseChannel;
#Value("${json_folder}")
private String jsonPath;
#Value("${json_success_folder}")
private String jsonSuccessPath;
#Value("${json_error_folder}")
private String jsonErrorPath;
#Autowired
private MockIntegrationContext mockIntegrationContext;
#Autowired
private MessageHandler reactiveOutbound;
#Before
public void setUp() throws Exception {
Files.createDirectories(Paths.get(jsonPath));
Files.createDirectories(Paths.get(jsonSuccessPath));
Files.createDirectories(Paths.get(jsonErrorPath));
}
#After
public void tearDown() throws Exception {
FileUtils.deleteDirectory(new File(jsonPath));
FileUtils.deleteDirectory(new File(jsonSuccessPath));
FileUtils.deleteDirectory(new File(jsonErrorPath));
}
#Test
public void shouldSendJsonToRestEndpointAndReceiveOK() throws Exception {
File jsonFile = new ClassPathResource("/test.json").getFile();
Path targetFilePath = Paths.get(jsonPath + "/" + jsonFile.getName());
Files.copy(jsonFile.toPath(), targetFilePath);
httpResponseChannel.subscribe(httpResponseHandler());
this.mockIntegrationContext.substituteMessageHandlerFor("", reactiveOutbound);
}
private MessageHandler httpResponseHandler() {
return message -> Assert.assertThat(message.getPayload(), is(notNullValue()));
}
#Configuration
#Import({JsonToRestIntegration.class})
public static class JsonToRestIntegrationTest {
#Autowired
public MessageChannel httpResponseChannel;
#Bean
public MessageHandler reactiveOutbound() {
ArgumentCaptor<Message<?>> messageArgumentCaptor = ArgumentCaptor.forClass(Message.class);
MockMessageHandler mockMessageHandler = mockMessageHandler(messageArgumentCaptor).handleNextAndReply(m -> m);
mockMessageHandler.setOutputChannel(httpResponseChannel);
return mockMessageHandler;
}
}
}
Updated Working Example with mocked WebFluX web client:
Implementation
public class JsonToRestIntegration {
private final LoadBalancerExchangeFilterFunction lbFunction;
private final BatchConfigurationProperties batchConfigurationProperties;
#Bean
public DirectChannel httpResponseChannel() {
return new DirectChannel();
}
#Bean
public DirectChannel errorChannel() {
return new DirectChannel();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedDelay(100, TimeUnit.MILLISECONDS).get();
}
#Bean
public IntegrationFlow jsonFileToRestFlow() {
return IntegrationFlows
.from(fileReadingMessageSource(), e -> e.id("fileReadingEndpoint"))
.transform(org.springframework.integration.file.dsl.Files.toStringTransformer("UTF-8"))
.enrichHeaders(s -> s.header("Content-Type", "application/json; charset=utf8"))
.handle(reactiveOutbound())
.channel(httpResponseChannel())
.get();
}
#Bean
public FileReadingMessageSource fileReadingMessageSource() {
FileReadingMessageSource source = new FileReadingMessageSource();
source.setDirectory(new File(batchConfigurationProperties.getJsonImportFolder()));
source.setFilter(new SimplePatternFileListFilter("*.json"));
source.setUseWatchService(true);
source.setWatchEvents(FileReadingMessageSource.WatchEventType.CREATE);
return source;
}
#Bean
public WebFluxRequestExecutingMessageHandler reactiveOutbound() {
WebClient webClient = WebClient.builder()
.baseUrl("http://service")
.filter(lbFunction)
.build();
WebFluxRequestExecutingMessageHandler handler = new WebFluxRequestExecutingMessageHandler(batchConfigurationProperties.getServiceUrl(), webClient);
handler.setHttpMethod(HttpMethod.POST);
handler.setCharset(StandardCharsets.UTF_8.displayName());
handler.setOutputChannel(httpResponseChannel());
handler.setExpectedResponseType(String.class);
handler.setAdviceChain(singletonList(expressionAdvice()));
return handler;
}
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setTrapException(true);
advice.setFailureChannel(errorChannel());
return advice;
}
#Bean
public IntegrationFlow responseFlow() {
return IntegrationFlows.from(httpResponseChannel())
.handle(message -> {
MessageHeaders messageHeaders = message.getHeaders();
File originalFile = (File) messageHeaders.get(ORIGINAL_FILE);
String originalFileName = (String) messageHeaders.get(FILENAME);
if (originalFile != null && originalFileName != null) {
File jsonSuccessFolder = new File(batchConfigurationProperties.getJsonSuccessFolder());
File jsonSuccessFile = new File(jsonSuccessFolder, originalFileName);
try {
Files.move(originalFile.toPath(), jsonSuccessFile.toPath());
} catch (IOException e) {
log.error("Could not move file", e);
}
}
})
.get();
}
#Bean
public IntegrationFlow failureFlow() {
return IntegrationFlows.from(errorChannel())
.handle(message -> {
Message<?> failedMessage = ((MessagingException) message.getPayload()).getFailedMessage();
if (failedMessage != null) {
File originalFile = (File) failedMessage.getHeaders().get(ORIGINAL_FILE);
String originalFileName = (String) failedMessage.getHeaders().get(FILENAME);
if (originalFile != null && originalFileName != null) {
File jsonErrorFolder = new File(batchConfigurationProperties.getJsonErrorFolder());
File jsonErrorFile = new File(jsonErrorFolder, originalFileName);
try {
Files.move(originalFile.toPath(), jsonErrorFile.toPath());
} catch (IOException e) {
log.error("Could not move file", originalFileName, e);
}
}
}
})
.get();
}
}
Test
#RunWith(SpringRunner.class)
#SpringIntegrationTest(noAutoStartup = "fileReadingEndpoint")
#Import({JsonToRestIntegration.class, BatchConfigurationProperties.class})
#JsonTest
#DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD)
public class JsonToRestIntegrationIT {
private static final FilenameFilter JSON_FILENAME_FILTER = (dir, name) -> name.endsWith(".json");
#Autowired
private BatchConfigurationProperties batchConfigurationProperties;
#Autowired
private ObjectMapper om;
#Autowired
private MessageHandler reactiveOutbound;
#Autowired
private DirectChannel httpResponseChannel;
#Autowired
private DirectChannel errorChannel;
#Autowired
private FileReadingMessageSource fileReadingMessageSource;
#Autowired
private SourcePollingChannelAdapter fileReadingEndpoint;
#MockBean
private LoadBalancerExchangeFilterFunction lbFunction;
private String jsonImportPath;
private String jsonSuccessPath;
private String jsonErrorPath;
#Before
public void setUp() throws Exception {
jsonImportPath = batchConfigurationProperties.getJsonImportFolder();
jsonSuccessPath = batchConfigurationProperties.getJsonSuccessFolder();
jsonErrorPath = batchConfigurationProperties.getJsonErrorFolder();
Files.createDirectories(Paths.get(jsonImportPath));
Files.createDirectories(Paths.get(jsonSuccessPath));
Files.createDirectories(Paths.get(jsonErrorPath));
}
#After
public void tearDown() throws Exception {
FileUtils.deleteDirectory(new File(jsonImportPath));
FileUtils.deleteDirectory(new File(jsonSuccessPath));
FileUtils.deleteDirectory(new File(jsonErrorPath));
}
#Test
public void shouldMoveJsonFileToSuccessFolderWhenHttpResponseIsOk() throws Exception {
final CountDownLatch latch = new CountDownLatch(1);
httpResponseChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
latch.countDown();
super.postSend(message, channel, sent);
}
});
errorChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
fail();
}
});
ClientHttpConnector httpConnector = new HttpHandlerConnector((request, response) -> {
response.setStatusCode(HttpStatus.OK);
response.getHeaders().setContentType(MediaType.APPLICATION_JSON_UTF8);
DataBufferFactory bufferFactory = response.bufferFactory();
String valueAsString = null;
try {
valueAsString = om.writeValueAsString(new ResponseDto("1"));
} catch (JsonProcessingException e) {
fail();
}
return response.writeWith(Mono.just(bufferFactory.wrap(valueAsString.getBytes())))
.then(Mono.defer(response::setComplete));
});
WebClient webClient = WebClient.builder()
.clientConnector(httpConnector)
.build();
new DirectFieldAccessor(this.reactiveOutbound)
.setPropertyValue("webClient", webClient);
File jsonFile = new ClassPathResource("/test.json").getFile();
Path targetFilePath = Paths.get(jsonImportPath + "/" + jsonFile.getName());
Files.copy(jsonFile.toPath(), targetFilePath);
fileReadingEndpoint.start();
assertThat(latch.await(12, TimeUnit.SECONDS), is(true));
File[] jsonImportFolder = new File(jsonImportPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonImportFolder, is(notNullValue()));
assertThat(filesInJsonImportFolder.length, is(0));
File[] filesInJsonSuccessFolder = new File(jsonSuccessPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonSuccessFolder, is(notNullValue()));
assertThat(filesInJsonSuccessFolder.length, is(1));
File[] filesInJsonErrorFolder = new File(jsonErrorPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonErrorFolder, is(notNullValue()));
assertThat(filesInJsonErrorFolder.length, is(0));
}
#Test
public void shouldMoveJsonFileToErrorFolderWhenHttpResponseIsNotOk() throws Exception {
final CountDownLatch latch = new CountDownLatch(1);
errorChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
latch.countDown();
super.postSend(message, channel, sent);
}
});
httpResponseChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
fail();
}
});
ClientHttpConnector httpConnector = new HttpHandlerConnector((request, response) -> {
response.setStatusCode(HttpStatus.BAD_REQUEST);
response.getHeaders().setContentType(MediaType.APPLICATION_JSON_UTF8);
DataBufferFactory bufferFactory = response.bufferFactory();
return response.writeWith(Mono.just(bufferFactory.wrap("SOME BAD REQUEST".getBytes())))
.then(Mono.defer(response::setComplete));
});
WebClient webClient = WebClient.builder()
.clientConnector(httpConnector)
.build();
new DirectFieldAccessor(this.reactiveOutbound)
.setPropertyValue("webClient", webClient);
File jsonFile = new ClassPathResource("/error.json").getFile();
Path targetFilePath = Paths.get(jsonImportPath + "/" + jsonFile.getName());
Files.copy(jsonFile.toPath(), targetFilePath);
fileReadingEndpoint.start();
assertThat(latch.await(11, TimeUnit.SECONDS), is(true));
File[] filesInJsonImportFolder = new File(jsonImportPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonImportFolder, is(notNullValue()));
assertThat(filesInJsonImportFolder.length, is(0));
File[] filesInJsonSuccessFolder = new File(jsonSuccessPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonSuccessFolder, is(notNullValue()));
assertThat(filesInJsonSuccessFolder.length, is(0));
File[] filesInJsonErrorFolder = new File(jsonErrorPath).listFiles(JSON_FILENAME_FILTER);
assertThat(filesInJsonErrorFolder, is(notNullValue()));
assertThat(filesInJsonErrorFolder.length, is(1));
}
}
this.mockIntegrationContext.substituteMessageHandlerFor("", reactiveOutbound);
The first parameter of this method is an endpoint id. (I guess we are just missing Javadocs there on those methods...).
So, what you need is something like this:
.handle(reactiveOutbound(), e -> e.id("webFluxEndpoint"))
And then in that test-case you do:
this.mockIntegrationContext.substituteMessageHandlerFor("webFluxEndpoint", reactiveOutbound);
You don't need to override bean in the test class config. The MockMessageHandler can be just used in the test method body.
You poll files via .from(fileReadingMessageSource()). To do a manual control you need to have it stopped in the beginning. For this purpose you add an endpoint id again:
.from(fileReadingMessageSource(), e -> e.id("fileReadingEndpoint"))
And then in the test configuration you do this:
#SpringIntegrationTest(noAutoStartup = "fileReadingEndpoint")
Another approach for the WebFlux would be via customized WebClient to mock request to the server. For example:
ClientHttpConnector httpConnector = new HttpHandlerConnector((request, response) -> {
response.setStatusCode(HttpStatus.OK);
response.getHeaders().setContentType(MediaType.TEXT_PLAIN);
DataBufferFactory bufferFactory = response.bufferFactory();
return response.writeWith(Mono.just(bufferFactory.wrap("FOO\nBAR\n".getBytes())))
.then(Mono.defer(response::setComplete));
});
WebClient webClient = WebClient.builder()
.clientConnector(httpConnector)
.build();
new DirectFieldAccessor(this.reactiveOutbound)
.setPropertyValue("webClient", webClient);
I am using an api to call the springbatch job. when i call for the first time it works fine. second time gives the below error.
Caused by: java.lang.IllegalArgumentException: Sheet index (3) is out of range (0..2).
Below is my code.-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Config:
#Configuration
#EnableBatchProcessing
#PropertySource("classpath:batch.properties")
public class ExcelFileToDatabaseJobConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
ItemReader<StudentDTO> excelStudentReader() throws Exception {
System.out.println("inside excelStudentReader");
PoiItemReader<StudentDTO> reader = new PoiItemReader<>();
reader.setLinesToSkip(1);
//reader.setResource(new FileSystemResource("C:/Users/lenovo/IdeaProjects/test/out/production/classes/Price Change.xlsx"));
reader.setResource(new UrlResource("file:///C:/Users/lenovo/IdeaProjects/test/out/production/classes/inserttest.xlsx"));
reader.setRowMapper(excelRowMapper());
return reader;
}
private RowMapper<StudentDTO> excelRowMapper() {
System.out.println("inside excelRowMapper");
BeanWrapperRowMapper<StudentDTO> rowMapper = new BeanWrapperRowMapper<>();
rowMapper.setTargetType(StudentDTO.class);
return rowMapper;
}
#Bean
public JdbcBatchItemWriter<StudentDTO> writer(DataSource dataSource) {
System.out.println("inside writer");
return new JdbcBatchItemWriterBuilder<StudentDTO>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO tbl_item_master (mrp_rate,price,item_code) VALUES (:mrp_rate,:rate,:u_item_code)")
//.sql("update ignore tbl_item_master set mrp_rate=:mrp_rate,price=:rate where item_code=:u_item_code")
.dataSource(dataSource)
.build();
}
#Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
System.out.println("inside importUserJob");
return jobBuilderFactory.get("importUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
}
#Bean
public Step step1(JdbcBatchItemWriter<StudentDTO> writer,CustomChunkListener customChunkListener,
CustomItemReaderListener customItemReaderListener,CustomItemWriterListener customItemWriterListener,
CustomStepListener customStepListener,CustomSkipListener customSkipListener) throws Exception {
System.out.println("inside step1");
return stepBuilderFactory.get("step1")
.<StudentDTO, StudentDTO>chunk(3000)
.reader(excelStudentReader())
//.processor(processor())
.writer(writer)
.faultTolerant()
.skip(Exception.class)
.noRetry(Exception.class)
.noRollback(Exception.class)
.skipLimit(100)
.listener(customItemWriterListener)
.listener(customSkipListener)
.build();
}
#Bean
public CustomStepListener customStepListener() {
return new CustomStepListener();
}
#Bean
public CustomItemWriterListener itemWriterListener() {
return new CustomItemWriterListener();
}
#Bean
public CustomItemReaderListener itemReaderListener() {
return new CustomItemReaderListener();
}
#Bean
public CustomSkipListener customSkipListener() {
return new CustomSkipListener();
}
}
Controller:
#Controller
public class FileController1 {
#Autowired
private JobLauncher jobLauncher;
#Autowired
private Job importUserJob;
#RequestMapping(value = "/echofile", method = RequestMethod.POST, produces = {"application/json"})
public #ResponseBody String echoFile(MultipartHttpServletRequest request,
HttpServletResponse response) throws Exception {
MultipartFile multipartFile = request.getFile("file");
String path = new ClassPathResource("/").getURL().getPath();
File fileToImport = new File(path + multipartFile.getOriginalFilename());
//filePath = fileToImport.getAbsolutePath();
OutputStream outputStream = new FileOutputStream(fileToImport);
IOUtils.copy(multipartFile.getInputStream(), outputStream);
outputStream.flush();
outputStream.close();
//Launch the Batch Job
JobExecution jobExecution = jobLauncher.run(importUserJob,new JobParametersBuilder()
.addString("fullPathFileName", fileToImport.getAbsolutePath()).addLong("time",System.currentTimeMillis()).toJobParameters());
return "something";
}
}
batch.properties:
spring.batch.job.enabled=false
I'm new to the Spring Batch. I have written code for a spring batch job and it is working also. But it is not exiting after finishing the job.Even during debugging I cannot stop it by keyboard shortcut to exit.
If there is no file , it exits. But when the file is available it is ending after execution. I don't prefer to use System.exit.
Any help on this will be great and thanks in advance.
public class OfferRedemptionJobConfig {
#Bean
public Job RedemptionJob(){
Flow flow = new FlowBuilder<SimpleFlow>("RedemptionJobFlow")
.start(jobValidatorStep())
.next(fileDecisionDecider)
.on("COMPLETED")
.end()
.from(fileDecisionDecider)
.on("CONTINUE")
.to(RedemptionFileTaskletStep())
.next(redemptionFileReprocessStep())
.next(fileDecisionDecider)
.on("COMPLETED")
.end().build();
return jobBuilderFactory.get("RedemptionJob")
.incrementer(new RunIdIncrementer())
.listener(new DefaultJobListener())
.start(flow).end().build();
}
#Bean
public Step redemptionFileReprocessStep() {
return stepBuilderFactory.get("redemptionFileReprocessStep")
.<RedemptionVO, RedemptionVO>chunk(100)
.reader(RedemptionFileReader(WILL_BE_INJECTED))
.processor(FileProcessor(WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED))
.listener(RedemptionFileListenerStep(WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED,WILL_BE_INJECTED))
.build();
}
#Bean
#StepScope
public FlatFileItemReader<RedemptionVO> RedemptionFileReader(#Value("#{jobExecutionContext['localDownloadedPath']}") String filepath) {
FlatFileItemReader<RedemptionVO> reader = new FlatFileItemReader<>();
logger.info("FILE READER FILE PATH {}",filepath);
reader.setResource(new FileSystemResource(new File(filepath)));
reader.setLineMapper(RedemptionFileLineMapper());
reader.setLinesToSkip(1);
return reader;
}
}
}
#Component
public class OfferRedemptionJob {
private static final Logger logger = LoggerFactory.getLogger(YTBSOfferRedemptionJob.class);
#Autowired
JobRequestLauncher jobRequestLauncher;
#Autowired
BatchJobConfigurationRepo batchJobConfigurationRepo;
#Autowired
CommonJobsHelper commonJobsHelper;
public void run() {
logger.info("Job is runnning");
List<BatchJobConfigurationEntity> batchJobConfigurationRepoList = batchJobConfigurationRepo.getAllActiveJobConfigsByJobName("ytbsRedemptionJob");
Map<String, String> jobConfigs = commonJobsHelper.extractConfigs(batchJobConfigurationRepoList);
if ((!jobConfigs.containsKey("enabled")) || jobConfigs.get("enabled").equalsIgnoreCase("true")) {
JobLaunchRequest jobLaunchRequest = new JobLaunchRequest("RedemptionJob", jobConfigs);
JobExecution jobExecution = jobRequestLauncher.launch(jobLaunchRequest);
logger.info("jobExecution ExitStatus {}", jobExecution.getExitStatus());
}
}
}
public class JobRequestLauncher {
private static final Logger logger = LoggerFactory.getLogger(JobRequestLauncher.class);
#Autowired
JobRegistry jobRegistry;
#Autowired
SimpleLauncher simpleLauncher;
public JobExecution launch(JobLaunchRequest request){
JobExecution jobExecution = null;
try {
Job job = jobRegistry.getJob(request.getJobName());
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
for (Map.Entry<String, String> entry : request.getJobParameters().entrySet()) {
jobParametersBuilder.addString(entry.getKey(),entry.getValue());
}
logger.info(jobParametersBuilder.toString());
jobParametersBuilder.addLong("jobStartTime",new Date().getTime());
jobExecution = simpleLauncher.launchJob(job,jobParametersBuilder.toJobParameters());
} catch (NoSuchJobException | JobInstanceAlreadyCompleteException | JobExecutionAlreadyRunningException | JobRestartException | JobParametersInvalidException e) {
logger.error(e.getClass().getSimpleName() + " occured while launching job: {} with params: {}. {}",request.getJobName(),request.getJobParameters(),e.getMessage(),e);
} catch(Exception e){
logger.error(e.getClass().getSimpleName() + " occured while launching job: {} with params: {}. {}",request.getJobName(),request.getJobParameters(),e.getMessage(),e);
}
return jobExecution;
}
}
#Component
public class SimpleLauncher {
#Autowired
BatchRepoConfig batchRepoConfig;
#Retryable()
public JobExecution launchJob(Job job, JobParameters jobParameters) throws Exception {
System.out.println(job.getName());
return batchRepoConfig.getJobLauncher().run(job,jobParameters);
}
}
Hello I have the follow Tasklet in my spring batch application:
public class FileMovingTasklet implements Tasklet, InitializingBean {
#Value("${positionFile.source-path}")
private String sourcePath;
#Value("${positionFile.local-path}")
private String localDirectory;
#Value("${positionFile.patternName}")
private String fileNamePattern;
#Value("${positionFile.suffix}")
private String suffix;
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
//***code***
List<PathResource> resources = FileManager.getInputFileList(sourcePath, fileNamePattern, suffix, fileDate);
if (!CollectionUtils.isEmpty(resources)) {
copyFiles(resources, localDirectory);
log.info("Copied files to local directory...");
}
} catch (IOException e) {
log.error("Position File not found with sourcePatch={}, fileName={}, suffix={}, filedate={}", sourcePath, fileNamePattern, suffix, fileDate);
throw new TaskletException("Could not move file from source to local " + e);
}
return RepeatStatus.FINISHED;
}
private void copyFiles(List<PathResource> resources, String localDirectory) {
for (Resource resource : resources) {
File source;
File target;
try {
source = resource.getFile();
target = new File(localDirectory + File.separator + source.getName());
try {
FileCopyUtils.copy(source, target);
} catch (IOException e) {
}
} catch (IOException e) {
}
}
}
}
I am moving some files from source to my local destination. The local destination is where I want to to read in files and process and write in my subsequent steps.
I have configured my Job as follows:
#Configuration
public class BatchConfiguration {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private FileMovingTasklet fileMovingTasklet;
#Value("${positionFile.local-path}")
private Resource[] resources;
#Autowired
private PriceDao scpDao;
#Autowired
public PositionRowReader positionRowReader;
#Bean
public MultiResourceItemReader<Pos> multiResourceItemReader() {
MultiResourceItemReader<Pos> resourceItemReader = new MultiResourceItemReader<>();
resourceItemReader.setResources(resources);
resourceItemReader.setDelegate(posRowReader());
return resourceItemReader;
}
#Bean
public FlatFileItemReader<Pos> posRowReader() {
return positionRowReader.getReader();
}
#Bean
public ItemProcessor<Pos, Price> posRowProcessor() {
return new PosRowItemProcessor();
}
#Bean
public JobExecutionListener listener() {
return new JobCompletionNotificationListener(scpDao);
}
#Bean
public Job import() {
return jobBuilderFactory.get("import")
.incrementer(new RunIdIncrementer())
.listener(listener())
.start(getPositionFileStep())
.next(step1())
.build();
}
#Bean
public Step getPositionFileStep() {
return stepBuilderFactory.get("getPositionFileStep")
.tasklet(fileMovingTasklet)
.build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<Pos, Price>chunk(50)
.reader(multiResourceItemReader())
.processor(posRowProcessor())
.writer(new PriceWriter(scpDao))
.build();
}
}
I get failed to initialize the reader when executing step1:
org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
my local-path property is as follows:
positionFile.local-path=C:\\Dev\\workspace\\batch\\src\\main\\resources\\localPath
positionFile.patterName=PositionFile*
my question is how can I access the resources (files) which can be multiple files in step1 after files have been copied over from source folder.
My resources size is 0 even though files are there:
#Value("${positionFile.local-path}")
private String filePath;
#Value("${positionFile.patternName}")
private String filePattern;
#Bean
#StepScope
public MultiResourceItemReader<PosRow> multiResourceItemReader() {
MultiResourceItemReader<PosRow> resourceItemReader = new MultiResourceItemReader<>();
Resource[] resources = new Resource[0];
try {
ClassLoader cl = this.getClass().getClassLoader();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);
resources = resolver.getResources("file:" + filePath + File.separator + filePattern + "*");
} catch (IOException e) {
log.error("Problem with getting resource files");
}
resourceItemReader.setResources(resources);
resourceItemReader.setDelegate(posRowReader());
return resourceItemReader;
}
You need to make your MultiResourceItemReader and FlatFileItemReader lazy by using #StepScope
#Bean
#StepScope
public MultiResourceItemReader<Pos> multiResourceItemReader() {
MultiResourceItemReader<Pos> resourceItemReader = new MultiResourceItemReader<>();
resourceItemReader.setResources(resources);
resourceItemReader.setDelegate(posRowReader());
return resourceItemReader;
}
#Bean
#StepScope
public FlatFileItemReader<Pos> posRowReader() {
return positionRowReader.getReader();
}