I am trying to send below Jsonobject as request parameter to rest API.
{ "date": "2022-01-01", "value": [ "TST/USED" ] }
Value field contains the list of values, but when I add the value in this format as part of request it replaces string / to \/ due to which the request is not processing and it throws 415 : [no body] exception.
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.add(AUTHORIZATION, "Bearer " + token);
headers.setAccept(Arrays.asList(MediaType.APPLICATION_JSON));
JSONObject req = new JSONObject();
req.put("date", "2022-01-01");
req.put("value", "TST/USED");
HttpEntity<Object> object = new HttpEntity<>(req.toString(), headers);
Object response = restTemplate.exchange(apiUrl, HttpMethod.POST, object, Object.class)
.getBody();
I don't see the issue you mention, here is what I am running:
Main application with Spring boot and a RestTemplate bean.
package com.so.so72424428;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;
#SpringBootApplication
public class So72424428Application {
public static void main(String[] args) {
SpringApplication.run(So72424428Application.class, args);
}
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder.build();
}
}
An api to test:
package com.so.so72424428;
import org.springframework.util.MimeTypeUtils;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
#RestController
public class ApiEndPoint {
#PostMapping(path = "/test-api", consumes = MimeTypeUtils.APPLICATION_JSON_VALUE, produces = MimeTypeUtils.APPLICATION_JSON_VALUE)
public String echo (#RequestBody String jsonMessage) {
return jsonMessage;
}
}
A class to run your code:
package com.so.so72424428;
import java.util.Arrays;
import org.json.JSONObject;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestTemplate;
import lombok.extern.slf4j.Slf4j;
#Component
#Slf4j
public class So72424428 implements CommandLineRunner {
#Autowired
private RestTemplate restTemplate;
#Override
public void run(String... args) throws Exception {
HttpHeaders headers = new HttpHeaders();
//headers.add(AUTHORIZATION, "Bearer " + token);
headers.setAccept(Arrays.asList(MediaType.APPLICATION_JSON));
headers.setContentType(MediaType.APPLICATION_JSON);
JSONObject req = new JSONObject();
req.put("date", "2022-01-01");
req.put("value", "TST/USED");
log.info(req.toString());
HttpEntity<Object> object = new HttpEntity<>(req.toString(), headers);
String apiUrl = "http://localhost:8080/test-api";
Object response = restTemplate.exchange(apiUrl, HttpMethod.POST, object, Object.class)
.getBody();
log.info(response.toString());
}
}
When I run the code I print out the content of the req variable:
{"date":"2022-01-01","value":"TST/USED"}
Also after round trip of the request I print out the response:
{date=2022-01-01, value=TST/USED}
This is the log:
2022-05-29 13:39:24.416 INFO 32332 --- [ restartedMain] com.so.so72424428.So72424428Application : Starting So72424428Application using Java 15.0.2 on ...)
2022-05-29 13:39:24.418 INFO 32332 --- [ restartedMain] com.so.so72424428.So72424428Application : No active profile set, falling back to 1 default profile: "default"
2022-05-29 13:39:24.462 INFO 32332 --- [ restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable
2022-05-29 13:39:24.462 INFO 32332 --- [ restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : For additional web related logging consider setting the 'logging.level.web' property to 'DEBUG'
2022-05-29 13:39:25.310 INFO 32332 --- [ restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2022-05-29 13:39:25.320 INFO 32332 --- [ restartedMain] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2022-05-29 13:39:25.320 INFO 32332 --- [ restartedMain] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.63]
2022-05-29 13:39:25.387 INFO 32332 --- [ restartedMain] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2022-05-29 13:39:25.388 INFO 32332 --- [ restartedMain] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 925 ms
2022-05-29 13:39:25.716 INFO 32332 --- [ restartedMain] o.s.b.d.a.OptionalLiveReloadServer : LiveReload server is running on port 35729
2022-05-29 13:39:25.760 INFO 32332 --- [ restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2022-05-29 13:39:25.768 INFO 32332 --- [ restartedMain] com.so.so72424428.So72424428Application : Started So72424428Application in 1.678 seconds (JVM running for 2.616)
2022-05-29 13:39:25.778 INFO 32332 --- [ restartedMain] com.so.so72424428.So72424428 : {"date":"2022-01-01","value":"TST/USED"}
2022-05-29 13:39:25.869 INFO 32332 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2022-05-29 13:39:25.869 INFO 32332 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2022-05-29 13:39:25.870 INFO 32332 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 1 ms
2022-05-29 13:39:25.921 INFO 32332 --- [ restartedMain] com.so.so72424428.So72424428 : {date=2022-01-01, value=TST/USED}
2022-05-29 13:39:33.346 INFO 32332 --- [on(2)-127.0.0.1] inMXBeanRegistrar$SpringApplicationAdmin : Application shutdown requested.
As you can see there is no backslash, nor no issue for completing the request.
Related
My multi-tenant application needs to periodically check a set of SFTP folders for an ingestion file. The SFTP folders and parameters are defined in my application.properties file, and the number is dynamic.
ingestion.carriers=carrier1,carrier2
ingestion.carrier1=sftp
ingestion.carrier1.sftp.host=localhost
ingestion.carrier1.sftp.username=sftp
ingestion.carrier1.sftp.password=sftp
ingestion.carrier1.sftp.remotedir=carrier1
ingestion.carrier1.sftp.localdir=sftp/carrier1
ingestion.carrier1.sftp.archivedir=sftp/carrier1/archive
ingestion.carrier1.sftp.errordir=sftp/carrier1/error
ingestion.carrier1.ping=7000
ingestion.carrier2=sftp
ingestion.carrier2.sftp.host=localhost
ingestion.carrier2.sftp.username=sftp
ingestion.carrier2.sftp.password=sftp
ingestion.carrier2.sftp.remotedir=carrier2
ingestion.carrier2.sftp.localdir=sftp/carrier2
ingestion.carrier2.sftp.archivedir=sftp/carrier2/archive
ingestion.carrier2.sftp.errordir=sftp/carrier2/error
ingestion.carrier2.pingFrequency=13000
I need to dinamically create all the necessary beans to enable spring integration flow. To do so, I've tried to set up a BeanFactoryPostProcessor, as I cannot declare the beans in a "static" way. This processor is thought to be configured, in future, with different methods to retrieve the file: because of this, the actual creation of beans is delegated to another class.
This is the post processor...
package mypkg.batch.config.integration;
import mypkg.batch.config.integration.factory.SFTPBeansFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.config.BeanFactoryPostProcessor;
import org.springframework.beans.factory.config.ConfigurableListableBeanFactory;
import org.springframework.core.env.Environment;
import org.springframework.stereotype.Component;
import java.util.List;
#Component
public class IngestionBeanConfigurator implements BeanFactoryPostProcessor {
public static final Logger LOG = LoggerFactory.getLogger(IngestionBeanConfigurator.class);
#Override
public void postProcessBeanFactory(ConfigurableListableBeanFactory fact) throws BeansException {
Environment env = fact.getBean(Environment.class);
String carriersList = env.getProperty("ingestion.carriers");
if (carriersList == null) {
LOG.info("No ingestion has been defined");
return;
}
List<String> carriers = List.of(carriersList.split(","));
for (String carrier : carriers) {
String carrierMethod = env.getProperty("ingestion.%s".formatted(carrier));
if (carrierMethod != null) {
if ("sftp".equals(carrierMethod)) {
new SFTPBeansFactory(carrier, env).loadBeans(fact);
} else {
LOG.warn("Invalid carrier method {} for carrier {}", carrierMethod, carrier);
}
}
}
}
}
... and this is the class creating SFTP beans
package com.eyemed.foodogs.batch.config.integration.factory;
import com.eyemed.foodogs.model.exception.MembersMessageHandler;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.beans.factory.config.ConfigurableBeanFactory;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.GenericApplicationContext;
import org.springframework.core.env.Environment;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.integration.dsl.MessageChannelSpec;
import org.springframework.integration.dsl.Pollers;
import org.springframework.integration.dsl.StandardIntegrationFlow;
import org.springframework.integration.dsl.context.IntegrationFlowContext;
import org.springframework.integration.file.filters.AcceptOnceFileListFilter;
import org.springframework.integration.sftp.filters.SftpSimplePatternFileListFilter;
import org.springframework.integration.sftp.inbound.SftpInboundFileSynchronizer;
import org.springframework.integration.sftp.inbound.SftpInboundFileSynchronizingMessageSource;
import org.springframework.integration.sftp.session.DefaultSftpSessionFactory;
import java.io.File;
import java.math.BigInteger;
import java.util.concurrent.TimeUnit;
public class SFTPBeansFactory implements BeanFactory {
public static final Logger LOG = LoggerFactory.getLogger(SFTPBeansFactory.class);
private final String carrierId;
private final String sftpHost;
private final String sftpUsername;
private final String sftpPassword;
private final String sftpRemoteDir;
private final String sftpLocalDir;
private final String sftpArchiveDir;
private final String sftpErrorDir;
private final BigInteger pingFrequency;
public SFTPBeansFactory(final String carrierId, final Environment props) {
String prefix = "ingestion.%s".formatted(carrierId);
this.carrierId = carrierId;
this.sftpHost = props.getProperty("%s.sftp.host".formatted(prefix));
this.sftpUsername = props.getProperty("%s.sftp.username".formatted(prefix));
this.sftpPassword = props.getProperty("%s.sftp.password".formatted(prefix));
this.sftpRemoteDir = props.getProperty("%s.sftp.remotedir".formatted(prefix));
this.sftpLocalDir = props.getProperty("%s.sftp.localdir".formatted(prefix));
this.sftpArchiveDir = props.getProperty("%s.sftp.archivedir".formatted(prefix));
this.sftpErrorDir = props.getProperty("%s.sftp.errordir".formatted(prefix));
String pingFrequencyString = props.getProperty("%s.ping".formatted(prefix));
if (pingFrequencyString != null) {
this.pingFrequency = new BigInteger(pingFrequencyString);
} else {
this.pingFrequency = BigInteger.valueOf(3600000);
}
}
public void loadBeans(ConfigurableBeanFactory fact) {
DefaultSftpSessionFactory sf = _buildSessionFactory();
SftpInboundFileSynchronizer sync = _buildInboundFileSynchronizer(sf);
fact.registerSingleton("sftp-sync-%s".formatted(carrierId), sync);
SftpInboundFileSynchronizingMessageSource src = _buildMessageSource(sync);
MembersMessageHandler handler = new MembersMessageHandler(carrierId, fact.getBean(JobLauncher.class), fact.getBean("readMembersJob", Job.class));
String beanName = "sftp-flow-%s".formatted(carrierId);
String channelName = "sftp-ingestion-channel-%s".formatted(carrierId);
LOG.info("Creating bean %s based on channel %s".formatted(beanName, channelName));
StandardIntegrationFlow flow = IntegrationFlows
.from(src, c -> c.poller(Pollers.fixedRate(pingFrequency.longValue(), TimeUnit.MILLISECONDS, 0)))
.channel(channelName)
.handle(handler)
.get();
IntegrationFlowContext ctx = fact.getBean(IntegrationFlowContext.class);
ctx.registration(flow).id(beanName).autoStartup(true).register();
flow.start();
}
private SftpInboundFileSynchronizingMessageSource _buildMessageSource(SftpInboundFileSynchronizer sync) {
var src = new SftpInboundFileSynchronizingMessageSource(sync);
src.setLocalDirectory(new File(sftpLocalDir));
src.setAutoCreateLocalDirectory(true);
src.setLocalFilter(new AcceptOnceFileListFilter<>());
return src;
}
private SftpInboundFileSynchronizer _buildInboundFileSynchronizer(DefaultSftpSessionFactory sf) {
var sync = new SftpInboundFileSynchronizer(sf);
sync.setDeleteRemoteFiles(true);
sync.setRemoteDirectory(sftpRemoteDir);
sync.setFilter(new SftpSimplePatternFileListFilter("*.csv"));
sync.setLocalFilenameGeneratorExpressionString(
"#this.substring(0, #this.length - 4) + '_%s_' + new com.eyemed.foodogs.application.util.TimestampProvider().currentTimestamp() + '.txt'".formatted(carrierId));
return sync;
}
private DefaultSftpSessionFactory _buildSessionFactory() {
var sf = new DefaultSftpSessionFactory();
sf.setHost(sftpHost);
sf.setUser(sftpUsername);
sf.setPassword(sftpPassword);
sf.setPort(22);
sf.setAllowUnknownKeys(true);
return sf;
}
}
Unfortunately, this doesn't seem to work: SFTP files are not read and remain sadly in the source folder. The local SFTP works, as the previous version with the "static" beans used to work correctly.
Also, I do not see errors in the log
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.5.5)
2022-10-18 11:42:15.396 INFO 8226 --- [ main] com.eyemed.foodogs.application.App : Starting App using Java 17 on A-M-L-MUTI.local with PID 8226 (/Users/lorenzomuti/Repositories/FooDogs/backend/foodogsbootapplication/target/classes started by lorenzomuti in /Users/lorenzomuti/Repositories/FooDogs/backend/foodogsbootapplication)
2022-10-18 11:42:15.399 INFO 8226 --- [ main] com.eyemed.foodogs.application.App : The following profiles are active: dev
2022-10-18 11:42:17.988 INFO 8226 --- [ main] c.e.f.b.c.i.factory.SFTPBeansFactory : Creating bean sftp-flow-carrier1 based on channel sftp-ingestion-channel-carrier1
2022-10-18 11:42:18.028 INFO 8226 --- [ main] c.e.f.b.c.i.factory.SFTPBeansFactory : Creating bean sftp-flow-carrier2 based on channel sftp-ingestion-channel-carrier2
2022-10-18 11:42:18.038 INFO 8226 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2022-10-18 11:42:18.049 INFO 8226 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2022-10-18 11:42:18.311 INFO 8226 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-10-18 11:42:18.322 INFO 8226 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationChannelResolver' of type [org.springframework.integration.support.channel.BeanFactoryChannelResolver] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-10-18 11:42:18.324 INFO 8226 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-10-18 11:42:18.659 INFO 8226 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2022-10-18 11:42:18.675 INFO 8226 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2022-10-18 11:42:18.676 INFO 8226 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.53]
2022-10-18 11:42:18.816 INFO 8226 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2022-10-18 11:42:18.816 INFO 8226 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 3337 ms
2022-10-18 11:42:18.892 DEBUG 8226 --- [ main] o.s.w.f.CommonsRequestLoggingFilter : Filter 'logFilter' configured for use
2022-10-18 11:42:19.474 INFO 8226 --- [ main] c.e.f.application.config.UnionPayConfig : Activating UnionPay Service logger
2022-10-18 11:42:20.478 INFO 8226 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: MYSQL
2022-10-18 11:42:20.501 INFO 8226 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2022-10-18 11:42:20.730 INFO 8226 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter#7a0f06ad, org.springframework.security.web.context.SecurityContextPersistenceFilter#3e9fb485, org.springframework.security.web.header.HeaderWriterFilter#580ffea, org.springframework.security.web.authentication.logout.LogoutFilter#7fe87c0e, org.springframework.security.web.authentication.www.BasicAuthenticationFilter#c82d925, org.springframework.security.web.savedrequest.RequestCacheAwareFilter#38dbeb39, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter#48106381, org.springframework.security.web.authentication.AnonymousAuthenticationFilter#1fa9692b, org.springframework.security.web.session.SessionManagementFilter#2ffb0d10, org.springframework.security.web.access.ExceptionTranslationFilter#f76872f, org.springframework.security.web.access.intercept.FilterSecurityInterceptor#6df2a206]
2022-10-18 11:42:20.849 INFO 8226 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2022-10-18 11:42:20.849 INFO 8226 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'application.errorChannel' has 1 subscriber(s).
2022-10-18 11:42:20.849 INFO 8226 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started bean '_org.springframework.integration.errorLogger'
2022-10-18 11:42:20.867 INFO 8226 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2022-10-18 11:42:20.887 INFO 8226 --- [ main] com.eyemed.foodogs.application.App : Started App in 6.399 seconds (JVM running for 7.221)
Is my approach correct? Should I do something more? Please help me, as I do not know where to bump my head :)
Consider to move you logic into some #PostConstruct method instead.
I think getting access to bean factory and starting beans from the BeanFactoryPostProcessor is too early.
Also consider to use a SftpInboundChannelAdapterSpec instead of manual creation. And don't register those beans manually - rely on the IntegrationFlowContext.
I wanted to suggest you to look into this also: https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-rotating-server-advice.
But looks like you use that carrierId in the message handler. Although it may come as a message header. Not sure also if you really need that .channel(channelName) in between.
I am trying to read data from multiple csv file present in resources/input/user_*.csv
and writing in one csv file present in resources/output/user.csv
Below is my Config File
#Configuration
#EnableBatchProcessing
public class BatchConfig {
private static final Logger log = LoggerFactory.getLogger(BatchConfig.class);
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Value(value = "classpath:input/user_*.csv")
private Resource[] resources;
#Bean
public FlatFileItemReader<User> flatFileItemReader(){
log.info("flatFileItemReader execution started");
FlatFileItemReader<User> reader = new FlatFileItemReader<>();
reader.setName("ReadMultipleCsv");
reader.setLineMapper(lineMapper());
log.info("flatFileItemReader execution completed");
return reader;
}
private LineMapper<User> lineMapper(){
DefaultLineMapper<User> lineMapper = new DefaultLineMapper<>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(new String[] {"id","name","department","salary"});
BeanWrapperFieldSetMapper<User> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
fieldSetMapper.setTargetType(User.class);
lineMapper.setLineTokenizer(lineTokenizer);
lineMapper.setFieldSetMapper(fieldSetMapper);
return lineMapper;
}
#Bean
public MultiResourceItemReader<User> multiResourceItemReader(){
log.info("multiResourceItemReader() execution started");
MultiResourceItemReader<User> reader = new MultiResourceItemReader<>();
reader.setResources(resources);
reader.setDelegate(flatFileItemReader());
log.info("multiResourceItemReader() execution completed");
return reader;
}
#Bean
public FlatFileItemWriter<User> writer(){
log.info("writer() execution started");
FlatFileItemWriter<User> writer = new FlatFileItemWriter<>();
writer.setResource(new ClassPathResource("output/user.csv"));
writer.setAppendAllowed(true);
DelimitedLineAggregator<User> delimitedLineAggregator = new DelimitedLineAggregator<>();
delimitedLineAggregator.setDelimiter(",");
BeanWrapperFieldExtractor<User> beanWrapperFieldExtractor = new BeanWrapperFieldExtractor<>();
beanWrapperFieldExtractor.setNames(new String[] {"id","name","department","salary"});
delimitedLineAggregator.setFieldExtractor(beanWrapperFieldExtractor);
writer.setLineAggregator(delimitedLineAggregator);
log.info("writer() execution completed");
return writer;
}
#Bean
public Step step(){
return stepBuilderFactory.get("get-student").<User, User>chunk(5)
.reader(multiResourceItemReader())
.writer(writer()).build();
}
#Bean
public Job job(){
return jobBuilderFactory.get("process-student").incrementer(new RunIdIncrementer())
.flow(step()).end().build();
}
}
And this is my user entity
#Getter
#Setter
#AllArgsConstructor
#NoArgsConstructor
public class User {
private String id;
private String name;
private String department;
private String salary;
}
logs =>
2022-08-22 16:40:06.295 INFO 17264 --- [ main] c.s.SpringBatchMultipleCsvToApplication : Starting SpringBatchMultipleCsvToApplication using Java 1.8.0_121 on GBMLVVCSW3823 with PID 17264 (C:\Users\NavghS\Downloads\spring-batch-multiple-csv-to\spring-batch-multiple-csv-to\target\classes started by NavghS in C:\Users\NavghS\Downloads\spring-batch-multiple-csv-to\spring-batch-multiple-csv-to)
2022-08-22 16:40:06.300 INFO 17264 --- [ main] c.s.SpringBatchMultipleCsvToApplication : No active profile set, falling back to 1 default profile: "default"
2022-08-22 16:40:08.999 INFO 17264 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2022-08-22 16:40:09.017 INFO 17264 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2022-08-22 16:40:09.018 INFO 17264 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.65]
2022-08-22 16:40:09.349 INFO 17264 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2022-08-22 16:40:09.350 INFO 17264 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2936 ms
2022-08-22 16:40:09.542 INFO 17264 --- [ main] com.shrikant.config.BatchConfig : flatFileItemReader execution started
2022-08-22 16:40:09.554 INFO 17264 --- [ main] com.shrikant.config.BatchConfig : flatFileItemReader execution completed
2022-08-22 16:40:09.559 INFO 17264 --- [ main] com.shrikant.config.BatchConfig : multiResourceItemReader() execution started
2022-08-22 16:40:09.574 INFO 17264 --- [ main] com.shrikant.config.BatchConfig : multiResourceItemReader() execution completed
2022-08-22 16:40:09.576 INFO 17264 --- [ main] com.shrikant.config.BatchConfig : writer() execution started
2022-08-22 16:40:09.579 INFO 17264 --- [ main] com.shrikant.config.BatchConfig : writer() execution completed
2022-08-22 16:40:10.299 INFO 17264 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2022-08-22 16:40:10.313 INFO 17264 --- [ main] c.s.SpringBatchMultipleCsvToApplication : Started SpringBatchMultipleCsvToApplication in 5.205 seconds (JVM running for 6.176)
2022-08-22 16:40:10.316 INFO 17264 --- [ main] o.s.b.a.b.JobLauncherApplicationRunner : Running default command line with: []
2022-08-22 16:40:10.317 WARN 17264 --- [ main] o.s.b.c.c.a.DefaultBatchConfigurer : No datasource was provided...using a Map based JobRepository
2022-08-22 16:40:10.318 WARN 17264 --- [ main] o.s.b.c.c.a.DefaultBatchConfigurer : No transaction manager was provided, using a ResourcelessTransactionManager
2022-08-22 16:40:10.339 INFO 17264 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2022-08-22 16:40:10.393 INFO 17264 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=process-student]] launched with the following parameters: [{run.id=1}]
2022-08-22 16:40:10.612 INFO 17264 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [get-student]
2022-08-22 16:40:10.894 INFO 17264 --- [ main] o.s.batch.core.step.AbstractStep : Step: [get-student] executed in 282ms
2022-08-22 16:40:10.907 INFO 17264 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=process-student]] completed with the following parameters: [{run.id=1}] and the following status: [COMPLETED] in 344ms
When I run this code, I don't get error, but I don't see the data in output/user.csv file
I don't what's wrong, can anyone help me ?
Thanks
The problem is in the writer you have to use new FileSystemResource("src/main/resources/output/user.csv") instead of new ClassPathResource("output/user.csv"). With this you also don't have to worry to create the file first, it will create it if not exists.
#Bean
public FlatFileItemWriter<User> writer(){
log.info("writer() execution started");
FlatFileItemWriter<User> writer = new FlatFileItemWriter<>();
writer.setResource(new FileSystemResource("src/main/resources/output/user.csv"));
writer.setAppendAllowed(true);
DelimitedLineAggregator<User> delimitedLineAggregator = new DelimitedLineAggregator<>();
delimitedLineAggregator.setDelimiter(",");
BeanWrapperFieldExtractor<User> beanWrapperFieldExtractor = new BeanWrapperFieldExtractor<>();
beanWrapperFieldExtractor.setNames(new String[] {"id","name","department","salary"});
delimitedLineAggregator.setFieldExtractor(beanWrapperFieldExtractor);
writer.setLineAggregator(delimitedLineAggregator);
log.info("writer() execution completed");
return writer;
}
I am trying to use spring.gson.disable-html-escaping=false in my spring boot application.
but it is returning default value.
application.properties:
spring.mvc.converters.preferred-json-mapper=gson
spring.gson.disable-html-escaping=false
GsonHttpMessageConverterConfig.java
package com.example.demo;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.MediaType;
import org.springframework.http.converter.json.GsonHttpMessageConverter;
import java.util.*;
import java.util.stream.Collectors;
#Configuration
#Slf4j
public class GsonHttpMessageConverterConfig {
#Value("${grove.gson.http.converter.mediaTypes}")
private String configuredMediaTypes;
#Value("${spring.gson.disable-html-escaping}")
private String disabledHtmlEscaping;
#Bean
#Autowired
public GsonHttpMessageConverter gsonHttpMessageConverter(Gson gson) {
GsonHttpMessageConverter converter = new GsonHttpMessageConverter();
List<MediaType> configuredMediaTypes = httpConverterMediaTypes();
if (!configuredMediaTypes.isEmpty()){
converter.setSupportedMediaTypes(configuredMediaTypes);
}
converter.setGson(gson());
System.out.println("converter.getGson().htmlSafe() : "+converter.getGson().htmlSafe());
return converter;
}
#Bean
public Gson gson() {
//String disabledHtmlEscaping = System.getenv("spring.gson.disable-html-escaping");
final GsonBuilder builder = new GsonBuilder();
System.out.println("spring.gson.disable-html-escaping : "+disabledHtmlEscaping);
return builder.create();
}
private List<MediaType> httpConverterMediaTypes(){
//String configuredMediaTypes = System.getenv("grove.gson.http.converter.mediaTypes");
if (configuredMediaTypes != null && !configuredMediaTypes.trim().equals("")){
return Arrays.stream(configuredMediaTypes.split(","))
.map(mediaType -> mediaType.split("/"))
.map(mediaType -> new MediaType(mediaType[0],mediaType[1]))
.collect(Collectors.toList());
}
return Collections.emptyList();
}
}
can somebody help me why that property is not working.
I dont want to use builder.disableHtmlEscaping().create()
output is :
2022-03-28 11:20:06.930 INFO 11920 --- [ restartedMain] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.60]
2022-03-28 11:20:06.980 INFO 11920 --- [ restartedMain] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2022-03-28 11:20:06.980 INFO 11920 --- [ restartedMain] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1290 ms
spring.gson.disable-html-escaping : false
converter.getGson().htmlSafe() : true
2022-03-28 11:20:07.730 INFO 11920 --- [ restartedMain] o.s.b.d.a.OptionalLiveReloadServer : LiveReload server is running on port 35729
2022-03-28 11:20:07.735 INFO 11920 --- [ restartedMain] o.s.b.a.e.web.EndpointLinksResolver : Exposing 13 endpoint(s) beneath base path '/actuator'
2022-03-28 11:20:07.774 INFO 11920 --- [ restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2022-03-28 11:20:07.789 INFO 11920 --- [ restartedMain] com.example.demo.DemoApplication : Started DemoApplication in 2.566 seconds (JVM running for 2.965)
the expected output is :
converter.getGson().htmlSafe() is should return false if
I have a simple application to test Spring Boot.
Below is the configuration and the packages are set as per Spring documentation.
There is no error in application startup. Using Spring boot with eclipse.
Still the controller is not mapped to the server and when I do Post or Get it says :
{
"timestamp": 1547026379146,
"status": 404,
"error": "Not Found",
"message": "No message available",
"path": "/postdata"
}
Repository:
package com.abc.nice.repo;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
import com.abc.nice.entity.Messages;
#Repository
public interface MessagesRepository extends JpaRepository<Messages, Long> {
}
Entity:
package com.abc.nice.entity;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import javax.persistence.Entity;
import javax.persistence.FetchType;
import javax.persistence.CascadeType;
import javax.persistence.Column;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.JoinColumn;
import javax.persistence.JoinTable;
import javax.persistence.ManyToMany;
import javax.persistence.ManyToOne;
import javax.persistence.NamedQuery;
import javax.persistence.OneToMany;
import javax.persistence.Table;
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Size;
#Entity
#Table(name = "Messages")
public class Messages{
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#Size(max = 100)
private String toNumber;
#Size(max = 250)
private String hsm;
#Size(max = 250)
private String template;
#Size(max = 250)
private String parameters;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getTo() {
return toNumber;
}
public void setTo(String toNumber) {
this.toNumber = toNumber;
}
public String getHsm() {
return hsm;
}
public void setHsm(String hsm) {
this.hsm = hsm;
}
public String getTemplate() {
return template;
}
public void setTemplate(String template) {
this.template = template;
}
public String getParameters() {
return parameters;
}
public void setParameters(String parameters) {
this.parameters = parameters;
}
}
Controller :
package com.abc.nice.controller;
import java.util.List;
import java.util.Map;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;
import com.abc.nice.entity.Messages;
import com.abc.nice.repo.MessagesRepository;
#RestController
public class ControllerRest {
#Autowired
private Messages msgDao;
private MessagesRepository msgRepo;
RestTemplate restTemplate;
#GetMapping({ "/StatusData" })
public List<Messages> index() {
return this.msgRepo.findAll();
}
#PostMapping(path = {"/postdata"})
public ResponseEntity<Messages> createBody(#RequestBody Map<String, String> body) {
this.msgDao = new Messages();
this.msgDao.setTemplate((String) body.get("template"));
this.msgDao.setParameters((String) body.get("parameters"));
this.msgDao.setHsm((String) body.get("hsm"));
this.msgDao.setTo((String) body.get("to"));
return new ResponseEntity<Messages>(this.msgRepo.save(this.msgDao), HttpStatus.OK);
}
}
MainClass :
package com.abc.nice;
import org.hibernate.HibernateException;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.domain.EntityScan;
import org.springframework.boot.web.support.SpringBootServletInitializer;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
#SpringBootApplication
//#EntityScan("com.abc.nice.entity")
#ComponentScan({"com.abc.nice.entity","com.abc.nice.controller"})
#EnableJpaRepositories({"com.abc.nice.repo"})
//#EnableAutoConfiguration
public class TestApplication extends SpringBootServletInitializer {
public static void main(String[] args) {
SpringApplication.run(TestApplication .class, args);
}
}
Startup Log :
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.9.RELEASE)
2019-01-09 17:33:27.724 INFO 19732 --- [ main] com.abc.nice.TestApplication : Starting TestApplication on abcD02 with PID 19732 (C:\Users\abcd\Desktop\test\target\classes started by abcD in C:\Users\abcd\Desktop\test)
2019-01-09 17:33:27.728 INFO 19732 --- [ main] com.abc.nice.TestApplication : No active profile set, falling back to default profiles: default
2019-01-09 17:33:27.827 INFO 19732 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#4206a205: startup date [Wed Jan 09 17:33:27 SGT 2019]; root of context hierarchy
2019-01-09 17:33:29.489 INFO 19732 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http)
2019-01-09 17:33:29.506 INFO 19732 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2019-01-09 17:33:29.507 INFO 19732 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.23
2019-01-09 17:33:29.746 INFO 19732 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2019-01-09 17:33:29.746 INFO 19732 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 1922 ms
2019-01-09 17:33:30.005 INFO 19732 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean : Mapping servlet: 'dispatcherServlet' to [/]
2019-01-09 17:33:30.009 INFO 19732 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'characterEncodingFilter' to: [/*]
2019-01-09 17:33:30.009 INFO 19732 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2019-01-09 17:33:30.009 INFO 19732 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2019-01-09 17:33:30.009 INFO 19732 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'requestContextFilter' to: [/*]
2019-01-09 17:33:30.598 INFO 19732 --- [ main] j.LocalContainerEntityManagerFactoryBean : Building JPA container EntityManagerFactory for persistence unit 'default'
2019-01-09 17:33:30.610 INFO 19732 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [
name: default
...]
2019-01-09 17:33:30.680 INFO 19732 --- [ main] org.hibernate.Version : HHH000412: Hibernate Core {5.0.12.Final}
2019-01-09 17:33:30.682 INFO 19732 --- [ main] org.hibernate.cfg.Environment : HHH000206: hibernate.properties not found
2019-01-09 17:33:30.683 INFO 19732 --- [ main] org.hibernate.cfg.Environment : HHH000021: Bytecode provider name : javassist
2019-01-09 17:33:30.749 INFO 19732 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.0.1.Final}
2019-01-09 17:33:30.864 INFO 19732 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect
2019-01-09 17:33:31.033 INFO 19732 --- [ main] org.hibernate.tool.hbm2ddl.SchemaUpdate : HHH000228: Running hbm2ddl schema update
2019-01-09 17:33:31.058 INFO 19732 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2019-01-09 17:33:31.388 INFO 19732 --- [ main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for #ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#4206a205: startup date [Wed Jan 09 17:33:27 SGT 2019]; root of context hierarchy
2019-01-09 17:33:31.454 INFO 19732 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2019-01-09 17:33:31.455 INFO 19732 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2019-01-09 17:33:31.489 INFO 19732 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2019-01-09 17:33:31.489 INFO 19732 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2019-01-09 17:33:31.527 INFO 19732 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2019-01-09 17:33:31.866 INFO 19732 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2019-01-09 17:33:31.937 INFO 19732 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
2019-01-09 17:33:31.942 INFO 19732 --- [ main] com.abc.nice.WhatsapptestApplication : Started WhatsapptestApplication in 4.532 seconds (JVM running for 6.213)
2019-01-09 17:39:45.612 INFO 19732 --- [on(2)-127.0.0.1] inMXBeanRegistrar$SpringApplicationAdmin : Application shutdown requested.
2019-01-09 17:39:45.613 INFO 19732 --- [on(2)-127.0.0.1] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#4206a205: startup date [Wed Jan 09 17:33:27 SGT 2019]; root of context hierarchy
2019-01-09 17:39:45.616 INFO 19732 --- [on(2)-127.0.0.1] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
2019-01-09 17:39:45.617 INFO 19732 --- [on(2)-127.0.0.1] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
Application.properties:
#spring.datasource.type=org.apache.commons.dbcp.BasicDataSource
spring.datasource.url=jdbc:mysql://localhost:3306/ptpreconn?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC&useSSL=false
spring.datasource.username=root
spring.datasource.password=admin
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.jpa.show-sql=true
spring.data.jpa.repositories.enabled=true
#spring.jpa.database-platform=org.hibernate.dialect.MYSQL5Dialect
spring.jpa.hibernate.ddl-auto=update
spring.jpa.hibernate.naming.implicit-strategy=org.hibernate.boot.model.naming.ImplicitNamingStrategyLegacyJpaImpl
spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
server.port=${PORT:8080}
Please attach request from e.g. the developers' tools with HTTP verb that you are using and exact response from that request. I am concerned about #GetMapping - why it is in curly braces?
About post mapping, maybe you should describe consumes and produces part of mapping?
E.g.
#PostMapping(path = "/members", consumes = "application/json", produces = "application/json")
I have never used #PostMapping annotation without those parts.
UPDATE:
After removal of unnecessary component scan in Main class change controller as follow:
#RestController
public class ControllerRest {
#Autowired
private MessagesRepository msgRepo;
RestTemplate restTemplate;
#GetMapping({ "/StatusData" })
public List<Messages> index() {
return this.msgRepo.findAll();
}
#PostMapping(path = {"/postdata"})
public ResponseEntity<Messages> createBody(#RequestBody Map<String, String> body) {
Messages msgDao = new Messages();
msgDao.setTemplate((String) body.get("template"));
msgDao.setParameters((String) body.get("parameters"));
msgDao.setHsm((String) body.get("hsm"));
msgDao.setTo((String) body.get("to"));
return new ResponseEntity<Messages>(this.msgRepo.save(this.HttpStatus.OK);
}
}
Add a context path to your application.properties file or application.yml, whichever you are using.
server.servlet.contextPath=/springbootapi
Hit your endpoint: The endpoint for your POST method will be:
http://localhost:{PORT}/springbootapi/postdata
Why dont you statically supply your port. I think the port should be something like:
server.port=8080
I don't understand why you using #RequestBody with "Map<String,String>".
Try this code :
#Autowired
private MessagesRepository msgRepo;
#PostMapping("/postdata")
public ResponseEntity<Object> createBody(#RequestBody Message message) {
Message saveMessage= msgRepo.save(message);
URI location = ServletUriComponentsBuilder.fromCurrentRequest().path("/{id}")
.buildAndExpand(saveMessage.getId()).toUri();
return ResponseEntity.created(location).build();
}
[New to SpringBatch] Using Spring Boot, I am trying to create a job which reads names from MongoDB, converts to lowercase, and outputs to CSV file. My reader and processor are working but the writer isn't.
My code is as follows.
Configuration file:
package bbye;
import java.util.HashMap;
import java.util.Map;
import javax.sql.DataSource;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.data.MongoItemReader;
import org.springframework.batch.item.data.builder.MongoItemReaderBuilder;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor;
import org.springframework.batch.item.file.transform.DelimitedLineAggregator;
import org.springframework.batch.item.file.transform.FieldExtractor;
import org.springframework.batch.item.file.transform.LineAggregator;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.domain.Sort;
import org.springframework.data.mongodb.core.MongoTemplate;
import hello.Person;
#Configuration
#EnableBatchProcessing
public class BatchConfigProcessing {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
private MongoTemplate mongoTemplate;
private String readQuery = "{}";
// tag::readerwriterprocessor[]
#Bean
public MongoItemReader<Person> readMongo(DataSource dataSource) {
return new MongoItemReaderBuilder<Person>()
.name("mongoDocReader")
.jsonQuery(readQuery)
.targetType(Person.class)
.sorts(sort())
.template(mongoTemplate)
.collection("people")
.build();
}
#Bean
public PersonDocProcessor processor() {
return new PersonDocProcessor();
}
#Bean
public FlatFileItemWriter<Person> writer() {
/*FlatFileItemWriterBuilder<Person> writePerson = new FlatFileItemWriterBuilder<Person>();
writePerson.name("personDocWriter");
writePerson.resource(new ClassPathResource("PersonExtracted.csv"));
writePerson.lineAggregator(new DelimitedLineAggregator<Person>());
writePerson.shouldDeleteIfExists(true);
writePerson.build();*/
FlatFileItemWriter<Person> fileWriter = new FlatFileItemWriter<>();
fileWriter.setName("csvWriter");
fileWriter.setResource(new ClassPathResource("PersonExtracted.csv"));
fileWriter.setLineAggregator(lineAggregator());
fileWriter.setForceSync(true);
fileWriter.close();
return fileWriter;
}
// end::readerwriterprocessor[]
// tag::jobstep[]
#Bean
public Job exportUserJob(FileUploadNotificationListener listener, Step step1) {
return jobBuilderFactory.get("exportUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
}
#Bean
public Step step2(MongoItemReader<Person> reader) {
return stepBuilderFactory.get("step2")
.<Person, Person> chunk(10)
.reader(reader)
.processor(processor())
.writer(writer())
.build();
}
// end::jobstep[]
public FieldExtractor<Person> fieldExtractor()
{
BeanWrapperFieldExtractor<Person> extractor = new BeanWrapperFieldExtractor<>();
extractor.setNames( new String[] { "firstName",
"lastName"});
return extractor;
}
public LineAggregator<Person> lineAggregator() {
DelimitedLineAggregator<Person> la = new DelimitedLineAggregator<Person>();
la.setDelimiter(",");
la.setFieldExtractor(fieldExtractor());
return la;
}
public Map<String, Sort.Direction> sort(){
String firstName = "firstName";
Map<String, Sort.Direction> sortMap = new HashMap();
sortMap.put(firstName, Sort.DEFAULT_DIRECTION);
return sortMap;
}
}
Processor file
package bbye;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.item.ItemProcessor;
import org.springframework.stereotype.Component;
import hello.Person;
#Component
public class PersonDocProcessor implements ItemProcessor<Person, Person> {
private static final Logger log = LoggerFactory.getLogger(PersonDocProcessor.class);
#Override
public Person process(final Person person) throws Exception {
final String firstName = person.getFirstName().toLowerCase();
final String lastName = person.getLastName().toLowerCase();
final Person transformedPerson = new Person(firstName, lastName);
log.info("Converting (" + person + ") into (" + transformedPerson + ")");
return transformedPerson;
}
}
Listener
package bbye;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobExecutionListener;
import org.springframework.stereotype.Component;
#Component
public class FileUploadNotificationListener implements JobExecutionListener {
#Override
public void beforeJob(JobExecution jobExecution) {
System.out.println("===== listening for job - mongoReader - fileWriter ====");
}
#Override
public void afterJob(JobExecution jobExecution) {
System.out.println("==== file write job completed =====");
}
}
Here Person is a simple POJO. The stack trace with and without manual file creation is as follows:
If the file is not present under src/main/resources FlatFileItemWriter does not create a new file and throws
org.springframework.batch.item.ItemStreamException: Could not convert resource to file: [class path resource [PersonExtracted.csv]]
at org.springframework.batch.item.file.FlatFileItemWriter.getOutputState(FlatFileItemWriter.java:399) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.item.file.FlatFileItemWriter.open(FlatFileItemWriter.java:337) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
........
Caused by: java.io.FileNotFoundException: class path resource [PersonExtracted.csv] cannot be resolved to URL because it does not exist
If I create the PersonExtracted.csv file manually, the program runs without errors but does not write anything to the file. In fact, a blank file is returned. The stack trace is as below.
:: Spring Boot :: (v2.0.2.RELEASE)
2018-06-19 11:35:17.663 INFO 25136 --- [ main] hello.Application : Starting Application on MyPC with PID 25136 (C:\eclipse-workspace\gs-batch-processing-master\complete\target\classes started by shristi in C:\eclipse-workspace\gs-batch-processing-master\complete)
2018-06-19 11:35:17.666 INFO 25136 --- [ main] hello.Application : No active profile set, falling back to default profiles: default
2018-06-19 11:35:17.689 INFO 25136 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext#15bb6bea: startup date [Tue Jun 19 11:35:17 EDT 2018]; root of context hierarchy
2018-06-19 11:35:18.135 INFO 25136 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2018-06-19 11:35:18.136 WARN 25136 --- [ main] com.zaxxer.hikari.util.DriverDataSource : Registered driver with driverClassName=org.hsqldb.jdbcDriver was not found, trying direct instantiation.
2018-06-19 11:35:18.282 INFO 25136 --- [ main] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Driver does not support get/set network timeout for connections. (feature not supported)
2018-06-19 11:35:18.284 INFO 25136 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2018-06-19 11:35:18.293 INFO 25136 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executing SQL script from URL [file:/C:/eclipse-workspace/gs-batch-processing-master/complete/target/classes/schema-all.sql]
2018-06-19 11:35:18.297 INFO 25136 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executed SQL script from URL [file:/C:/eclipse-workspace/gs-batch-processing-master/complete/target/classes/schema-all.sql] in 4 ms.
2018-06-19 11:35:18.518 INFO 25136 --- [ main] org.mongodb.driver.cluster : Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
2018-06-19 11:35:18.552 INFO 25136 --- [localhost:27017] org.mongodb.driver.connection : Opened connection [connectionId{localValue:1, serverValue:140}] to localhost:27017
2018-06-19 11:35:18.554 INFO 25136 --- [localhost:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 0]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1438717}
2018-06-19 11:35:18.723 INFO 25136 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: HSQL
2018-06-19 11:35:18.770 INFO 25136 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2018-06-19 11:35:18.778 INFO 25136 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql]
2018-06-19 11:35:18.781 INFO 25136 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 3 ms.
2018-06-19 11:35:18.870 INFO 25136 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2018-06-19 11:35:18.871 INFO 25136 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Bean with name 'dataSource' has been autodetected for JMX exposure
2018-06-19 11:35:18.873 INFO 25136 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]
2018-06-19 11:35:18.880 INFO 25136 --- [ main] hello.Application : Started Application in 1.357 seconds (JVM running for 2.284)
2018-06-19 11:35:18.881 INFO 25136 --- [ main] o.s.b.a.b.JobLauncherCommandLineRunner : Running default command line with: []
2018-06-19 11:35:18.908 INFO 25136 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=exportUserJob]] launched with the following parameters: [{run.id=1}]
===== listening for job - mongoReader - fileWriter ====
2018-06-19 11:35:18.917 INFO 25136 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [step2]
2018-06-19 11:35:18.995 INFO 25136 --- [ main] org.mongodb.driver.connection : Opened connection [connectionId{localValue:2, serverValue:141}] to localhost:27017
2018-06-19 11:35:19.022 INFO 25136 --- [ main] bbye.PersonDocProcessor : Converting (firstName: ALICE, lastName: WONDERLAND) into (firstName: alice, lastName: wonderland)
2018-06-19 11:35:19.022 INFO 25136 --- [ main] bbye.PersonDocProcessor : Converting (firstName: FIRSTNAME, lastName: LASTNAME) into (firstName: firstname, lastName: lastname)
2018-06-19 11:35:19.022 INFO 25136 --- [ main] bbye.PersonDocProcessor : Converting (firstName: JANE, lastName: DOE) into (firstName: jane, lastName: doe)
2018-06-19 11:35:19.022 INFO 25136 --- [ main] bbye.PersonDocProcessor : Converting (firstName: JOHN, lastName: DOE) into (firstName: john, lastName: doe)
2018-06-19 11:35:19.022 INFO 25136 --- [ main] bbye.PersonDocProcessor : Converting (firstName: MARK, lastName: WINN) into (firstName: mark, lastName: winn)
==== file write job completed =====
2018-06-19 11:35:19.031 INFO 25136 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=exportUserJob]] completed with the following parameters: [{run.id=1}] and the following status: [COMPLETED]
2018-06-19 11:35:19.032 INFO 25136 --- [ Thread-2] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext#15bb6bea: startup date [Tue Jun 19 11:35:17 EDT 2018]; root of context hierarchy
2018-06-19 11:35:19.033 INFO 25136 --- [ Thread-2] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
2018-06-19 11:35:19.034 INFO 25136 --- [ Thread-2] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans
2018-06-19 11:35:19.035 INFO 25136 --- [ Thread-2] org.mongodb.driver.connection : Closed connection [connectionId{localValue:2, serverValue:141}] to localhost:27017 because the pool has been closed.
2018-06-19 11:35:19.036 INFO 25136 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2018-06-19 11:35:19.037 INFO 25136 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
I think we should use FileSystemResource instead of ClassPathResource. Could you please try and let us know.
Answering the question. Turns out the writer was working fine but I was looking at the wrong file.
When using ClassPathResource the file gets created and updated under target/classes directory. However, I was looking at PersonExtracted.csv under src/main/resources directory, which was never updated.
If I specify FileSystemResource, the file gets created and updated at specified location.