I have a consumer.properties file with the following contents in src/main/resources, and an accompanying Configuration class that loads and stores the file's contents into class member variables:
//consumer.properties file in src/main/resources:
com.training.consumer.hostname=myhost
com.training.consumer.username=myusername
com.training.consumer.password=mypassword
//ConsumerConfig.java
#Configuration
#PropertySource(
value= {"classpath:consumer.properties"}
)
#ConfigurationProperties(prefix="com.training.consumer")
public class ConsumerConfig {
private String hostname;
private String username;
private String password;
public ConsumerConfig() { }
public String getHostname() {
return hostname;
}
public void setHostname(String hostname) {
this.hostname = hostname;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
#Override
public String toString() {
return "ConsumerConfig [hostname=" + hostname + ", username=" + username + ", password=" + password + "]";
}
}
I also have a ConfigsService class that autowires the ConsumerConfig class to retrieve the individual properties:
#Component
public class ConfigsService {
#Autowired
ConsumerConfig consumerConfig;
public ConsumerConfig getConsumerConfig() {
return consumerConfig;
}
public void showConfig() {
consumerConfig.toString();
}
public ConsumerConfig getConfig() {
return consumerConfig;
}
}
The properties are loaded up just fine when running the ConfigsService's methods. The problem is in the unit tests, where invoking configService.getConfig().getHostname() returns a null value -- even after having created a src/test/resources directory, and adding my consumer.properties file in it:
#TestPropertySource("classpath:consumer.properties")
public class ConfigsServiceTest {
#Mock
ConsumerConfig consumerConfig;
#InjectMocks
ConfigsService configService;
#Before
public void beforeEach() {
MockitoAnnotations.initMocks(this);
}
#Test
public void someTest() {
System.out.println(configService.getConfig().getHostname()); //outputs null here -- wth!
Assert.assertTrue(true);
}
}
you are getting null values because of the mock object. I will suggest using spring runner with context configuration which will load properties in the config class and create the bean.
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {ConsumerConfig.class, ConfigsService.class})
#TestPropertySource(locations = "classpath:consumer.properties")
public class ConfigsServiceTest {
#Autowired
private ConfigsService configsService;
#Test
public void someTest() {
Assert.assertNotNull(configService.getConfig().getHostname());
}
}
The error may be because in the ConfigsServiceTest class you are establishing that your ConsumerConfig is a Mock and that is why it is not loading your configuration if you remove that code it should work
Related
I want to get values from my application.properties from spring.datasource.* fields but those fields are null.
Here is what I'm doing:
#Configuration
#ComponentScan
public class DatabaseConfig {
#Value("${spring.datasource.url}")
private String url;
#Value("${spring.datasource.username}")
private String username;
#Value("${spring.datasource.password}")
private String password;
#Bean(name = "database_url")
public String getDatabaseUrl() {
return url+"?user="+username+"&password="+password;
}
}
application.properties
spring.datasource.url=jdbc:postgresql://db:5432/endlessblow_db
spring.datasource.username=kuba
spring.datasource.password=pass
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.platform=postgres
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect
this db in url is my docker postgre container host and it works fine.
EDIT
DatabaseConnection.java
public class DatabaseConnection {
private String databaseUrl;
#Autowired
void setDatabaseUrl(#Qualifier("database_url") String databaseUrl) {
this.databaseUrl = databaseUrl;
}
private static DatabaseConnection instance;
private Connection connection;
private DatabaseConnection() {
try {
Class.forName("org.postgresql.Driver");
this.connection = DriverManager.getConnection(databaseUrl);
} catch (Exception ex) {
System.out.println("Database Connection Creation Failed : " + ex.getMessage());
}
}
public Connection getConnection() {
return connection;
}
public static DatabaseConnection getInstance() throws SQLException, URISyntaxException, IOException {
if (instance == null) {
instance = new DatabaseConnection();
} else if (instance.getConnection().isClosed()) {
instance = new DatabaseConnection();
}
return instance;
}
}
What's wrong with this code? Thank you!
For some reasons #Value annotated fields are not initialized I guess. try sending values as #Bean method params.
#Configuration
#ComponentScan
public class DatabaseConfig {
#Bean(name = "database_url")
public String getDatabaseUrl(#Value("${spring.datasource.url}") String url,
#Value("${spring.datasource.username}")String username,
#Value("${spring.datasource.password}") String password
) {
return url+"?user="+username+"&password="+password;
}
}
String is immutable class, meaning each time a new String is created normally a new String might be created in the string pool and a new reference might be returned.
For this reason you can't have String object as a bean in your application context.
#Bean(name = "database_url")
public String getDatabaseUrl() { <----------------The return type of String is wrong
return url+"?user="+username+"&password="+password;
}
So just create a custom object in this #Bean method which could be transferred and used in the application context
Why not return a DatabaseConnection from that #Bean method this would seem the optimal solution in your problem.
You have to create a new file where you 2st read data from property file
#Configuration
#Getter
#Setter
#ConfigurationProperties(prefix = "spring.datasource")
public class DatabaseLoginProperties{
private String url;
private String username;
private String password;
}
----Now above value you can uses in your java file -------------------
#Configuration
public class DatabaseConfig {
#Autowired
private DatabaseLoginProperties databaseLoginProperties;
#Bean
public String getDatabaseUrl() {
return databaseLoginProperties.getUrl()+"?user="+databaseLoginProperties.getUsername()+"&password="+databaseLoginProperties.getPassword();
}
}
EDITS BASED ON SUGGESTION:
For brevity, I will remove older code and long part and re-phrase the issue.
I am trying to build the app (Spring boot + Spring Batch) taking the date and config information from command line. Based on suggestions, I can use the application properties?
The main aim is to use the same job (task of the job) to download different files form different host/time etc. So, properties file can give the information to use for download and compiled jar should read the info and do its tasks.
Main Entry point.
#SpringBootApplication
public class CoreApplication implements ApplicationRunner {
#Autowired
JobLauncher jobLauncher;
#Autowired
Job processJob;
#Value("${rundate}")
private String run_date;
private static final Logger logger = LoggerFactory.getLogger(CoreApplication.class);
public static void main(String[] args) {
SpringApplication.run(CoreApplication.class, args);
}
#Override
public void run(ApplicationArguments args) throws Exception {
JobParameters jobParameters = new JobParametersBuilder()
.addLong("JobID", System.currentTimeMillis())
.addString("RunDate", run_date)
.toJobParameters();
try {
jobLauncher.run(processJob, jobParameters);
} catch (Exception e) {
logger.error("Exception while running a batch job {}", e.getMessage());
}
}
}
I rearranged the code, to use the values of server, user, etc from application.properties file. Please let me know if it is wrong way to inject the properties.
application.properties file:
spring.datasource.url=jdbc:postgresql://dbhost:1000/db
spring.datasource.username=username
spring.datasource.password=password
spring.datasource.platform=postgresql
spring.batch.job.enabled=false
local.directory="/my/local/path/"
file.name="file_name_20200601.csv"
remote.directory="/remote/ftp/location"
remote.host="remotehost"
remote.port=22
remote.user="remoteuser"
private.key.location="/key/file/location"
My Batch Configuration:
#Configuration
#EnableBatchProcessing
#EnableIntegration
#EnableAutoConfiguration
public class BatchConfiguration {
private Logger logger = LoggerFactory.getLogger(BatchConfiguration.class);
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
public Job ftpJob() {
return jobBuilderFactory.get("FTP Job")
.incrementer(new RunIdIncrementer())
.start(getFilesFromFTPServer())
.build();
}
#Bean
public Step getFilesFromFTPServer() {
return stepBuilderFactory.get("Get file from server")
.tasklet(new RemoteFileInboundTasklet())
.build();
}
}
My Tasklet:
public class RemoteFileInboundTasklet implements Tasklet {
private Logger logger = LoggerFactory.getLogger(RemoteFileInboundTasklet.class);
#Value("${file.name}")
private String fileNamePattern;
private String clientName;
private boolean deleteLocalFiles = true;
private boolean retryIfNotFound = false;
#Value("${local.directory}")
private String local_directory_value;
private File localDirectory;
private int downloadFileAttempts = 12;
private long retryIntervalMilliseconds = 300000;
#Value("${remote.directory}")
private String remoteDirectory;
#Value("${remote.host}")
private String remoteHost;
#Value("${remote.user}")
private String remoteUser;
#Value("${remote.port}")
private int remotePort;
#Value("${private.key.location}")
private String private_key_file;
public SessionFactory<ChannelSftp.LsEntry> clientSessionFactory() {
DefaultSftpSessionFactory ftpSessionFactory = new DefaultSftpSessionFactory();
ftpSessionFactory.setHost(remoteHost);
ftpSessionFactory.setPort(remotePort);
ftpSessionFactory.setUser(remoteUser);
ftpSessionFactory.setPrivateKey(new FileSystemResource(private_key_file));
ftpSessionFactory.setAllowUnknownKeys(true);
return ftpSessionFactory;
}
private SessionFactory sessionFactory = clientSessionFactory();
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer sftpInboundFileSynchronizer = new SftpInboundFileSynchronizer(sessionFactory);
sftpInboundFileSynchronizer.setDeleteRemoteFiles(false);
sftpInboundFileSynchronizer.setRemoteDirectory(remoteDirectory);
return sftpInboundFileSynchronizer;
}
private SftpInboundFileSynchronizer ftpInboundFileSynchronizer = sftpInboundFileSynchronizer();
private SftpInboundFileSynchronizingMessageSource sftpInboundFileSynchronizingMessageSource;
public boolean isDeleteLocalFiles() {
return deleteLocalFiles;
}
public void setDeleteLocalFiles(boolean deleteLocalFiles) {
this.deleteLocalFiles = deleteLocalFiles;
}
public SftpInboundFileSynchronizer getFtpInboundFileSynchronizer() {
return ftpInboundFileSynchronizer;
}
public void setFtpInboundFileSynchronizer(SftpInboundFileSynchronizer ftpInboundFileSynchronizer) {
this.ftpInboundFileSynchronizer = ftpInboundFileSynchronizer;
}
public SessionFactory getSessionFactory() {
return sessionFactory;
}
public void setSessionFactory(SessionFactory sessionFactory) {
this.sessionFactory = sessionFactory;
}
public SftpInboundFileSynchronizingMessageSource getSftpInboundFileSynchronizingMessageSource() {
return sftpInboundFileSynchronizingMessageSource;
}
public void setSftpInboundFileSynchronizingMessageSource(SftpInboundFileSynchronizingMessageSource sftpInboundFileSynchronizingMessageSource) {
this.sftpInboundFileSynchronizingMessageSource = sftpInboundFileSynchronizingMessageSource;
}
public String getRemoteDirectory() {
return remoteDirectory;
}
public void setRemoteDirectory(String remoteDirectory) {
this.remoteDirectory = remoteDirectory;
}
private SFTPGateway sftpGateway;
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler clientMessageHandler() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(clientSessionFactory(), "mget", "payload");
sftpOutboundGateway.setAutoCreateLocalDirectory(true);
sftpOutboundGateway.setLocalDirectory(new File(local_directory_value));
sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
sftpOutboundGateway.setFilter(new AcceptOnceFileListFilter<>());
return sftpOutboundGateway;
}
private void deleteLocalFiles()
{
if (deleteLocalFiles)
{
localDirectory = new File(local_directory_value);
SimplePatternFileListFilter filter = new SimplePatternFileListFilter(fileNamePattern);
List<File> matchingFiles = filter.filterFiles(localDirectory.listFiles());
if (CollectionUtils.isNotEmpty(matchingFiles))
{
for (File file : matchingFiles)
{
FileUtils.deleteQuietly(file);
}
}
}
}
#Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
deleteLocalFiles();
ftpInboundFileSynchronizer.synchronizeToLocalDirectory(localDirectory);
if (retryIfNotFound) {
SimplePatternFileListFilter filter = new SimplePatternFileListFilter(fileNamePattern);
int attemptCount = 1;
while (filter.filterFiles(localDirectory.listFiles()).size() == 0 && attemptCount <= downloadFileAttempts) {
logger.info("File(s) matching " + fileNamePattern + " not found on remote site. Attempt " + attemptCount + " out of " + downloadFileAttempts);
Thread.sleep(retryIntervalMilliseconds);
ftpInboundFileSynchronizer.synchronizeToLocalDirectory(localDirectory);
attemptCount++;
}
if (attemptCount >= downloadFileAttempts && filter.filterFiles(localDirectory.listFiles()).size() == 0) {
throw new FileNotFoundException("Could not find remote file(s) matching " + fileNamePattern + " after " + downloadFileAttempts + " attempts.");
}
}
return RepeatStatus.FINISHED;
}
public String getFileNamePattern() {
return fileNamePattern;
}
public void setFileNamePattern(String fileNamePattern) {
this.fileNamePattern = fileNamePattern;
}
public String getClientName() {
return clientName;
}
public void setClientName(String clientName) {
this.clientName = clientName;
}
public boolean isRetryIfNotFound() {
return retryIfNotFound;
}
public void setRetryIfNotFound(boolean retryIfNotFound) {
this.retryIfNotFound = retryIfNotFound;
}
public File getLocalDirectory() {
return localDirectory;
}
public void setLocalDirectory(File localDirectory) {
this.localDirectory = localDirectory;
}
public int getDownloadFileAttempts() {
return downloadFileAttempts;
}
public void setDownloadFileAttempts(int downloadFileAttempts) {
this.downloadFileAttempts = downloadFileAttempts;
}
public long getRetryIntervalMilliseconds() {
return retryIntervalMilliseconds;
}
public void setRetryIntervalMilliseconds(long retryIntervalMilliseconds) {
this.retryIntervalMilliseconds = retryIntervalMilliseconds;
}
}
My understanding (please correct here if wrong) that the application.properties file properties can be injected in the tasklet (as above).
Then I try to build the package.
mvn clean package
I get the following error:
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.batch.core.Step]: Factory method 'getFilesFromFTPServer' threw exception; nested exception is java.lang.IllegalArgumentException: Path must not be null
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
... 122 common frames omitted
Caused by: java.lang.IllegalArgumentException: Path must not be null
at org.springframework.util.Assert.notNull(Assert.java:198) ~[spring-core-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.core.io.FileSystemResource.<init>(FileSystemResource.java:80) ~[spring-core-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at com.my.batch.core.tasklet.RemoteFileInboundTasklet.clientSessionFactory(RemoteFileInboundTasklet.java:78) ~[classes/:na]
at com.my.batch.core.tasklet.RemoteFileInboundTasklet.<init>(RemoteFileInboundTasklet.java:83) ~[classes/:na]
at com.my.batch.core.BatchConfiguration.getFilesFromFTPServer(BatchConfiguration.java:71) ~[classes/:na]
at com.my.batch.core.BatchConfiguration$$EnhancerBySpringCGLIB$$17d8a6d9.CGLIB$getFilesFromFTPServer$1(<generated>) ~[classes/:na]
The line in the code is:
ftpSessionFactory.setPrivateKey(new FileSystemResource(private_key_file));
called via BatchConfiguration.java -> getFilesFromFTPServer.
This means my values from applcation.properties is not passed?
What changes I need to do?
And, while compiling or building the jar, why is it checking the value of variable?
NEW EDITS:
I tried to declare my tasklet as a bean in Configuration and build the package again. However, it is giving the same error.
My application.properties file after change:
spring.datasource.url=jdbc:postgresql://dbhost:1000/db
spring.datasource.username=username
spring.datasource.password=password
spring.datasource.platform=postgresql
spring.batch.job.enabled=false
local.directory=/my/local/path/
file.name=file_name_20200601.csv
remote.directory=/remote/ftp/location
remote.host=remotehost
remote.port=22
remote.user=remoteuser
private.key.location=/key/file/location
No change in tasklet.
Changed Configuration:
#Configuration
#EnableBatchProcessing
#EnableIntegration
#EnableAutoConfiguration
public class BatchConfiguration {
private Logger logger = LoggerFactory.getLogger(BatchConfiguration.class);
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
public RemoteFileInboundTasklet remoteFileInboundTasklet() {
return new RemoteFileInboundTasklet();
}
#Bean
public Job ftpJob() {
return jobBuilderFactory.get("FTP Job")
.incrementer(new RunIdIncrementer())
.start(getFilesFromFTPServer())
.build();
}
#Bean
public Step getFilesFromFTPServer() {
return stepBuilderFactory.get("Get file from server")
.tasklet(remoteFileInboundTasklet())
.build();
}
}
When I tried to build the package(mvn clean package), I still get the same error.
Path must not be null.
It is not able to read the properties. Any idea what is wrong?
EDITS BASED ON DIFFERENT APPROACH:
I tried to further see how to use configuration and found the following approach to use the #ConfigurationProperties annotation (How to access a value defined in the application.properties file in Spring Boot)
I created a new ftp config class:
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
#ConfigurationProperties(prefix = "ftp")
#Configuration("coreFtpProperties")
public class CoreFtp {
private String host;
private String port;
private String user;
private String passwordKey;
private String localDirectory;
private String remoteDirectory;
private String fileName;
public String getHost() {
return host;
}
public void setHost(String host) {
this.host = host;
}
public String getPort() {
return port;
}
public void setPort(String port) {
this.port = port;
}
public String getUser() {
return user;
}
public void setUser(String user) {
this.user = user;
}
public String getPasswordKey() {
return passwordKey;
}
public void setPasswordKey(String passwordKey) {
this.passwordKey = passwordKey;
}
public String getLocalDirectory() {
return localDirectory;
}
public void setLocalDirectory(String localDirectory) {
this.localDirectory = localDirectory;
}
public String getRemoteDirectory() {
return remoteDirectory;
}
public void setRemoteDirectory(String remoteDirectory) {
this.remoteDirectory = remoteDirectory;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
}
Minor change to application.properties file:
spring.datasource.url=jdbc:postgresql://dbhost:1000/db
spring.datasource.username=username
spring.datasource.password=password
spring.datasource.platform=postgresql
spring.batch.job.enabled=false
ftp.local_directory=/my/local/path/
ftp.file_name=file_name_20200601.csv
ftp.remote_directory=/remote/ftp/location
ftp.host=remotehost
ftp.port=22
ftp.user=remoteuser
ftp.password_key=/key/file/location
In my batch configuration I made this changes:
#Configuration
#EnableBatchProcessing
#EnableIntegration
public class BatchConfiguration {
private Logger logger = LoggerFactory.getLogger(BatchConfiguration.class);
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
private CoreFtp coreFtpProperties;
#Bean
public RemoteFileInboundTasklet remoteFileInboundTasklet() {
RemoteFileInboundTasklet ftpTasklet = new RemoteFileInboundTasklet();
ftpTasklet.setRetryIfNotFound(true);
ftpTasklet.setDownloadFileAttempts(3);
ftpTasklet.setRetryIntervalMilliseconds(10000);
ftpTasklet.setFileNamePattern(coreFtpProperties.getFileName());
ftpTasklet.setRemoteDirectory(coreFtpProperties.getRemoteDirectory());
ftpTasklet.setLocalDirectory(new File(coreFtpProperties.getLocalDirectory()));
ftpTasklet.setSessionFactory(clientSessionFactory());
ftpTasklet.setFtpInboundFileSynchronizer(sftpInboundFileSynchronizer());
ftpTasklet.setSftpInboundFileSynchronizingMessageSource(new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer()));
return ftpTasklet;
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer sftpInboundFileSynchronizer = new SftpInboundFileSynchronizer(clientSessionFactory());
sftpInboundFileSynchronizer.setDeleteRemoteFiles(false);
sftpInboundFileSynchronizer.setRemoteDirectory(coreFtpProperties.getRemoteDirectory());
return sftpInboundFileSynchronizer;
}
#Bean(name = "clientSessionFactory")
public SessionFactory<LsEntry> clientSessionFactory() {
DefaultSftpSessionFactory ftpSessionFactory = new DefaultSftpSessionFactory();
ftpSessionFactory.setHost(coreFtpProperties.getHost());
ftpSessionFactory.setPort(Integer.parseInt(coreFtpProperties.getPort()));
ftpSessionFactory.setUser(coreFtpProperties.getUser());
ftpSessionFactory.setPrivateKey(new FileSystemResource(coreFtpProperties.getPasswordKey()));
ftpSessionFactory.setPassword("");
ftpSessionFactory.setAllowUnknownKeys(true);
return ftpSessionFactory;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler clientMessageHandler() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(clientSessionFactory(), "mget", "payload");
sftpOutboundGateway.setAutoCreateLocalDirectory(true);
sftpOutboundGateway.setLocalDirectory(new File(coreFtpProperties.getLocalDirectory()));
sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
sftpOutboundGateway.setFilter(new AcceptOnceFileListFilter<>());
return sftpOutboundGateway;
}
#Bean
public Job ftpJob() {
return jobBuilderFactory.get("FTP Job")
.incrementer(new RunIdIncrementer())
.start(getFilesFromFTPServer())
.build();
}
#Bean
public Step getFilesFromFTPServer() {
return stepBuilderFactory.get("Get file from server")
.tasklet(remoteFileInboundTasklet())
.build();
}
}
So, accordingly my Tasklet is changed as:
public class RemoteFileInboundTasklet implements Tasklet {
private Logger logger = LoggerFactory.getLogger(RemoteFileInboundTasklet.class);
private String fileNamePattern;
private String clientName;
private boolean deleteLocalFiles = true;
private boolean retryIfNotFound = false;
private File localDirectory;
private int downloadFileAttempts = 12;
private long retryIntervalMilliseconds = 300000;
private String remoteDirectory;
private SessionFactory sessionFactory;
private SftpInboundFileSynchronizer ftpInboundFileSynchronizer;
private SftpInboundFileSynchronizingMessageSource sftpInboundFileSynchronizingMessageSource;
public boolean isDeleteLocalFiles() {
return deleteLocalFiles;
}
public void setDeleteLocalFiles(boolean deleteLocalFiles) {
this.deleteLocalFiles = deleteLocalFiles;
}
public SftpInboundFileSynchronizer getFtpInboundFileSynchronizer() {
return ftpInboundFileSynchronizer;
}
public void setFtpInboundFileSynchronizer(SftpInboundFileSynchronizer ftpInboundFileSynchronizer) {
this.ftpInboundFileSynchronizer = ftpInboundFileSynchronizer;
}
public SessionFactory getSessionFactory() {
return sessionFactory;
}
public void setSessionFactory(SessionFactory sessionFactory) {
this.sessionFactory = sessionFactory;
}
public SftpInboundFileSynchronizingMessageSource getSftpInboundFileSynchronizingMessageSource() {
return sftpInboundFileSynchronizingMessageSource;
}
public void setSftpInboundFileSynchronizingMessageSource(SftpInboundFileSynchronizingMessageSource sftpInboundFileSynchronizingMessageSource) {
this.sftpInboundFileSynchronizingMessageSource = sftpInboundFileSynchronizingMessageSource;
}
public String getRemoteDirectory() {
return remoteDirectory;
}
public void setRemoteDirectory(String remoteDirectory) {
this.remoteDirectory = remoteDirectory;
}
private SFTPGateway sftpGateway;
private void deleteLocalFiles()
{
if (deleteLocalFiles)
{
SimplePatternFileListFilter filter = new SimplePatternFileListFilter(fileNamePattern);
List<File> matchingFiles = filter.filterFiles(localDirectory.listFiles());
if (CollectionUtils.isNotEmpty(matchingFiles))
{
for (File file : matchingFiles)
{
FileUtils.deleteQuietly(file);
}
}
}
}
#Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
deleteLocalFiles();
ftpInboundFileSynchronizer.synchronizeToLocalDirectory(localDirectory);
if (retryIfNotFound) {
SimplePatternFileListFilter filter = new SimplePatternFileListFilter(fileNamePattern);
int attemptCount = 1;
while (filter.filterFiles(localDirectory.listFiles()).size() == 0 && attemptCount <= downloadFileAttempts) {
logger.info("File(s) matching " + fileNamePattern + " not found on remote site. Attempt " + attemptCount + " out of " + downloadFileAttempts);
Thread.sleep(retryIntervalMilliseconds);
ftpInboundFileSynchronizer.synchronizeToLocalDirectory(localDirectory);
attemptCount++;
}
if (attemptCount >= downloadFileAttempts && filter.filterFiles(localDirectory.listFiles()).size() == 0) {
throw new FileNotFoundException("Could not find remote file(s) matching " + fileNamePattern + " after " + downloadFileAttempts + " attempts.");
}
}
return RepeatStatus.FINISHED;
}
}
Based on above changes, I am able to compile the code and create the necessary Jar, and run the code using the jar.
You are declaring a bean jobExecutionListener() in which you create new FileSystemResource(config_file_path);. The config_file_path is injected from job parameters #Value("#{jobParameters['ConfigFilePath']}") which are not available at configuration time but only when a job/step is run. This is called late binding.
So in your case, when Spring tries to create the bean jobExecutionListener(), it tries to inject config_file_path but it is null at that time (at this point Spring is only creating beans to configure the application context) and the job is not run yet hence the method beforeJob is not executed yet. This is the reason you have a NullPointerException. Adding #JobScope on the jobExecutionListener() bean should fix the issue but I do not recommend that. The reason is that you are trying to configure some properties in the wrong way and in the wrong place, so I would fix that design instead of working around the issue by adding an annotation.
Job parameters are used for business parameters and not technical details. In your case, runDate is a good choice for a job parameter but not ConfigFilePath. Moreover, since you use Spring, why do you inject the file path then do properties = PropertiesLoaderUtils.loadProperties(resource); and Integer.parseInt(properties.getProperty("remote.port"));? Spring will do that for you if tell it to inject properties where needed.
I would remove this config_file_path job parameter as well as the job listener and inject the properties in the remoteFileInboundTasklet directly, that is, as close as possible to where these properties are needed.
Edit: Add code example
Can you help to understand where can I declare the tasklet as a bean?
In your step getFilesFromFTPServer , you are creating the tasklet manually, so dependency injection is not performed. You need to declare the tasklet as a Spring bean for this to work, something like:
#Bean
public Tasklet myTasklet() {
return new RemoteFileInboundTasklet()
}
#Bean
public Step getFilesFromFTPServer() {
return stepBuilderFactory.get("Get file from server")
.tasklet(myTasklet())
.build();
}
You need to change getFilesFromFTPServer bean to JobScope and read all the job runtime parameters from there.
#Bean
#JobScope
public Step getFilesFromFTPServer() {
I'm using Spring Boot 2.0 with default application.yml properties file. I would like to split it to separate property files because it becomes huge.
Also I would like to write tests to check properties correctness: values that will present on production application context (not the test one).
Here is my property file: src/main/resources/config/custom.yml
my-property:
value: 'test'
Property class:
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
#Data
#Configuration
#ConfigurationProperties(prefix = "my-property")
#PropertySource("classpath:config/custom.yml")
public class MyProperty {
private String value;
}
Test:
import static org.junit.Assert.assertEquals;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
#RunWith(SpringRunner.class)
#SpringBootTest(classes = MyProperty.class)
#EnableConfigurationProperties
public class MyPropertyTest {
#Autowired
private MyProperty property;
#Test
public void test() {
assertEquals("test", property.getValue());
}
}
But test fails with error:
java.lang.AssertionError:
Expected :test
Actual :null
Also I see that property value is null when running the application by printing it in ApplicationRunner.
When I used application.yml for all properties it were well with the same configuration.
How to put correct configuration for properties and tests for make it work?
Link to Github repo
Finely I found the right way for having custom yaml properties in my app.
The issue is that Spring doesn't support yaml files as #PropertySource (link to issue). And here is a workaround how to deal with that described in spring documentation.
So, to be able to load properties from yaml files you need:
* To implement EnvironmentPostProcessor
* To register it in spring.factories
Please visit this github repo for complete example.
Also, thanks a lot for your support, guys!
#TestPropertySource can solve your problem.
#RunWith(SpringRunner.class)
#SpringBootTest(classes = MyProperty.class)
#TestPropertySource(locations="classpath:test.properties")
public class MyPropertyTest {
#Autowired
private MyProperty property;
#Test
public void test() {
assertEquals("test", property.getValue());
}
}
Hope it helps.
I am a little late to the party, but this might also help. The solution provided as answer is the best approach so far, but here is an alternative that I used
Make use of profiles and modify the PropertySoucesPlaceHolderConfiguration bean to do load the necessary property files based on the profiles. It loads the application.properties as the default one but the other propertyfiles -oauth_DEV and oauth_QA are loaded based on the profiles set
#Bean
public PropertySourcesPlaceholderConfigurer propertyPlaceholderConfigurerconfigurer() {
System.out.println("Inside Placeholder bean");
PropertySourcesPlaceholderConfigurer cfg = new PropertySourcesPlaceholderConfigurer();
ClassPathResource cls1= new ClassPathResource("application.properties");
ClassPathResource cls2 = null;
Map<String, Object> propMap = ((ConfigurableEnvironment) ctx.getEnvironment()).getSystemProperties();
for(Map.Entry<String, Object> entrySet: propMap.entrySet()) {
System.out.println("Map.Key:"+entrySet.getKey()+" Map.valiue:"+entrySet.getValue());
}
List<String> profiles= Arrays.asList(ctx.getEnvironment().getActiveProfiles());
if(profiles == null || profiles.isEmpty()) {
if(!propMap.containsKey("spring.profiles.active")) {
cls2 = new ClassPathResource("oauth-default.properties");
} else {
cls2 = new ClassPathResource("oauth-"+propMap.get("spring.profiles.active")+".properties");
}
}else {
for(String profile:profiles) {
if(profile.equalsIgnoreCase("DEV")) {
cls2 = new ClassPathResource("oauth-DEV.properties");
}else if(profile.equalsIgnoreCase("QA")) {
cls2 = new ClassPathResource("oauth-QA.properties");
}else if (profile.equalsIgnoreCase("UAT")) {
cls2 = new ClassPathResource("oauth-UAT.properties");
}else if(profile.equalsIgnoreCase("PROD")){
cls2 = new ClassPathResource("oauth-PROD.properties");
}else {
cls2 = new ClassPathResource("oauth-default.properties");
}
}
}
cfg.setLocations(cls1,cls2);
//cfg.setPlaceholderPrefix("#{");
return cfg;
}
Then create another bean that reads the properties based on the prefix - "security.oauth2.client"
#Configuration
#ConfigurationProperties(prefix="security.oauth2.client")
public class OauthSecurityConfigurationDto {
private String clientId;
private String clientSecret;
private String scope;
private String accessTokenUri;
private String userAuthorizationUri;
private String grantType;
private String resourceIds;
private String registeredRedirectUri;
private String preEstablishedRedirectUri;
private String useCurrentUri;
private String userInfoUri;
public String getClientId() {
return clientId;
}
public void setClientId(String clientId) {
this.clientId = clientId;
}
public String getClientSecret() {
return clientSecret;
}
public void setClientSecret(String clientSecret) {
this.clientSecret = clientSecret;
}
public String getScope() {
return scope;
}
public void setScope(String scope) {
this.scope = scope;
}
public String getAccessTokenUri() {
return accessTokenUri;
}
public void setAccessTokenUri(String accessTokenUri) {
this.accessTokenUri = accessTokenUri;
}
public String getUserAuthorizationUri() {
return userAuthorizationUri;
}
public void setUserAuthorizationUri(String userAuthorizationUri) {
this.userAuthorizationUri = userAuthorizationUri;
}
public String getGrantType() {
return grantType;
}
public void setGrantType(String grantType) {
this.grantType = grantType;
}
public String getResourceIds() {
return resourceIds;
}
public void setResourceIds(String resourceIds) {
this.resourceIds = resourceIds;
}
public String getRegisteredRedirectUri() {
return registeredRedirectUri;
}
public void setRegisteredRedirectUri(String registeredRedirectUri) {
this.registeredRedirectUri = registeredRedirectUri;
}
public String getPreEstablishedRedirectUri() {
return preEstablishedRedirectUri;
}
public void setPreEstablishedRedirectUri(String preEstablishedRedirectUri) {
this.preEstablishedRedirectUri = preEstablishedRedirectUri;
}
public String getUseCurrentUri() {
return useCurrentUri;
}
public void setUseCurrentUri(String useCurrentUri) {
this.useCurrentUri = useCurrentUri;
}
public String getUserInfoUri() {
return userInfoUri;
}
public void setUserInfoUri(String userInfoUri) {
this.userInfoUri = userInfoUri;
}
}
Remember the setters are important because the ConfigurationProperties load the values into the properties of the class only when getters and setters are defined
Now we can autowire the dependency wherever needed and use the property.
If This is your exact code that means you are reading your property from a wrong property file.
replace your property resource to this line.
#PropertySource("classpath:config/services.yml")
I have the following code in a sample Spring Boot Application
#Configuration
#PropertySource("classpath:second.properties")
public class PropertyConfig {
#Value("#{guru.username}")
String user;
#Value("#{guru.password}")
String password;
#Value("#{guru.url}")
String url;
#Bean
FakeDataSource getFakeDataSource() {
FakeDataSource fk = new FakeDataSource();
fk.setName(user);
fk.setPassword(password);
fk.setUrl(url);
return fk;
}
#Bean
PropertySourcesPlaceholderConfigurer getPropertySourcesPlaceholderConfigurer() {
PropertySourcesPlaceholderConfigurer placeholderConfigurer= new PropertySourcesPlaceholderConfigurer();
//placeholderConfigurer.setLocation(new ClassPathResource("second.properties"));
return placeholderConfigurer;
}
}
And FakeDataSource is a simple pojo with the name, passowrd, url properties.
Then my main application
#SpringBootApplication
public class SpringGuru101DependencyInjectionApplication {
public static void main(String[] args) {
ApplicationContext ctx = SpringApplication.run(SpringGuru101DependencyInjectionApplication.class, args);
// Step 2: Make Class
FakeDataSource fakeDataSource = ctx.getBean(FakeDataSource.class);
System.out.println(fakeDataSource.getName());
}
}
but the sout statement is printing null,
my second.properties file is present in my resources directory with following content
guru.username=Saurabh
guru.password=ido
guru.url=http://example.com
There are two places should be corrected:
(1) As I said in the comment of your question, you should replace the wellhead sign (#) to dollar sign ($) for reading values from your configuration file. For example: #Value("${guru.username}").
(2) You missed public static in front of the method getPropertySourcesPlaceholderConfigurer.
And this modified method should be looked like as follows:
#Bean
public static PropertySourcesPlaceholderConfigurer getPropertySourcesPlaceholderConfigurer() {
PropertySourcesPlaceholderConfigurer placeholderConfigurer= new PropertySourcesPlaceholderConfigurer();
//placeholderConfigurer.setLocation(new ClassPathResource("second.properties"));
return placeholderConfigurer;
}
If you are using Spring-boot, you can use another method for reading configuration which is Spring boot recommended. This will help you get rid of all the #Value notations, allowing spring inject properties without additional hints.
You can potentially do something like:
#ConfigurationProperties("foo")
public class FooProperties {
private boolean enabled;
private InetAddress remoteAddress;
private final Security security = new Security();
public boolean isEnabled() { ... }
public void setEnabled(boolean enabled) { ... }
public InetAddress getRemoteAddress() { ... }
public void setRemoteAddress(InetAddress remoteAddress) { ... }
public Security getSecurity() { ... }
public static class Security {
private String username;
private String password;
private List<String> roles = new ArrayList<>(Collections.singleton("USER"));
public String getUsername() { ... }
public void setUsername(String username) { ... }
public String getPassword() { ... }
public void setPassword(String password) { ... }
public List<String> getRoles() { ... }
public void setRoles(List<String> roles) { ... }
}
}
The POJO above defines the following properties:
foo.enabled, false by default
foo.remote-address, with a type that can be coerced from String
foo.security.username, with a nested "security" whose name is determined by the name of the property. In particular the return type is not used at all there and could have been SecurityProperties
foo.security.password
foo.security.roles, with a collection of String
More details: https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html
I am creating a project utilizing the new Spring configurations. I have a base class which holds a few properties:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {Basic.class, Protected.class})
public class BaseTest {
#Autowired(required = false) protected String userName;
#Autowired(required = false) protected String password;
#Autowired protected String baseURL;
#Test
public void outputData() {
System.out.println("UserName: " + userName + " Password: "
+ password + "Base URL: " + baseURL);
}
}
#ActiveProfiles("default,protected")
public abstract class ProtectedTest extends BaseTest
{
#Autowired protected String userName;
#Autowired protected String password;
}
#Configuration #Profile("default")
public class Basic {
#Bean public String baseURL() { return "http://www.baseURL.com"; }
}
#Configuration #Profile("protected")
public class Protected {
#Bean public String userName() { return "userName"; }
#Bean public String password() { return "password"; }
}
However, when I go to run my protected tests I receive a notification that the base URL is not wired in properly. Since it extends the BaseTest, and has both profiles active, why am I not receiving the baseURL bean?
It has to be #ActiveProfiles({"default","protected"}), in your case it will assume that the profile by name default, protected is active, not default and protected
One more thing is that annotation in base class BaseTest is not derived by the ProtectedTest, so you will again need to put #RunWith and #ContextConfiguration for your test to run