Using Spring Boot 1.3.0.RELEASE
I have a couple of yaml files that describe several instances of a program. I now want to parse all those files into a List<Program> (Map, whatever), so I can later on search for the most appropriate instance for a given criteria in all the programs.
I like the approach with #ConfigurationProperties a lot, and it works good enough for a single yaml-file, but I haven't found a way yet to read all files in a directory using that method.
Current approach working for a single file:
programs/program1.yml
name: Program 1
minDays: 4
maxDays: 6
can be read by
#Configuration
#ConfigurationProperties(locations = "classpath:programs/program1.yml", ignoreUnknownFields = false)
public class ProgramProperties {
private Program test; //Program is a POJO with all the fields in the yml.
//getters+setters
I tried changing the locations to an Array listing all of my files locations = {"classpath:programs/program1.yml", "classpath:programs/program2.yml"} as well as using locations = "classpath:programs/*.yml", but that still only loads the first file (array-approach) or nothing at all (wildcard-approach).
So, my question is, what is the best way in Spring Boot to load a bunch of yaml files in a classpath-directory and parse them into a (List of) POJO, so they can be autowired in a Controller? Do I need to use Snakeyaml directly, or is there an integrated mechanism that I just haven't found yet?
EDIT:
A working approach is doing it manually:
private static final Yaml yaml = new Yaml(new Constructor(Program.class));
private static final ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
try {
for (Resource resource : resolver.getResources("/programs/*.yml")) {
Object data = yaml.load(resource.getInputStream());
programList.add((Program) data);
}
}
catch (IOException ioe) {
logger.error("failed to load resource", ioe);
}
In Spring, it is possible to load multiple configuration properties files using PropertySource annotation, but not YAML files. See section 26.6.4 in link below:
https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-typesafe-configuration-properties
However, from your problem, it seems that you can configure all your programs in single YAML and then get all list of programs in a single list.
Sample YAML (all.yaml)
programs:
- name: A
min: 1
max: 2
- name: B
min: 3
max: 4
Config.java
#Configuration
#ConfigurationProperties(locations={"classpath:all.yaml"})
public class Config{
private List<Program> programs;
public void setPrograms(List<Program> programs) {
this.programs = programs;
}
public List<Program> getPrograms() {
return programs;
}
}
What I am currently doing, as far as I understood your question, is nearly the same.
I am having an application.yml and also profile-specific yml files, e.g. application-{profile}.yml in my src/main/resources.
In the application.yml I have defined the default profile key-values, which are partially overridden by the profile-specific yml files.
If you want to have a type-safe and well defined access of your YML key/values, then you can use the following approach:
#ConfigurationProperties
public class AppSettings {
String name; // has to be the same as the key in your yml file
// setters/getters
}
In your Spring-Boot config, you have to add the following annotations onto your config class:
#ComponentScan
#EnableAutoConfiguration
#EnableConfigurationProperties( value = { AppSettings.class, SomeOtherSettings.class } )
public class SpringContextConfig {
#Autowired
private AppSettings appSettings;
public void test() {
System.out.println(appSettings.getName());
}
}
The #Autowiring is also accessible from other Beans.
The other way around (without an extra separated and type-safe class, is to access the YML-values via #Value("${name}").
To bring it together in a short manner:
Yes, it is possible to use several YAML files for your application via Spring-profiles. You define your current active spring profile via command args, programmatically or via your system env (SPRING_PROFILES_ACTIVE=name1,name2).
Therefore you can have several application.yml files for each profile (see above).
Related
NOTE: NO SpringBoot. ONLY Spring Framework.
I see this works with #ConfigurationProperties in a springboot project. But I can't get it to work with #Value in a spring framework only project. It seems the #Value annotation cannot resolve the list correctly.
Here is a sample project: https://github.com/KiranMohan/spring-yaml.
For loading the yaml file, I used the YamlPropertiesFactoryBean.
The code is tested in a junit ExampleTest class.
Output of
log.debug("myList: {}",example.getMyList());
is
DEBUG [main] org.ktest.ExampleTest: myList: [${example.myList}]
Example yaml:
example:
enabled: true
name: "org.ktest"
myArray: >
abc,
def
myList:
- "ghi"
- "jkl"
Example code:
#Configuration
#PropertySource(value = "classpath:example.yml", factory = YamlPropertySourceFactory.class)
public class Example {
#Value("${example.enabled}")
private boolean enabled;
#Value("${example.name}")
private String name;
#Value("${example.myArray}")
private String[] myArray;
#Value("${example.myList}")
private List<String> myList;
Your YamlPropertySourceFactory uses YamlPropertiesFactoryBean whose Javadoc states
Lists are split as property keys with [] dereferencers, for example
this YAML:
servers:
- dev.bar.com
- foo.bar.com.
becomes properties like this:
servers[0]=dev.bar.com
servers[1]=foo.bar.com
This is also mentioned in the Spring documentation here.
In other words, using these types, you can't feed that example.myList YAML element into a List through property resolution. You can only access individual elements like
#Value("${example.myList[0]}")
private String myListFirstElement;
The Spring documentation here also mentions this. It suggests
Properties that use the [index] notation can be bound to Java List or
Set objects using Spring Boot’s Binder class.
You can try looking at how that class does it and reproduce it in your own code.
I want to group configuration items in Quarkus and use them as a Map. Something along the lines of the next application.properties:
greeting.names = tom,jane
greeting.tom.message = hello
greeting.tom.name = tom
greeting.jane.message = hi
greeting.jane.name = jane
And have them in the application in a Map, like this:
#ConfigProperties
public class GreetingConfig {
private String name;
private String message;
// getters / setters
}
And use them like this:
private Map<String, GreetingConfig> config;
They are now name indexed, but a List would also be fine (and is what I actually need):
greeting.1.message = hello
greeting.1.name = tom
greeting.2.message = hi
greeting.2.name = jane
Any ideas on how to realize this? I have a programmatic solution, but would prefer a solution by annotation only
I've had a similar problem these days.
So I wrote a simple Quarkus extension which helped me with the configuration.
You can use the guides from the Quarkus site: Quarkus - Writing Your Own Extension and Quarkus - Building my first extension,
but basically these are the steps taken in order to create the configuration:
Have some maven multi module project (not the project where the configuration will be consumed).
Execute a similar command form the project directory:
mvn io.quarkus:quarkus-maven-plugin:1.4.2.Final:create-extension -N \
-Dquarkus.artifactIdBase=keycloak-extension \
-Dquarkus.artifactIdPrefix=otaibe-commons-quarkus- \
-Dquarkus.nameBase="Keycloak extension"
This will create 'sub multi module' project there with the following modules: runtume and deployment.
Go to the runtime module and add and annotate your config class. It should be something similar to this class
In runtime module create a producer class which should register the configuration class as CDI bean:
#ApplicationScoped
public class OtaibeKeycloakQuarkusProducer {
private volatile OtaibeKeycloakConfig otaibeKeycloakConfig;
void initialize(OtaibeKeycloakConfig config) {
this.otaibeKeycloakConfig = config;
}
#Singleton
#Produces
public OtaibeKeycloakConfig otaibeKeycloakConfig() {
return otaibeKeycloakConfig;
}
}
Create a Recorder class which should initialize the Producer:
#Recorder
public class OtaibeKeycloakRecorder {
public void initOtaQuarkusProducer(BeanContainer container, OtaibeKeycloakConfig configuration) {
OtaibeKeycloakQuarkusProducer producer = container.instance(OtaibeKeycloakQuarkusProducer.class);
producer.initialize(configuration);
}
}
In deployment module you have a Processor class. Go there and register your Producer to be injectable as CDI bean and your Recorder to initialize it. Add the similar code:
#BuildStep
AdditionalBeanBuildItem beans() {
return AdditionalBeanBuildItem.builder().addBeanClasses(OtaibeKeycloakQuarkusProducer.class).build();
}
#BuildStep
#Record(ExecutionTime.RUNTIME_INIT)
void otaibeKeycloackConfigBuildItem(BeanContainerBuildItem beanContainer,
OtaibeKeycloakConfig otaibeKeycloakConfig,
OtaibeKeycloakRecorder recorder) {
recorder.initOtaQuarkusProducer(beanContainer.getValue(), otaibeKeycloakConfig);
}
You can find my implementation here.
Now, go to your initial project where the configuration will be consumed and add the runtime module as a dependency.
In order to ensure that the configuration is added properly execute the following maven command:
mvn quarkus:generate-config
Now, you can check the file src/main/resources/application.properties.example and verify whether your properties are added there. The property group should start with quarkus. plus the name of your #ConfigRoot annotation. In my case for example it will start with quarkus.otaibe.keycloak
That's it!
Now in Quarkus 2.9, you do it with #ConfigMapping (#ConfigProperties is deprecated).
#StaticInitSafe
#ConfigMapping
public interface GreetingConfigMapping {
Map<Integer, GreetingDetail> greeting();
}
interface GreetingDetail {
String name();
String message();
}
Try it in a #QuarkusTest; #Inject it.
Problem:
I have 3 parts in the software:
Client A service
Client B service
Target C service
I want to connect to C from A and B
I wrote a library with following setup:
/src/main/java/pkg.../TargetConnector.java
/src/main/java/pkg.../TargetConfig.java
/src/main/resources/application-dev.properties
/src/main/resources/application-tst.properties
/src/main/resources/application-prd.properties
My clients A and B both have there own sources and properties:
/src/main/java/pkg.../Client{A/B}Service.java
/src/main/java/pkg.../Client{A/B}Config.java
/src/main/resources/application-dev.properties
/src/main/resources/application-tst.properties
/src/main/resources/application-prd.properties
The properties of the Connector contains some login info for the service e.g.
target.url=https://....
target.usr=blablabla
target.key=mySHAkey
which is used in the TargetConfig to preconfigure the Connector e.g.
#Value("target.url")
String url;
#Value("target.usr")
String usr;
#Value("target.key")
String key;
#Bean
public TargetConnector connector() {
return new TargetConnector(url, usr, key);
}
Now when I use the connector jar in the client I can find the configuration via packagescan. The connector class is loaded but the problem is that it does not load the properties files.
Research
I found that multiple property files cannot have the same name (e.g. clients application-{profile}.properties clashes with the one from the connector), so I tried to rename application-{profile}.properties of the targetConnector to application-connector-{profile}.properties.
The properties whoever still do not get loaded, (which makes sense since I do not have a e.g connector-dev profile but my profile is simply named dev).
Furthermore, even if I try to explicitly load one of the property files from the connector with:
#PropertySource({"classpath*:application-connector-dev.properties"})
it cannot be found
Question
My question is actually 3 tiered:
How can I load a property file in a dependency jar at all?
How can I load the profiled version of the property file if the the properties file has a different name than application.properties? e.g. application-connector.properties
How can i combine the answers from question 1 and 2 to load the profiled version of the property in the jar?
If further explanation is needed, please ask.
Answer
I went for an approach as given in the accepted answer.
I Just created 3 configs for the dev, tst, prd profiles containing the values needed and annotated the config files with the correct profiles.
You are using #Configuration annotated class. Maybe you can have one per profile. Here you are an example:
#Configuration
#Profile("profileA")
#PropertySource({"classpath:application-profileA.properties"})
public class ConfigurationProfileA{
#Value("${target.url}")
String url;
#Value("${target.usr}")
String usr;
#Value("${target.key}")
String key;
#Bean
public TargetConnector connector() {
return new TargetConnector(url, usr, key);
}
}
Do the same for profile B (maybe you can structure this better but the key points here are the annotation #Profile("") and #PropertySource(""))
Once you have your config class, Spring will use the Configuration class you want by just filling -spring.profiles.active=profileA (or the name of the profile you have written in the #Profile("") annotation)
I think there is a typo in this line #PropertySource({"classpath*:application-connector-dev.properties"})
Please check by removing the asterik.
In order to run with a specific profile, you can run with option -spring.profiles.active=dev for example
If you don’t run with a profile, it will load the default profile in application.properties that you don’t seem to have.
Furthermore, an advice would be to always have an application.properties and put in it the common properties and the default values that you would override in other properties files.
Other mistake is how you assign properties with #Value annotation, you need to use #Value("${PROPERTY_FROM_PROPERTIES_FILE}")
I want to use a yml configuration file in my project. I am using jackson-dataformat-yaml for parsing yml files. But I need to parse yml comments as well. I used the similar approach in python using ruamel yaml. How can I do the same in java?
Upd.
What for? Well, I wanted to make it possible to override my configuration options by using command line arguments. So, to generate description message for each option, I wanted to use my comments. Like this:
In my config.yml
# Define a source directory
src: '/foo/bar'
# Define a destination directory
dst: '/foo/baz'
So when you run your program with the --help flag, you'll see the following output:
Your program can be ran with the following options:
--src Define a source directory
--dst Define a destination directory
The main benefit in such a model is that you don't ever need to repeat the same statement twice, because they can be retrieved from the configuration file.
Basically, you have three layers of data:
Your configuration schema. This defines the values that are to be defined in the configuration file.
The configuration file itself, which describes the usual configuration on the current machine.
One-time switches, which override the usual configuration.
The descriptions of what each value does belong to the schema, not to the configuration file itself. Think about it: If someone edits the configuration file on their machine and changes the comments, your help output would suddenly show different descriptions.
My suggestion would be to add the descriptions to the schema. The schema is the Java class you load your YAML into. I am not sure why you are using Jackson, since it uses SnakeYaml as parser and SnakeYaml is perfectly able to deserialize into Java classes, but has more configuration options since it does not generalize over JSON and YAML like Jackson does.
Here's a general idea how to do it with SnakeYaml (beware, untested):
// ConfigParam.java
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.FIELD)
public #interface ConfigParam { String description(); }
// Configuration.java
public class Configuration {
#ConfigParam("Define a source directory")
String src;
#ConfigParam("Define a destination directory")
String dst;
}
// loading code
Yaml yaml = new Yaml(new Constructor(Configuration.class));
Configuration config = yaml.loadAs(input, Configuration.class);
// help generation code
System.out.println("Your program can be ran with the following options:")
for (Field field: Configuration.class.getFields()) {
ConfigParam ann = field.getAnnotation(ConfigParam.class);
if (ann != null) {
System.out.println(String.format("--%s %s", field.getName(), ann.description());
}
}
For mapping actual parameters to the configuration, you can also loop over class fields and map the parameters to the field names after having loaded the configuration (to replace the standard values with the given ones).
I have a properties file called xyz.properties. Right now, I am able to load individual property of this file in my class annotated with #Component which is working perfectly fine.
However, I am thinking of making my project more modular, for which I need to read the whole xyz.properties file as one properties object so that i can pass it along. How can I do so ?
Update
Right now, I am loading individual property from the file like this
my applicationContext.xml has following entry
<context:property-placeholder location="classpath:xyz.properties" order="2" ignore-unresolvable="true"/>
and then i have the respective class as
#Component
public class XyzConfiguration {
#Value("${client.id}")
private String clientId;
#Value("${client.secret}")
private String clientSecret;
...
}
What I mean by pass it along
Right now, for each individual property I have to create a respective field in a class and then annotate it with respective property name. By doing so, I am making my nested module very spring framework specific. I might someday put this module on github for others as well and they may or may not use spring framework. For them it would be easier to create an object of this module by passing required parameters (ideally in case of a Properties object) so that my module will fetch the properties by itself.
You can try below code.
import org.springframework.core.io.support.PropertiesLoaderUtils;
import org.springframework.core.io.ResourceLoader;
import java.util.Properties;
private String fileLocator;
private Properties prop;
private ResourceLoader resourceLoader;
public void init() throws IOException {
//"fileLocator" must be set as a path of file.
final Resource resource = resourceLoader.getResource(fileLocator);
prop = PropertiesLoaderUtils.loadProperties(resource);
}
prop will have all values from your property file and then you can get any value by calling prop.getProperty() method.