I have a Maven scala project with a config file located here: src/main/resources/reference.yaml. This file contains particular configurable parameters that are used throughout the entire application.
Currently, I package the jar and read the file in the following manner, which is found in the classpath.
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import java.io.{File, InputStream}
val configInput: InputStream = getClass.getResourceAsStream("/" + configFileName)
val mapper = new ObjectMapper(new YAMLFactory)
mapper.registerModule(DefaultScalaModule)
mapper.readValue(configInput, classOf[Map[String, Any]])
I have custom functions to be able to parse the yaml file and load the variables into an object that stores all my config values.
object Paths {
// Config
val table: String = getConfigValue("sql.finance_table", "finance_v1")
}
This object is imported in various other scala files that define the pipeline, and use the config parameters to follow the specification of the user.
To execute the pipeline, the jar is packaged up and deployed to a cluster on databricks where it runs as part of a job. Thus, all config values are static and provided during build time.
While this works, I would like to separate the resource file customisation and the packaging and deployment of the jar. Is it possible to simply get the jar to read a resource file from an external location? This would allow the user to modify the config even after the jar has been built.
I'm not using lightbend or pureconfig.
There are several options:
add an "external" file in the classpath when running the code. The classpath does not only consists of the JARs of your app, it can also contain regular files. This way you don't even have to modify your current code. But.. I don't know if/how to customize the classpath with Databricks.
modify your code to read from "external" file in the first place rather than the classpath. You could hardcodd the path of the file or make it configurable via a system property.
Related
I want to set the log4j configuration file path/other folder paths that can be used across other class files, without hard-coding the folder path.
Rightnow, I have set the variables as Environment variable. But It can only be modified if I launch Eclipse. How do I set this variable in such away that anyone (doesn't want to launch Eclipse) can modify it, from outside. Also, it will be used in test configurations. So it's better to not hard-code it and have all the file paths etc. easy to refactor.
final static String log4jpath = System.getenv("LOG4J_PATH");
Paraphrasing a comment:
[How to get value from] outside of the Java program such as a separate file, that contains all other filepaths?
That is called a configuration file.
It is often a properties file, similar to a Log4j configuration file, but it can be any types of file, e.g. XML, JSON, YAML, ...
To identify a single such file, you can use:
An environment variable (like you are right now)
A system property (more common)
A specifically named file in the current directory
...
The entries in that file will identify all the values you really want.
For example, Spring, which is a populate Java framework, will look for configuration values in many places. See 24. Externalized Configuration for full detail, but here is a summary of the most common ones:
Command line arguments, e.g. java -jar MyApp.jar --foo=bar
Java System properties, e.g. set using -Dfoo=bar on the command-line
OS environment variables, e.g. SET foo=bar (Windows) or export foo=bar (Linux)
Application properties outside of your packaged jar, i.e. relative to current directory.
Name and location can be overridden on command-line.
config/application.properties
config/application.yaml
application.properties
application.yaml
Soni, If you want to put the log4j configuration file in one place so that everybody can access. Follow the steps.
Create a project with some name and inside src/main/resources folder keep the log4j configuration file.
Create a jar file which must contain this log4j configuration file.
Use this created jar file wherever it is required. Log4j will automatically use the configuration for desired logging. If you want, you can distribute this jar file to anybody who wants to use it.
The above option is if you do not want to change the configuration file.
Now if there is a situation where someone wants to modify the configuration file.
In this case, simply put the configuration in any project classpath, means inside resource folder. As long as log4j jar files are there in the classpath and configuration files. It will log everything.
However, if you want, you can extend the functionality of Log4j by passing configuration as an object. You can refer below the link to access pro grammatically.
https://howtodoinjava.com/log4j/how-to-programmatically-configure-appenders-in-log4j/
I have added all file and folder paths inside the properties file (example config.properties) and then used it inside the testsetup method by InputStream input = new FileInputStream("Path to//config.properties");
Properties prop = new Properties();
prop.load(input);
System.setProperty("log4j2.configurationFile", prop.getProperty("log4j.path"));
this way, all files/folder paths can be modifies from outside and there's no need to set environment variable from inside the project.
I have a Spring Java 1.8 project exported in a .jar. When I want to run this Java .jar application, I'd like to give a file into the argument which .xml config file should the program use in the relative folder. I tried so many ways, but none of them works.
For example:
sampleJavaPorgram.jar config1.xml
(config1.xml and sampleJavaProgram.jar is in the same directory)
shoud load config1.xml like that:
ApplicationContext context = new ClassPathXmlApplicationContext("config1.xml");
Maybe use Command-Line Argument, to get name of this XML file and then pass it to AppContext.
Or you might use System properties
I have a need to load an external file into the classpath using Play Framework 2.3 (Java)
Stipulations:
The external file cannot live inside my Play app (i.e. the /conf and /lib directories, etc. are not an option)
The file needs to be a .properties or a .conf file so I can specify property values
Here's my scenario:
I have a custom JAR that has some code which is looking for a specific file (let's call it myproperties.properties) in the classpath when being used. The way that I'm attempting to find myproperties.properties is by doing this inside a class that resides inside that custom JAR:
ClassLoader classLoader = com.my.package.MyCustomJavaClass.class.getClassLoader();
InputStream inputStream = classLoader.getResourceAsStream("/path/to/myproperties.properties");
I have access to change the properties file name and the path to it inside the JAR.
My Play Framework App (using Java) has this custom JAR in it's /lib folder, so it gets automatically added to the classpath (this is tested and works correctly). My Play App calls MyCustomJavaClass when it first loads the / route (index route), so the class loader and input stream code above gets kicked off when I hit my play app in the browser.
Problem:
I have not been successful in my attempts to load /path/to/myproperties.properties into the classpath when starting the Play App in a way that my code in the custom JAR can see it.
I've been attempting to start play with the classpath command like so in an attempt to feed the JVM the external file:
activator start -J-classpath "-J-classpath:/path/to/myproperties.properties"
I'm adding -J-classpath; to the beginning of the path in an attempt to copy everything that's currently in the classpath and then just adding my single, external file. However, doing this doesn't seem to be working (i.e. my inputStream is null).
Questions:
Am I doing the activator start -J-classpath command correctly when starting the play app? Other variations in an attempt to copy the existing classpath first were not allowing the play app to start, but this command at least starts my app.
Reference (Specifying additional JVM arguments): https://www.playframework.com/documentation/2.3.x/ProductionConfiguration
What are some other ways that I could possibly get this done? I've explored overriding the application.conf file using activator start -Dconfig.file=/path/to/application-override.conf and putting my properties inside the new application-override.conf file. However, it doesn't seem to put that file into the classpath for MyCustomJavaClass to find using the Class Loader. Maybe I'm doing this command incorrectly as well?
Is it possible that somehow the Play Framework classpath is separate from the classpath that my custom JAR is seeing? I've been under the assumption that it's all in one JVM and classpath.
here's the solution I came up with, hopefully it helps someone else out there:
in my "upper environments" (AWS servers) where my play app is deployed, I put an application-override.conf file in the conf folder in the play framework app directory
the application-override.conf is the exact same as my application.conf but I have some custom properties in both whose values are different in each environment that the play app lives on
my play framework app is in a git repo, which is cloned on each upper environments, so I added application-override.conf to the .gitignore (I don't want it checked it to the repo so it only lives on the servers)
when starting the play app, I now use activator start "-Dconfig.trace=loads -Dconfig.file=conf/application-override.conf". this will override the application.conf file with application-override.conf and application-override.conf will be in the JVM classpath that play uses to run the app (since it's in the conf directory). -Dconfig.trace=loads spits out more logging to let you know if the .conf file was loaded properly or not; it's not a necessary flag if everything is working properly.
on the java side, in my custom JAR, I can now do the following:
Properties properties;
InputStream stream;
ClassLoader classLoader = com.my.package.MyCustomJavaClass.class.getClassLoader();
// first, look for application-override.conf in the classpath (upper environments)
stream = classLoader.getResourceAsStream("application-override.conf");
// if null, check for application.conf (local environment)
if (stream == null) {
stream = classLoader.getResourceAsStream("application.conf");
}
properties = new Properties();
properties.load(stream);
stream.close();
other notes:
I thought about doing a symlink/softlink in the conf directory and put the application-override.conf file somewhere else on my environment, but prior to Play 2.4, you can't have symlinks in the conf directory, so I just put the actual application-override.conf file in the conf folder
The application-override.conf file has different property values for each "upper environment", otherwise I would have just delivered a single override file to the git repo. And in the custom JAR, I didn't want to put in logic that looked for varying file names like dev-override.conf, pre-prod-override.conf, prod-override.conf, etc. I wanted a single upper environments override file.
I didn't have success with the -classpath=/path/to/myproperties.properties or -J-classpath=/path/to/myproperties.properties commands in conjuction with activator start. nor did I have success with attempting to append to the classpath, e.g. activator start -J-classpath=-J-classpath:/path/to/myproperties.properties or other similar combinations
going the route of putting properties in an application-override.conf file actually killed two birds with one stone for me because I've been wanting to make some environment specific changes by having overriding .conf files on each of my environments as well as custom properties
the HOCON format of the .conf files required me to put double quotes around my property values due to the nature of my property values. when reading in those properties in Java, the quotes were still there for me, so I had to do an str.replace("\"","") when reading in those properties
Several maven plugins need/support passing a java.io.File as a configuration parameter, wherein we specify the relative/absolute location of the file for the plugin to locate and use.
Is there a way I can specify a property file in the plugin configuration where the file has to be found from inside a jar in the classpath? I'm particularly wanting this to know and use with the aspectj-maven-plugin, where I can specify the Xlintfile value to be the custom XlinkDefault.properties file location. The file, in my case, will be found inside a classpath jar.
I use maven-2.2.1 by the way.
No, not in general; there's no magic that will turn something that's not a file on disk into a java.io.File. Many Maven plugins (e.g., maven-checkstyle-plugin's configLocation are designed to allow more flexible input for just these cases:
This parameter is resolved as resource, URL, then file. If successfully resolved, the contents of the configuration is copied into the ${project.build.directory}/checkstyle-configuration.xml file before being passed to Checkstyle as a configuration.
As a workaround, if the plugin cannot be changed, dependency:unpack may be a way to get a classpath resource into a local file (see Maven: extract files from jar).
I have a Spring 3.0 MVC project that uses a JAR file from a different project as a dependency. This JAR file has an "auth.properties" file in it's resource and has a string in it like this.
Ex: packages.redirectUrls.gotoUrl = 'http://myUrl.com';
Now, I am referring to this string in JAR File in my controller using:
#Value("${packages.redirectUrls.gotoUrl}")
I also have a local "auth.properties" file that consists of the same string with a different value in it.
Ex: packages.redirectUrls.gotoUrl = 'http://newUrl.com';
However, my Java code is not able to read this new configuration and always loads from the JAR File. Is there a way to override the JAR file setting with this new setting?
Thanks,