I implemented Java Spark application, which I'm running on EMR cluster with spark-submit command.
I want to pass app.properties which I use in my application.
app.properties looks as follows:
local_fetcher = false
local_storage = false
local_db = true
.
.
.
I want to be able to get this data in my application.
My questions are:
Where app.properties should be located?
How can I read it content in my Spark application?
Should I be able to read it from driver & executers?
I tried to use --properties-file flag but I understood it will override the default Spark configuration which is not what I want.
I saw that I might use --file flag, but didn't understand where should the file be located and how I can read it inside my application.
First option: --files
--files FILES Comma-separated list of files to be placed in the working directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).
spark-submit --files /path/to/app.properties /path/to/your/fat/jar.jar
You can get the exact location of the uploaded file using the SparkFiles.
Second option: getResourceAsStream
Put your app.properties inside your job's JAR file, and load it like this:
val appPropertiesStream = scala.io.Source.fromInputStream(
classOf[yourObject].getClassLoader.getResourceAsStream("/app.properties")
val appPropertiesString = scala.io.Source.fromInputStream(appPropertiesStream ).mkString
(note the forward slash before the "app.properties", as far as I remember it's important)
Related
My Springboot Application is running fine in IDE but when I create fat jar file and run on docker it gives the error. I am connecting my application with firebase so i want to include the serviceAccountKey.json file from the resource folder. The application runs fine in my ide, but while deploying it over the docker container it gives the error of file not found. Though when include the file and print it path it doesn't give any sort of error . But when i give the file path to fileInputStream it produces the error. I have tried multiple ways but nothing seems to work. I'm including the file using classLoader.getResource("filename.json").
I tried to skip the inclusion of file and do it by saving files content in a string and then sending it to stream but this method crashes the server whenever i query firebase.
this is the code where error is occurring. Notice that I'm printing the file path and it gets printed in the output before showing error. I have also tried file.getAbsoluteFile instead of path but doesn't work. Probably I'm doing it wrong or probably i have to mention the path in some other place as well which i don't know about. If anyone has done this before then please help me on this.
File path is getting printed but FileStream can't get it
An alternative to having a credential in a json file that is packaged with the application, would be that you set an environment variable upon starting the application, and instead load the key via application.yml. This way, you don't need to package the secrets into the application jar file.
1. create config class
#Configuration
#ConfigurationProperties(value = "my-custom-config")
public class MyConfig {
private String serviceAccountKey;
// getters/setters etc. if not using lombok
}
2. use above config
in other spring beans by injecting the config class, and retrieve the secret with getter, e.g. config.getServiceAccountKey();
3. Add config to your application.yml file
# application.yml
my-custom-config:
serviceAccountKey: ${ENV_VARIABLE_NAME} # <---- this way you can bind an env variable to your config.
4. Define env variable in container on startup
On instantiating the docker container, provide env option and define an environment variable.
docker run -env ENV_VARIABLE_NAME=<value> ...
I have a micronaut project where I want to have an unnversioned configuration file for private data (like database connections and so on)
This information have to be loaded through #Property annotation, but since there will be more than one .yml (there will also be at least an application.yml) y want to be able to provide file's path to #Properties to be able to differentiate where to look for property.
Since it's my first micronaut project I'm a bit lost with this stuff but taking springboot as an example, what I want to do is something like:
#PropertySource("classpath:configprops.properties")
But after reading micronaut documentation(https://docs.micronaut.io/latest/guide/index.html#configurationProperties) I found myself unable to do this (except from something like just reading the plain file which I guess would not be micronaut compliant)
I do it by passing jvm arguments.
For example, If I am running it on my local machine using gradle:run, I add following to build.grade
run.jvmArgs('-Dmicronaut.environments=dev', "-Dmicronaut.config.files=${System.getProperty("user.home")}/auth-config.groovy")
For my jar deployment, I have made a deploy.sh file as follows :
#!/bin/bash
fuser -k 8181/tcp
nohup java -Xmx512m -Dmicronaut.environments=staging -Dmicronaut.config.files=<path-to-config>/config.groovy -jar application-0.1-all.jar > application.log 2>&1 &
Also note that I am passing different environment names, this helps you to include development environment config directly in code if you want.
Like
application-[environment_name].groovy
application-[environment_name].yml
application-[environment_name].properties
This will help new contributors on your project to speedup the process project setup, I generally also include note in my application-dev.groovy file
DEVELOPER NOTE:
***** DO NOT COMMIT ANY CHANGE IN THIS FILE IF YOU MAKE ANY
*******************************************************
***** CREATE <config.groovy> file in your <HOME> folder and copy paste content of this file
***** Override properties as required
*******************************************************
I have a jar that is reading a file using below code:
Thread.currentThread().getContextClassLoader().getResource(fileName);
I want to run this jar using java -jar .jar command but I want to keep this file outside my jar, so that I can edit the jar file later on without touching the jar. Can anyone help me, how to run this jar so that it will pick up the file from outside.
There are multiple approaches you can do that and it will depend on where would you like to place this external file. For the sake of this answer, I will refer to this file as config file
Not In The Same Directory
The first approach is where you will need to place this file outside the JAR and not necessarily next to the JAR file in the same directory. In that case, you can pass the file location of the config file using either an environment variable (if you are running the JAR in a shell for example) or a Java property.
To use an environment variable, assuming you are using some Linux distro, then you can use the export command to set the value; something like this:
$ export CONFIG_FILE_LOC=/etc/myapp/config.file
You can then read the value in your code using the System class by using the following code:
String fileLocationEnv = System.getenv("CONFIG_FILE_LOC");
Alternatively, you can set this as a property by adding the following segment to your launch command:
$ java -Dconfig.file.location=/etc/myapp/config.file -jar myapp.jar
You can then read the value in your code using the System class for properties using the following code:
String fileLocationProp = System.getProperty("config.file.location");
In The Same Directory
If you need the config file to co-exist in the same directory as your JAR file, then you can use the following code to get the JAR directory and then append the filename to it. Here's the code (assuming a class named MyApp)
try{
new File(MyApp.class.getProtectionDomain().getCodeSource().getLocation().toURI());
} catch(URISyntaxException exception){
System.out.println("Exception");
}
Hope that helps.
To open the file as a resoure, add the folder containing the file(s) you want to use, to your classpath:
java -classpath .;config -jar myjar.jar
This example adds the current directory and the config directory to your classpath.
Multiple folders can be specified by using a separator. On windows use ';' , on unix use ':' .
To open the file as a File, you can just use
new File("configfile")
which will look in the working directory (directory where you launched your java)
I am trying to implement Configuration Management through Zookeeper. I have created few ZNodes from command line as:
create /config ""
create /config/mypocapp ""
create /config/mypocapp/name "John Doe"
Here, name is one of the properties that I want to access in my app called mypocapp.
Since we will be having a lot of properties in our application, we just can't use command line to create each and every property like this.
Is there a way we can create the properties in zookeeper through some UI or directly in a file (and import it to zookeeper).
I am completely new to zookeeper and not getting any help in this direction. Please help.
Exhibitor is one of the options that one can try to insert, modify or delete properties in ZNodes.
One can follow the steps given below:
Download the pom file of Exhibitor UI from the Github.
Built the pom file using maven that will generate a jar file.
Run the jar file as: java -jar <jar-file-name>.jar -c file
Go to your browser and type in: localhost:8080 to access Exhibitor UI.
Here, you can configure your Zookeeper ensemble and can edit the properties.
Please note that each instance of Zookeeper will have corresponding Exhibitor UI.
In order to run exhibitor on a different port, you can run:
java -jar <jar-file-name>.jar -c file --port <port-of-your-choice>
There are now also VS Code extensions that allow viewing and editing the Zookeeper node hierarchy and data, like this one:
https://marketplace.visualstudio.com/items?itemName=gaoliang.visual-zookeeper
When using the script ./chpasswd.sh to change the admin password, it reads all the jar files to setup a classpath. But it does NOT read the $CARBON_HOME/repository/components/lib/*.jar files. In this directory you put the ojdbc6.jar file when you use a database such as Oracle.
When I copy the ojdbc6.jar file into $CARBON_HOME/repository/lib directory, the script works.
My command is:
./chpasswd.sh --db-url "jdbc:oracle:thin:#localhost:1521:orcl" --db-username wso2carbon -db-password wso2carbon --username admin --new-password nimda
Could this be a bug in the script?
Yes.. I guess, according to the script, you can see that it adds following directories as the class path.. It does not contains component/lib. Also once ant script is run, It may not copy the jars in component/lib in to repository/lib.
for f in "$CARBON_HOME"/lib/*.jar
do
CARBON_CLASSPATH=$CARBON_CLASSPATH:$f
done
for g in "$CARBON_HOME"/repository/lib/*.jar
do
CARBON_CLASSPATH=$CARBON_CLASSPATH:$g
done
for h in "$CARBON_HOME"/lib/api/*.jar
do
CARBON_CLASSPATH=$CARBON_CLASSPATH:$h
done