I have a app.properties file in my maven project under resources folder as shown here (simplified):
myApp
|----src
| |
| |--main
| |--java
| | |--ApplicationInitializer.java
| |
| |--resources
| |--app.properties
|
|---target
|--myApp.jar
|--app.properties
In ApplicationInitializer class I want to load properties from the app.properties file with following piece of code:
Properties props = new Properties();
String path = "/app.properties";
try {
props.load(ApplicationInitializer.class.getResourceAsStream(path));
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(props.getProperty("property"));
This piece of code loads properties correctly when I run it from inside my IDE but fails with exception
Exception in thread "main" java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:434)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at cz.muni.fi.fits.ApplicationInitializer.main(ApplicationInitializer.java:18)
when trying to run in as JAR file.
For creating a jar file I am using combination of maven-shade-plugin, maven-jar-plugin (for excluding properties file outside of the JAR) and maven-resources-plugin (for copying properties file to specific folder) in pom.xml file as shown here:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>cz.muni.fi.fits.ApplicationInitializer</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.5</version>
<configuration>
<excludes>
<exclude>**/*.properties</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>2.7</version>
<executions>
<execution>
<id>copy-resource</id>
<phase>package</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target</outputDirectory>
<resources>
<resource>
<directory>src/main/resources</directory>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
I then switched the code in main method to this one:
Properties props = new Properties();
String path = "./app.properties";
try (FileInputStream file = new FileInputStream(path)) {
props.load(file);
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(props.getProperty("property"));
and managed to load properties from file when running JAR but this time I was not able to load them when running in IDE, it ended with the same exception as above.
So my question is: How to set the filepath (or pom.xml file?) that I will be able to load properties running from both IDE and JAR file?
Thanks in advance :)
In your Java code read your app.properties file like this:
final String ROOT_PATH = "custom.path"; // Or whatever you most like
final String PROPERTIES_FILE = "app.properties";
// start :: try-catch here to manage I/O resources
File directory = new File(System.getProperty(ROOT_PATH), "conf");
File file = new File(directory, PROPERTIES_FILE);
BufferedInputStream bufferedInputStream = new BufferedInputStream(new FileInputStream(file));
// ...
// end :: try-catch here to manage I/O resources
Then you have to ways for setting/declaring custom.path:
Declaring it as an environment variable
Setting it at run-time (environment properties) with the -Dcustom.path=/somewhere/in/your/filesystem (I prefer this one, unless the path is fixed and/or shared across different applications)
Then, eventually you need to copy/put your app.properties file inside /somewhere/in/your/filesystem/conf. And why inside /conf? Because you use it when you declare the directory field. If you don't want it in there just don't set it and delete the , "conf" part.
Additionally, for running it "locally" (in your IDE) use the VM options setting (IntelliJ IDEA):
The way you're working at the moment you depend on the working directory of your java process. Typically that's the your command line (aka shell) points to, when starting the application.
On most IDE's you can configure this directory in your launcher settings. (For eclipse it's on the second tab 'Arguments') So you will need to the target directory (For eclipse there's a button 'Workspace...')
Specifically in intelliJ you can find this setting under Run/Edit Configurations... This will open a window like this:
There you can edit the working directory. You simply add target at the end.
Edit:
Actually you've add src/main/ressources at the end.
Related
I have a program that scrapes a webpage. I'm using JSoup and Selenium.
To configure the user agent in the JSoup request, I have a userAgents.txt file containing a list of user agents. In each execution, I have a method that reads the .txt file, and returns a random user agent.
The program is working as expected when running in IntelliJ.
The problem happens when I try to build the .jar file, with mvn clean package. When running the .jar file, I get a FileNotFoundException, since the program can't find the userAgents.txt file.
If I remove this functionality, and hardcode the user agent, I have no problems.
The file currently is in src/main/resources. When executing the .jar, I get the exception:
java.io.FileNotFoundException: ./src/main/resources/userAgents.txt (No
such file or directory)
I tried the maven-resources-plugin to copy the files into the target folder:
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<id>copy-resources</id>
<phase>package</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target/extra-resources</outputDirectory>
<includeEmptyDirs>true</includeEmptyDirs>
<resources>
<resource>
<directory>${basedir}/src/main/resources</directory>
<filtering>false</filtering>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
Even changing the path inside the program (to open file from target/extra-resources) the error persists.
I also added this <resources>, and got nothing:
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.txt</include>
<include>**/*.csv</include>
</includes>
</resource>
</resources>
Inside the program, I'm reading the file using:
String filePath = "./src/main/resources/userAgents.txt";
File extUserAgentLst = new File(filePath);
Scanner usrAgentReader = new Scanner(extUserAgentLst);
So, my question is:
How to make sure the userAgents.txt file is inside the .jar file, so that when I run it, the program reads from this file and doesn't return any exception?
You can use getResourceAsStream instead, like so:
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.stream.Collectors;
public class MyClass {
public static void main(String[] args) {
InputStream inStream = MyClass.class.getClassLoader().getResourceAsStream("userAgents.txt");
if (inStream != null) {
BufferedReader reader = new BufferedReader(new InputStreamReader(inStream));
String usersTxt = reader.lines().collect(Collectors.joining());
System.out.println(usersTxt);
}
}
}
It shouldn't be necessary to specify the tag <resources> in the pom.xml file. You just need to place your file inside src/main/resources before running the mvn package command to build the project.
I am using flink in a maven/java project and need to include my configs internally in the created jar.
So, I have added the following in my pom file. This includes all my yml configs (located in src/main/resources folder) in the jar, whose name I will pass as argument while executing.
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.yml</include>
</includes>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<finalName>${project.artifactId}-${project.version}</finalName>
<shadedArtifactAttached>true</shadedArtifactAttached>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.exmaple.MyApplication</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
The following main class code receives an arg based on which I decide what config to pick from resource, read(using snakeyaml) and use.
public static void main(String[] args) throws Exception {
final ParameterTool parameterTool = ParameterTool.fromArgs(args);
ClassLoader classLoader = MyApplication.class.getClassLoader();
Yaml yaml = new Yaml();
String filename = parameterTool.getRequired("configFilename");
InputStream in = classLoader.getSystemResourceAsStream(filename);
MyConfigClass = yaml.loadAs(in, MyConfigClass.class);
...
}
mvn clean install creates "my-shaded-jar.jar"
which I execute using command
java -jar /path/to/my-shaded-jar.jar --configFilename filename
It works on multiple systems, if I share the jar with others.
However I am facing issue, when I try to run the same jar in a yarn cluster on Hadoop, using the following command:-
HADOOP_CLASSPATH=`hadoop classpath` HADOOP_CONF_DIR=/etc/hadoop/conf ./flink-1.6.2/bin/flink run -m yarn-cluster -yd -yn 5 -ys 30 -yjm 10240 -ytm 10240 -yst -ynm some-job-name -yqu queue-name ./my-shaded-jar.jar --configFilename filename
I am getting following Error:
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:238)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:216)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1053)
at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1129)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1129)
Caused by: org.yaml.snakeyaml.error.YAMLException: java.io.IOException: Stream closed
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:200)
at org.yaml.snakeyaml.reader.StreamReader.<init>(StreamReader.java:60)
at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:444)
at com.example.MyApplication.main(MyApplication.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
... 13 more
Caused by: java.io.IOException: Stream closed
at java.io.PushbackInputStream.ensureOpen(PushbackInputStream.java:74)
at java.io.PushbackInputStream.read(PushbackInputStream.java:166)
at org.yaml.snakeyaml.reader.UnicodeReader.init(UnicodeReader.java:90)
at org.yaml.snakeyaml.reader.UnicodeReader.read(UnicodeReader.java:122)
at java.io.Reader.read(Reader.java:140)
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:184)
Why does my solution works on any normal linux/mac systems, however the same jar with same args fails when running with flink run command on yarn cluster.
Is there a difference between how we generally execute jars and how yarn does it.
Any help appreciated.
Replace classLoader.getSystemResourceAsStream(filename) with classLoader.getResourceAsStream(filename).
java.lang.ClassLoader#getSystemResourceAsStream locates the resource through the system class loader, which is typically used to start the application.
java.lang.ClassLoader#getResourceAsStream will first search the parent class loader. That failing, it will invoke findResource of the current class loader.
To avoid dependency conflicts, classes in Flink applications are divided into two domains [1], which is also applied to Flink client, e.g. CliFrontend.
The Java Classpath includes the classes of Apache Flink and its core dependencies.
The Dynamic User Code includes the classes (and resources) of user jars.
So in order to find your "config file", which is packaged in your jar file, we should use the user code class loader (you can find the details of userCodeClassLoader in org.apache.flink.client.program.PackagedProgram), instead of the system classloader.
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html
I have a command line app that downloads some reports, processes them, then uploads data to Google Drive. I'm using Typesafe-Config for all the magic strings I need. Typesafe-Config looks on the classpath for my application.conf file and uses HOCON to map config objects to fields in my class, like this:
From ~/configs/application.conf:
my.homePageUrl = "https://my.home.com"
From MyClass.java:
private static Config config = ConfigFactory.load();
private static final String HOME_URL = config.getString("my.homePageUrl");
I'm using the maven-shade-plugin to build an executable .jar for easy deployment on a remote server. The plugin looks like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4</version>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>com.my.reports.ReportRunner</Main-Class>
<Class-Path>~/configs/application.conf</Class-Path>
</manifestEntries>
</transformer>
</transformers>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
The problem is, when I run the executable .jar, application.conf isn't found on my classpath (I guess this also could be a bug in the typesafe code). All this works just fine in Intellij.
dustinevan#iMac:~/bin$ java -jar my-reports-1.0-SNAPSHOT.jar
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'my'
at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:147)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
at com.typesafe.config.impl.SimpleConfig.getList(SimpleConfig.java:212)
at com.typesafe.config.impl.SimpleConfig.getHomogeneousUnwrappedList(SimpleConfig.java:271)
at com.typesafe.config.impl.SimpleConfig.getStringList(SimpleConfig.java:329)
at com.stellarliving.reports.ecp.ReportRunner.<clinit>(ReportRunner.java:19)
dustinevan#iMac:~/configs$ ls
total 8
-rw-r--r--# 1 dustinevan staff 1520 Jun 13 01:16 application.conf
I've tried MANY permutations, and done lots of reading to solve this, any help would be greatly appreciated.
As I just had the same problem...
The -jar overrides all classpath settings, so only the jar is seen. -Dconfig.trace=loads will show what is seen by java.
We want the application.conf on the classpath, as well as the jar, so:
java -cp .:my-reports-1.0-SNAPSHOT.jar full.path.to.main.Main
did the trick for me. application.conf found and overrides reference.conf in the jar.
I also had that issue. I've noticed that you use a Class-Path declaration in shaded configuration, so I've merged the answer by Lothar with your info and added:
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>com.my.reports.ReportRunner</Main-Class>
<Class-Path>.</Class-Path>
</manifestEntries>
</transformer>
Here i am trying to load an excel file from resources folder(src/main/resources) of a maven project.
Folder Structure
MyWebApp
|______src/main/java
| |____Test.java
|
|______src/main/resources
| |______test
| |___hello.properties
| |___template.xlsx
|______target
|___MyWebApp
|____WEB_INF
|___classes
|__test
|__hello.properties
|__template.xlsx
My Approach
//loading excel file
String resource = "/test/template.xlsx";
System.out.println(this.getClass().getResource(resource) == null); // prints true
//loading properties file
String resource = "/test/hello.properties";
System.out.println(this.getClass().getResource(resource) == null); //prints false
//I have also tried below methods
this.getClass().getClassLoader().getResourceAsStream(resource); //null
new ClassPathResource(resource).getInputStream(); //null
After doing some googling i came to know, maven filters binary contents. To over come that i modified my pom.xml to allow .xlsx,.xls file extenstions not to be filtered with this help.
pom.xml
<configuration>
<outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/classes</outputDirectory>
<resources>
<resource>
<directory>${basedir}/src/main/resources</directory>
<includes>
<include>**/*.xlsx</include>
<include>**/*.xls</include>
</includes>
</resource>
</resources>
</configuration>
I could able to load the properties file, but i could not able to load the excel file by using above approach. From my side i referred the below two links (Rererence-1,
Reference-2) but no success. Please help me if you have some thoughts/ideas on this issue.
In the maven documentation page which you've linked in your there is said:
If you have both text files and binary files as resources it is
recommended to have two separated folders. One folder
src/main/resources (default) for the resources which are not filtered
and another folder src/main/resources-filtered for the resources which
are filtered.
So you should hold the properties and xlsx files in separate directories.
There is also an information about excluding binary files from filtering:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.0</version>
<configuration>
...
<nonFilteredFileExtensions>
<nonFilteredFileExtension>pdf</nonFilteredFileExtension>
<nonFilteredFileExtension>swf</nonFilteredFileExtension>
</nonFilteredFileExtensions>
...
</configuration>
</plugin>
I have a java project which uses .properties files for configuration. On the server, when launching, I set the classpath to include a folder which contains all the properties files. On my local machine, I'd like to point to a different folder.
I'm looking to add to the classpath, ideally for all projects, but adding it to each project is also fine. I've tried changing the Run > VM Options to include classpath, but with that change it can't find the main class, and I get java.lang.NoClassDefFoundError. I've also tried changing nbactions.xml directly to set the classpath to -classpath ~\MyFolder\;%classpath, but this has the same problem.
To add to the difficulty, the server is running linux while my local machine is running Windows.
I stuck with topic-starter issue also for a long time. My goal - put config files for debug purpose in project root and extend classpath to ${basedir}, so this code:
String appConfigLocation = System.getProperty("config.location");
if (appConfigLocation == null) {
logger.error("System property 'config.location' is not set...");
System.exit(1);
}
InputStream appConfigStream = Main.class.getClassLoader().getResourceAsStream(appConfigLocation);
if (appConfigStream == null) {
logger.error("Can't find resource {} in classpath, fix 'config.location'...", appConfigLocation);
System.exit(1);
}
Properties appProps = new Properties();
try {
appProps.load(appConfigStream);
} catch (IOException ex) {
System.out.println("IO error during loading of {}...", appConfigLocation);
System.exit(1);
}
read configs from ${basedir}. I like that instead of putting them to src/main/resources.
Check sources of ExecMojo.java for v1.2.1 http://grepcode.com/file/repo1.maven.org/maven2/org.codehaus.mojo/exec-maven-plugin/1.2.1/org/codehaus/mojo/exec/ExecMojo.java?av=f :
if ( CLASSPATH_TOKEN.equals( args[i] ) ) {
commandArguments.add( computeClasspathString( null ) );
}
and and v1.3.2 http://grepcode.com/file/repo1.maven.org/maven2/org.codehaus.mojo/exec-maven-plugin/1.3.2/org/codehaus/mojo/exec/ExecMojo.java?av=f:
if ( args[i].contains( CLASSPATH_TOKEN ) ) {
commandArguments.add( args[i].replace( CLASSPATH_TOKEN,
computeClasspathString( null ) ) );
}
So update NB config Execute goals to new version:
process-classes org.codehaus.mojo:exec-maven-plugin:1.3.2:exec
and use complex -classpath args in exec.argsparams:
exec.args=-classpath %classpath:.:"${basedir}" \
-Dconfig.location=app.properties \
-Dlogback.configurationFile=logback.xml \
${packageClassName}
Fix any action you need to have such behavior:
See also:
http://wiki.netbeans.org/FaqEnvVarsDuringRun
http://wiki.netbeans.org/FaqSysPropsDuringRun
Hi I had a similar need to give NetBeans7.4 a classpath to a jar with a driver outwith Maven dependencies e.g. c:\Program Files\Java\jdk1.7.0_25\db\lib\derby.jar in a Java Maven project called MyProject.
As you were thinking with the Run > VM Options, I would suggest the following:
1) right-click on MyProject to open project Properties
2) in 'Project Properties' pop-up, select 'Actions'
3) locate 'Run pproject' among the 'Actions' and select it
4) edit in 'Set Properties' text box entering
exec.args=-cp %classpath;.;"c:\Program Files\Java\jdk1.7.0_25\db\lib\derby.jar" biz.letsweb.derbyconnect.App
exec.executable=java
exec.workingdir=c:\Users\Tomasz\Documents\NetBeansProjects\DerbyConnect\target\classes
Alternatively edit nbactions.xml analogically. Once I did this, I could simply run the project inside NetBeans by pressing green arrow.
what about having the properties included as ? and those you use locally only have in a 's and activate that profile on local machine?
This is how I've been adding classpath to many of my projects
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>com.nitinsurana.policereports.GUI</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
An outside the box solution I ended up using for my similar case in NetBeans 7.3.1:
Adding files to java classpath at runtime
private static void addSoftwareLibrary(File file) throws Exception {
Method method = URLClassLoader.class.getDeclaredMethod("addURL", new Class[]{URL.class});
method.setAccessible(true);
method.invoke(ClassLoader.getSystemClassLoader(), new Object[]{file.toURI().toURL()});
}
This gives me a hacky way to add files to my classpath at run time via program arguments. Related notes I compiled while researching:
To include a dependency for compilation only, not runtime set:
Dependency Scope
<dependency><scope>provided</scope>...</dependency>
To exclude a dependency from the shaded jar, set: Exclude
<exclude>groupId:artifactId[[:type]:classifier]</exclude>
To copy resources to the target directory from outside the typical source directory: Copy Resources
<project>
...
<build>
<plugins>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>2.7</version>
<executions>
<execution>
<id>copy-resources</id>
<!-- here the phase you need -->
<phase>validate</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>target/extra-resources</outputDirectory>
<resources>
<resource>
<directory>extra-resources</directory>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
...
</build>
...
</project>
Note that the base path for directory is the project home. The linked post has <filtering>true</filtering> which can cause "invalid mark" in Netbeans 7.3.1.