I have a program that scrapes a webpage. I'm using JSoup and Selenium.
To configure the user agent in the JSoup request, I have a userAgents.txt file containing a list of user agents. In each execution, I have a method that reads the .txt file, and returns a random user agent.
The program is working as expected when running in IntelliJ.
The problem happens when I try to build the .jar file, with mvn clean package. When running the .jar file, I get a FileNotFoundException, since the program can't find the userAgents.txt file.
If I remove this functionality, and hardcode the user agent, I have no problems.
The file currently is in src/main/resources. When executing the .jar, I get the exception:
java.io.FileNotFoundException: ./src/main/resources/userAgents.txt (No
such file or directory)
I tried the maven-resources-plugin to copy the files into the target folder:
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<id>copy-resources</id>
<phase>package</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target/extra-resources</outputDirectory>
<includeEmptyDirs>true</includeEmptyDirs>
<resources>
<resource>
<directory>${basedir}/src/main/resources</directory>
<filtering>false</filtering>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
Even changing the path inside the program (to open file from target/extra-resources) the error persists.
I also added this <resources>, and got nothing:
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.txt</include>
<include>**/*.csv</include>
</includes>
</resource>
</resources>
Inside the program, I'm reading the file using:
String filePath = "./src/main/resources/userAgents.txt";
File extUserAgentLst = new File(filePath);
Scanner usrAgentReader = new Scanner(extUserAgentLst);
So, my question is:
How to make sure the userAgents.txt file is inside the .jar file, so that when I run it, the program reads from this file and doesn't return any exception?
You can use getResourceAsStream instead, like so:
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.stream.Collectors;
public class MyClass {
public static void main(String[] args) {
InputStream inStream = MyClass.class.getClassLoader().getResourceAsStream("userAgents.txt");
if (inStream != null) {
BufferedReader reader = new BufferedReader(new InputStreamReader(inStream));
String usersTxt = reader.lines().collect(Collectors.joining());
System.out.println(usersTxt);
}
}
}
It shouldn't be necessary to specify the tag <resources> in the pom.xml file. You just need to place your file inside src/main/resources before running the mvn package command to build the project.
Related
Hello I'm using a configuration file from src/main/resources in my java application. I'm reading it in my Class like this :
new BufferedReader(new FileReader(new File("src/main/resources/config.txt")));
So now I'm building this with maven using mvn assembly:assembly. Here is the bit for that in my pom.xml :
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.3</version>
<configuration>
<finalName>TestSuite</finalName>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>com.some.package.Test</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
So when I run my app I get this error :
src\main\resources\config.txt (The system cannot find the path specified)
But when I right click on my assembled jar I can see it inside, anyone knows what I'm doing wrong?
Resources from src/main/resources will be put onto the root of the classpath, so you'll need to get the resource as:
new BufferedReader(new InputStreamReader(getClass().getResourceAsStream("/config.txt")));
You can verify by looking at the JAR/WAR file produced by maven as you'll find config.txt in the root of your archive.
FileReader reads from files on the file system.
Perhaps you intended to use something like this to load a file from the class path
// this will look in src/main/resources before building and myjar.jar! after building.
InputStream is = MyClass.class.getClassloader()
.getResourceAsStream("config.txt");
Or you could extract the file from the jar before reading it.
The resources you put in src/main/resources will be copied during the build process to target/classes which can be accessed using:
...this.getClass().getResourceAsStream("/config.txt");
Once after we build the jar will have the resource files under BOOT-INF/classes or target/classes folder, which is in classpath, use the below method and pass the file under the src/main/resources as method call getAbsolutePath("certs/uat_staging_private.ppk"), even we can place this method in Utility class and the calling Thread instance will be taken to load the ClassLoader to get the resource from class path.
public String getAbsolutePath(String fileName) throws IOException {
return Thread.currentThread().getContextClassLoader().getResource(fileName).getFile();
}
we can add the below tag to tag in pom.xml to include these resource files to build target/classes folder
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.properties</include>
<include>**/*.ppk</include>
</includes>
</resource>
</resources>
I think assembly plugin puts the file on class path. The location will be different in in the JAR than you see on disk. Unpack the resulting JAR and look where the file is located there.
You can replace the src/main/resources/ directly by classpath:
So for your example you will replace this line:
new BufferedReader(new FileReader(new File("src/main/resources/config.txt")));
By this line:
new BufferedReader(new FileReader(new File("classpath:config.txt")));
Here i am trying to load an excel file from resources folder(src/main/resources) of a maven project.
Folder Structure
MyWebApp
|______src/main/java
| |____Test.java
|
|______src/main/resources
| |______test
| |___hello.properties
| |___template.xlsx
|______target
|___MyWebApp
|____WEB_INF
|___classes
|__test
|__hello.properties
|__template.xlsx
My Approach
//loading excel file
String resource = "/test/template.xlsx";
System.out.println(this.getClass().getResource(resource) == null); // prints true
//loading properties file
String resource = "/test/hello.properties";
System.out.println(this.getClass().getResource(resource) == null); //prints false
//I have also tried below methods
this.getClass().getClassLoader().getResourceAsStream(resource); //null
new ClassPathResource(resource).getInputStream(); //null
After doing some googling i came to know, maven filters binary contents. To over come that i modified my pom.xml to allow .xlsx,.xls file extenstions not to be filtered with this help.
pom.xml
<configuration>
<outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/classes</outputDirectory>
<resources>
<resource>
<directory>${basedir}/src/main/resources</directory>
<includes>
<include>**/*.xlsx</include>
<include>**/*.xls</include>
</includes>
</resource>
</resources>
</configuration>
I could able to load the properties file, but i could not able to load the excel file by using above approach. From my side i referred the below two links (Rererence-1,
Reference-2) but no success. Please help me if you have some thoughts/ideas on this issue.
In the maven documentation page which you've linked in your there is said:
If you have both text files and binary files as resources it is
recommended to have two separated folders. One folder
src/main/resources (default) for the resources which are not filtered
and another folder src/main/resources-filtered for the resources which
are filtered.
So you should hold the properties and xlsx files in separate directories.
There is also an information about excluding binary files from filtering:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.0</version>
<configuration>
...
<nonFilteredFileExtensions>
<nonFilteredFileExtension>pdf</nonFilteredFileExtension>
<nonFilteredFileExtension>swf</nonFilteredFileExtension>
</nonFilteredFileExtensions>
...
</configuration>
</plugin>
I have a app.properties file in my maven project under resources folder as shown here (simplified):
myApp
|----src
| |
| |--main
| |--java
| | |--ApplicationInitializer.java
| |
| |--resources
| |--app.properties
|
|---target
|--myApp.jar
|--app.properties
In ApplicationInitializer class I want to load properties from the app.properties file with following piece of code:
Properties props = new Properties();
String path = "/app.properties";
try {
props.load(ApplicationInitializer.class.getResourceAsStream(path));
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(props.getProperty("property"));
This piece of code loads properties correctly when I run it from inside my IDE but fails with exception
Exception in thread "main" java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:434)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at cz.muni.fi.fits.ApplicationInitializer.main(ApplicationInitializer.java:18)
when trying to run in as JAR file.
For creating a jar file I am using combination of maven-shade-plugin, maven-jar-plugin (for excluding properties file outside of the JAR) and maven-resources-plugin (for copying properties file to specific folder) in pom.xml file as shown here:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>cz.muni.fi.fits.ApplicationInitializer</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.5</version>
<configuration>
<excludes>
<exclude>**/*.properties</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>2.7</version>
<executions>
<execution>
<id>copy-resource</id>
<phase>package</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target</outputDirectory>
<resources>
<resource>
<directory>src/main/resources</directory>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
I then switched the code in main method to this one:
Properties props = new Properties();
String path = "./app.properties";
try (FileInputStream file = new FileInputStream(path)) {
props.load(file);
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(props.getProperty("property"));
and managed to load properties from file when running JAR but this time I was not able to load them when running in IDE, it ended with the same exception as above.
So my question is: How to set the filepath (or pom.xml file?) that I will be able to load properties running from both IDE and JAR file?
Thanks in advance :)
In your Java code read your app.properties file like this:
final String ROOT_PATH = "custom.path"; // Or whatever you most like
final String PROPERTIES_FILE = "app.properties";
// start :: try-catch here to manage I/O resources
File directory = new File(System.getProperty(ROOT_PATH), "conf");
File file = new File(directory, PROPERTIES_FILE);
BufferedInputStream bufferedInputStream = new BufferedInputStream(new FileInputStream(file));
// ...
// end :: try-catch here to manage I/O resources
Then you have to ways for setting/declaring custom.path:
Declaring it as an environment variable
Setting it at run-time (environment properties) with the -Dcustom.path=/somewhere/in/your/filesystem (I prefer this one, unless the path is fixed and/or shared across different applications)
Then, eventually you need to copy/put your app.properties file inside /somewhere/in/your/filesystem/conf. And why inside /conf? Because you use it when you declare the directory field. If you don't want it in there just don't set it and delete the , "conf" part.
Additionally, for running it "locally" (in your IDE) use the VM options setting (IntelliJ IDEA):
The way you're working at the moment you depend on the working directory of your java process. Typically that's the your command line (aka shell) points to, when starting the application.
On most IDE's you can configure this directory in your launcher settings. (For eclipse it's on the second tab 'Arguments') So you will need to the target directory (For eclipse there's a button 'Workspace...')
Specifically in intelliJ you can find this setting under Run/Edit Configurations... This will open a window like this:
There you can edit the working directory. You simply add target at the end.
Edit:
Actually you've add src/main/ressources at the end.
I am trying to run 'mvn test' with my java code loading 'log4j.xml'. I am getting file not found exception. Would be great to get some help.
My java code:
public class MyClass {
static Logger logger = Logger.getLogger(MyClass.class);
public boolean check(){
try{
DOMConfigurator.configure("log4j.xml");
logger.info("into the method");
}
return true;
}
}
Junit test case:
import static org.junit.Assert.*;
import org.junit.Test;
public class MyClassTests {
#Test
public void testCheck() {
MyClass myc = new MyClass();
assertEquals(true, myc.check());
}
}
block of pom.xml
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.15</version>
<configuration>
<includes>
<include>**/*Tests.java</include>
<include>log4j.xml</include>
</includes>
<argLine>-Xmx512m</argLine>
</configuration>
<inherited>true</inherited>
</plugin>
File Structure:
pom.xml = */myproject/pom.xml
log4j.xml = */myproject/src/main/resources/log4j.xml
myclass.java = */myproject/src/main/java/MyClass.java
junit test = */myproject/src/test/java/MyClassTests.java
When I run the command "mvn test", I get a file not found exception for log4j.xml:
java.io.FileNotFoundException:
*/myproject/log4j.xml (No such file or
directory)
Basically the code is looking for log4j.xml in the root folder of the project rather than in the /src/main/resources folder. I do not want to change the code to "DOMConfigurator.configure("/src/main/resources/log4j.xml");" This will cause issue in the deployed application. Need a more elegant solution.
I can't see your POM but you should have a resources section in your build declaration to ensure the src/main/resources are available as a class resource.
<build>
<outputDirectory>build/maven/${artifactId}/target/classes</outputDirectory>
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.xml</include>
</includes>
</resource>
</resources>
</build>
Then the DOMConfigurator should be able to locate the log4j.xml as a class resource. If you run a maven package on your project you should look in the build or target folder and see that the log4j.xml is copied to the classes folder. Then you'll know it should be available as a resource.
Hello I'm using a configuration file from src/main/resources in my java application. I'm reading it in my Class like this :
new BufferedReader(new FileReader(new File("src/main/resources/config.txt")));
So now I'm building this with maven using mvn assembly:assembly. Here is the bit for that in my pom.xml :
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.3</version>
<configuration>
<finalName>TestSuite</finalName>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>com.some.package.Test</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
So when I run my app I get this error :
src\main\resources\config.txt (The system cannot find the path specified)
But when I right click on my assembled jar I can see it inside, anyone knows what I'm doing wrong?
Resources from src/main/resources will be put onto the root of the classpath, so you'll need to get the resource as:
new BufferedReader(new InputStreamReader(getClass().getResourceAsStream("/config.txt")));
You can verify by looking at the JAR/WAR file produced by maven as you'll find config.txt in the root of your archive.
FileReader reads from files on the file system.
Perhaps you intended to use something like this to load a file from the class path
// this will look in src/main/resources before building and myjar.jar! after building.
InputStream is = MyClass.class.getClassloader()
.getResourceAsStream("config.txt");
Or you could extract the file from the jar before reading it.
The resources you put in src/main/resources will be copied during the build process to target/classes which can be accessed using:
...this.getClass().getResourceAsStream("/config.txt");
Once after we build the jar will have the resource files under BOOT-INF/classes or target/classes folder, which is in classpath, use the below method and pass the file under the src/main/resources as method call getAbsolutePath("certs/uat_staging_private.ppk"), even we can place this method in Utility class and the calling Thread instance will be taken to load the ClassLoader to get the resource from class path.
public String getAbsolutePath(String fileName) throws IOException {
return Thread.currentThread().getContextClassLoader().getResource(fileName).getFile();
}
we can add the below tag to tag in pom.xml to include these resource files to build target/classes folder
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.properties</include>
<include>**/*.ppk</include>
</includes>
</resource>
</resources>
I think assembly plugin puts the file on class path. The location will be different in in the JAR than you see on disk. Unpack the resulting JAR and look where the file is located there.
You can replace the src/main/resources/ directly by classpath:
So for your example you will replace this line:
new BufferedReader(new FileReader(new File("src/main/resources/config.txt")));
By this line:
new BufferedReader(new FileReader(new File("classpath:config.txt")));