Develop Reference

Maven Package: .txt is not being included in the .jar file

I have a program that scrapes a webpage. I'm using JSoup and Selenium.
To configure the user agent in the JSoup request, I have a userAgents.txt file containing a list of user agents. In each execution, I have a method that reads the .txt file, and returns a random user agent.
The program is working as expected when running in IntelliJ.
The problem happens when I try to build the .jar file, with mvn clean package. When running the .jar file, I get a FileNotFoundException, since the program can't find the userAgents.txt file.
If I remove this functionality, and hardcode the user agent, I have no problems.
The file currently is in src/main/resources. When executing the .jar, I get the exception:
java.io.FileNotFoundException: ./src/main/resources/userAgents.txt (No
such file or directory)
I tried the maven-resources-plugin to copy the files into the target folder:
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<id>copy-resources</id>
<phase>package</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target/extra-resources</outputDirectory>
<includeEmptyDirs>true</includeEmptyDirs>
<resources>
<resource>
<directory>${basedir}/src/main/resources</directory>
<filtering>false</filtering>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
Even changing the path inside the program (to open file from target/extra-resources) the error persists.
I also added this <resources>, and got nothing:
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.txt</include>
<include>**/*.csv</include>
</includes>
</resource>
</resources>
Inside the program, I'm reading the file using:
String filePath = "./src/main/resources/userAgents.txt";
File extUserAgentLst = new File(filePath);
Scanner usrAgentReader = new Scanner(extUserAgentLst);
So, my question is:
How to make sure the userAgents.txt file is inside the .jar file, so that when I run it, the program reads from this file and doesn't return any exception?

You can use getResourceAsStream instead, like so:
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.stream.Collectors;
public class MyClass {
public static void main(String[] args) {
InputStream inStream = MyClass.class.getClassLoader().getResourceAsStream("userAgents.txt");
if (inStream != null) {
BufferedReader reader = new BufferedReader(new InputStreamReader(inStream));
String usersTxt = reader.lines().collect(Collectors.joining());
System.out.println(usersTxt);
}
}
}
It shouldn't be necessary to specify the tag <resources> in the pom.xml file. You just need to place your file inside src/main/resources before running the mvn package command to build the project.

Rerunning failed cucumber tests using cucumber-jvm

I have a Cucumber-JVM, JUnit, Selenium setup. I initiate the run by running RunSmokeTests.java using JUnit within Eclipse. I have also set up a maven profile to run the tests from command line, and possibly Jenkins in the future.
When the tests are run then some of them may fail sometimes, mainly due to the application taking longer than expected. I would then have to re-run these scenarios. At the moment I run them by manually attaching #rerun tag to the ones that failed and then running RunReruns.java, which is similar to RunSmokeTest.java but with #rerun tag.
With the increasing number of automated tests it is time consuming to tag the tests and start the run and clear the tags. Is there a automated way with Cucumber-JVM to re-run failed tests?
RunSmokeTests.java
package testGlueClasses;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
#RunWith(Cucumber.class)
#Cucumber.Options(features = "src/test/java", strict = true, format = {
"html:target/CucumberReport", "json:target/JSON/Cucumber.json",
"FrameworkCore.CustomTestReporter" }, tags = { "#SmokeTest" }, glue = {
"FrameworkCore", "MyApp.Utils", "MyApp.StepDefinitions" })
public class RunSmokeTests {
}
Maven snippet:
<profile>
<id>smoke</id>
<properties>
<include.tests>
**/RunSmokeTests.java
</include.tests>
</properties>
</profile>

I came up with another solution to rerun just failed test using maven & cucumber.
1) Record test failures using a RunNotifier
public class RerunningCucumber extends Cucumber {
private final String className;
#SuppressWarnings("rawtypes")
public RerunningCucumber(Class clazz) throws InitializationError, IOException {
super(clazz);
className = clazz.getSimpleName();
}
#Override
public void run(RunNotifier notifier) {
notifier.addListener(new RunListener(){
public void testFailure(Failure failure) throws Exception {
Throwable error = failure.getException();
if (error instanceof AssertionError){
//Nothing. This is a normal failure. Continue
return;
}
//No! A wild exception has appeared!
//Let's run this test again.
RerunningCucumber.addFile(className);
}
});
super.run(notifier);
}
private static final String filename = "target/rerun.properties";
private static final Set<String> addedClasses = new HashSet<String>();
public static synchronized void addFile(String className) throws IOException{
//First find the file
if (addedClasses.contains(className)){
return;
}
File file = new File(filename);
if (!file.exists()){
//Need to create the file
PrintWriter writer = new PrintWriter(file, "UTF-8");
writer.print("retryclasses=**/"+className+".class");
writer.close();
}
else {
PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(file, true)));
out.print(",**/"+className+".class");
out.close();
}
addedClasses.add(className);
}
}
2) Use custom class as a runner for the cucumber tests.
This will run the tests, and whenever there is a failure, output the failed class to a file. Trick is to keep features short and create a lot of test classes to avoid repeating tests.
#RunWith(RerunningCucumber.class)
#CucumberOptions(features = {"classpath:features/testFeature.feature}, format = {
"html:target/cucumber-html-report/testFeature.html",
"json:target/cucumber-json-report/testFeature.json"},
tags = {"#testFeature"})
public class RunTestFeature {
}
3) Add a Rerun profile to maven.
This does three things: 1) it loads the failed classes into memory, 2) cleans JUST the failed classes properties file, and 3) reruns ONLY the failed tests as loaded from the properties file:
<profile>
<id>retry</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>properties-maven-plugin</artifactId>
<version>1.0-alpha-2</version>
<executions>
<!-- Associate the read-project-properties goal with the initialize
phase, to read the properties file. -->
<execution>
<phase>pre-clean</phase>
<goals>
<goal>read-project-properties</goal>
</goals>
<configuration>
<files>
<file>target/rerun.properties</file>
</files>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-clean-plugin</artifactId>
<version>2.6.1</version>
<configuration>
<filesets>
<fileset>
<directory>target</directory>
<includes>
<include>rerun.properties</include>
</includes>
</fileset>
</filesets>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<echo>Retrying the following classes: "${retryclasses}"</echo>
</target>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<includes>
<include>${retryclasses}</include>
</includes>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
4) Usage
First test run:
mvn clean test
Next test runs:
mvn clean test -Pretry
mvn clean test -Pretry
mvn clean test -Pretry
...
You can repeat as many times as you want until there are no errors.

I don't have an executable example at hand, but you can do this also on the jvm. There is a RerunFormatter that writes a text file listing the file and line numbers of failed scenarios:
#CucumberOptions(format = {"rerun:target/rerun.txt"})
You should be able to specify this file as input for another test class by prefixing it with #:
#CucumberOptions(features = {"#target/rerun.txt"})

You can pass cucumber options to mvn as below
mvn clean verify -Dcucumber.options="#rerun.txt"
Note there is a tricky part here. If you are using the same test runner for both first run and rerun (and I believe that's what you want), then the test runner would contains something like
#CucumberOptions(plugin = { "rerun:target/rerun.txt"})
If you fire your rerun with maven using the same rerun file name as below
mvn clean verify -Dcucumber.options="#target/rerun.txt"
then cucumber will complain it could not find the rerun file. Why? Because the plugin "rerun:target/rerun.txt" will delete the file first with this test runner.
Workaround is copy/rename the file first, then kick off the mvn run like
mv target/rerun.txt rerun.txt && mvn clean verify -Dcucumber.options="#rerun.txt"
And this is actually what you want. Because say if there are 5 failed scenarios in file target/rerun.txt. And with the rerun after some fix, 2 of them passed. Now the target/rerun.txt will contain the remaining 3 failed scenarios only, which would be your new start point along the debugging way.

For cucumber + java on maven i found this command:
mvn clean test -Dsurefire.rerunFailingTestsCount=2
You must have an actual version of surefire plugin, mine is 3.0.0-M5.
And nothing else special u even need.
Found solution here Surefire rerun failing tests not working

1)
With junit4 (cucumber-junit engine) it can be done easily with rerun plugin
and features cucumber option. Add another maven profile for runner for failed scenarios, for example RerunCucumber.class.
Run your initial build with main test runner and pass rerun plugin:
#RunWith(Cucumber.class)
#CucumberOptions(tags = "#wip",
monochrome = true,
plugin = {"html:target/cucumber", "json:target/wip.json", "rerun:target/rerun_wip.txt"})
public class RunCucumber {
}
After build is finished, failed scenarios will be written to target/rerun_wip.txt.
Then failed scenarios can be executed via rerunner:
#RunWith(Cucumber.class)
#CucumberOptions(features = {"#features = {"#target/rerun_wip.txt"}"},
monochrome = true,
plugin = {"html:target/rerun/failed_tests", "json:target/rerun/failed_tests.json"})
public class RerunCucumber {
}
Will be executed tests from target/rerun_wip.txt.
2)
With junit5 (cucumber-junit-platform-engine) there is no such approach (no features cucumber option). More read about rerun failed scenarios with junit5: https://github.com/cucumber/cucumber-jvm/tree/main/junit-platform-engine, in 'Rerunning failed scenarios' section

You can use cucumber-jvm-parallel-plugin contributed code as a workaround until it goes live. Hit commands as shown below.
git clone -b tagwiseOutlinewiseIssueRerun https://github.com/sugatmankar/cucumber-jvm-parallel-plugin.git
mvn clean install.
Now edit you project pom file and use as stated here.
Example for using this plugin is here.

Maven resources/ in sibling modules

I have a issue with some tests in sibling modules.
I have the following folder in module 1 in resources/: sql/statements
all the files in that folder is loaded with the following code:
URL url = this.getClass().getClassLoader().getResource("sql/statements");
assert url != null;
Collection<File> files = FileUtils.listFiles(new File(url.getPath()),
FileFilterUtils.suffixFileFilter("sql.xml"), TrueFileFilter.INSTANCE);
This works fine:
inside intellij for the sibling module
inside intellij in the module with the xml files
with maven clean install in the module with the xml files
but for some reason when I reference this class in a test in the sibling module it is unable to find the xml files when I run clean install from command line.
how can I solve this?
edit: when it fails it print the follwing url path, which seem ok (?):
url: jar:file:/home/<user>/.m2/repository/<path>/1.0.0/<jar-name>1.0.0.jar!/sql/statements

Check your POM files : you are probably only copying the resources in the build of your "sql/xml" module. Better move it up to the parent POM if you want to use it in sibling modules.

As for this maven doc, test resources should be stored in
src/test/resources
But you can set parameter testResources to the same directory as main resources
<build>
<testResources>
<testResource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</testResource>
<testResources>
<resources> <!-- this is default setting -->
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</resource>
</resources>

surefire not picking up properties file

Hi I have module inside a project which i'm trying to run from the modules directory as such:
mvn integration-test -PmyProps
in pom
<properties>
<user>admin</user>
</properties
in src/test/resources/test.properties
user=${user}
When i execute the test i get user={user} instead of user=admin
If i run it via intellij or eclipse i have no problems with it and properties get picked up?
is it because the property values are getting assigned to the target directory?
This is the class loader:
InputStream testPropertiesInput = WebDriverConfiguration.class.getClassLoader().getResourceAsStream("smoke.properties");
i've tried making it "target/classes/smoke.properties" but no luck.

First if you have unit tests which seemed to be in your case use simply:
mvn test
furthermore to get resource files beeing filtered you need to active filtering
<build>
<resources>
<resource>
<directory>...</directory>
<filtering>true</filtering>
</resource>
...
</build>

Error when using XmlBeans generated classes

I've generated classes with XMLBeans from an xsd file and packed them in a jar file. then I've added that jar to the project classpath in eclipse and everything compiles and runs fine.
I built a stand alone jar file from my project with Maven and again the build is successful, but when i try running it i get this error:
Exception in thread "main" java.lang.ExceptionInInitializerError
at com.oblicore.oblisync.resolutions.TestsDocument$Factory.parse(TestsDo
cument.java:126)
at com.oblicore.oblisync.handlers.TransferEntitiesHandler.getResolution(
TransferEntitiesHandler.java:117)
at com.oblicore.oblisync.handlers.TransferEntitiesHandler.resolveConflic
ts(TransferEntitiesHandler.java:103)
at com.oblicore.oblisync.main.Orchestrator.run(Orchestrator.java:107)
at com.oblicore.oblisync.main.Orchestrator.main(Orchestrator.java:58)
Caused by: java.lang.RuntimeException: Cannot load SchemaTypeSystem. Unable to l
oad class with name schemaorg_apache_xmlbeans.system.s8B21CFFFCFED0B2438C2585C61
F113F7.TypeSystemHolder. Make sure the generated binary files are on the classpa
th.
at org.apache.xmlbeans.XmlBeans.typeSystemForClassLoader(XmlBeans.java:7
83)
at com.oblicore.oblisync.resolutions.TestsDocument.<clinit>(TestsDocumen
t.java:19)
... 5 more
Caused by: java.lang.ClassNotFoundException: schemaorg_apache_xmlbeans.system.s8
B21CFFFCFED0B2438C2585C61F113F7.TypeSystemHolder
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.apache.xmlbeans.XmlBeans.typeSystemForClassLoader(XmlBeans.java:7
69)
... 6 more
The missing class is in the jar i created with XmlBeans, how do i tell maven to add it to the jar it creates from my project?

While doing WSDL2Java a directory named resources will be created. Copy the schemaorg_apache_xmlbeans which presents under resources to classpath of your project. This should be the fix.

In your generated jar file make sure you have included the class files generated from your xmlbeans.
From the stacktrace :
Caused by: java.lang.ClassNotFoundException: schemaorg_apache_xmlbeans.system.s8
B21CFFFCFED0B2438C2585C61F113F7.TypeSystemHolder
it suggests that during compile time the required class files are in classpath but in your built jar these files are missing.
Check your jar file to see if these classes are present.
EDIT: As per question rephrased
For building jar with dependecies in Maven use jar-with-dependencies option, example
Two very good reference :
http://www.sonatype.com/books/mvnref-book/reference/assemblies-sect-basics.html
http://thomassundberg.wordpress.com/2011/03/05/create-an-executable-jar-from-maven/
In the second one you don't need a main class if your jar is not an executable jar

Please add below tag in pom.xml. Error wil go
<!--
this tells maven to copy the openejb-javaagent jar into your target/
directory
-->
<!-- where surefire can see it -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<id>copy</id>
<phase>process-resources</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>org.apache.openejb</groupId>
<artifactId>openejb-javaagent</artifactId>
<version>3.0-beta-2</version>
<outputDirectory>${project.build.directory}</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<resources>
<resource>
<directory>target/generated-sources/axis2/wsdl2code/resources</directory>
</resource>
<resource>
<directory>target/generated-sources/xmlbeans/resources</directory>
</resource>
<resource>
<directory>src/main/resources</directory>
</resource>
</resources>
</build>

When you have this kind of error
The TypeSystemHolder.class generated by WSDL2Java is not be placed in your classpath in order to avoid this error.
Please copy TypeSystemHolder.class from "resource/schemaorg_apache_xmlbeans/system/s68C41DB812F52C975439BA10FE4FEE54" folder.
And Paste TypeSystemHolder.class file into your classpath folder (build/classes/schemaorg_apache_xmlbeans/system/s68C41DB812F52C975439BA10FE4FEE54) folder

Extract jar in which you want to include schemaorg_apache_xmlbeans folder.
Copy schemaorg_apache_xmlbeans folder in extracted folder (result from jar extraction).
open command prompt in extracted folder.
make jar again using jar cf command. e.g jar cf test.jar *,
to include all folders.
Deploy that jar .

I meet this problem. In our project we use customize classloader to compatible poi old version. when parse '.xlsx' file ,It's occurs the same error. follow is the origin code
private static String EXCEL_PATH = "com.alibaba.excel";
private static String POI_PATH = "org.apache.poi";
private static String OOXML_PATH="org.openxmlformats.schemas";
private static String MICRO_OOXML_PATH="com.microsoft.schemas";
private static String SCHEMAORG_APACHE_XMLBEANS="schemaorg_apache_xmlbeans";
#Override
protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
if (name.startsWith(POI_PATH)
|| name.startsWith(EXCEL_PATH)
|| name.startsWith(OOXML_PATH)
|| name.startsWith(MICRO_OOXML_PATH)
) {
synchronized (getClassLoadingLock(name)) {
// First, check if the class has already been loaded
Class<?> c = findLoadedClass(name);
if (c == null) {
c = findClass(name);
if (c == null) {
throw new ClassNotFoundException("custom classloader can't find this class");
}
}
if (resolve) {
resolveClass(c);
}
return c;
}
} else {
return oldClassLoader.loadClass(name);
}
}
we use customize classloader to load ooxml-schemes.jar,but schemaorg_apache_xmlbeans.system.s8
B21CFFFCFED0B2438C2585C61F113F7.TypeSystemHolder is not in our classloader load path ,default classloader load it .
so i solved it by change my classloader code
protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
*// we change here to load schemaorg...*
if (name.startsWith("schemaorg_apache_xmlbeans")
) {
//todo
}
//todo
}
to loader this class ,then it's be ok