I have a Cucumber-JVM, JUnit, Selenium setup. I initiate the run by running RunSmokeTests.java using JUnit within Eclipse. I have also set up a maven profile to run the tests from command line, and possibly Jenkins in the future.
When the tests are run then some of them may fail sometimes, mainly due to the application taking longer than expected. I would then have to re-run these scenarios. At the moment I run them by manually attaching #rerun tag to the ones that failed and then running RunReruns.java, which is similar to RunSmokeTest.java but with #rerun tag.
With the increasing number of automated tests it is time consuming to tag the tests and start the run and clear the tags. Is there a automated way with Cucumber-JVM to re-run failed tests?
RunSmokeTests.java
package testGlueClasses;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
#RunWith(Cucumber.class)
#Cucumber.Options(features = "src/test/java", strict = true, format = {
"html:target/CucumberReport", "json:target/JSON/Cucumber.json",
"FrameworkCore.CustomTestReporter" }, tags = { "#SmokeTest" }, glue = {
"FrameworkCore", "MyApp.Utils", "MyApp.StepDefinitions" })
public class RunSmokeTests {
}
Maven snippet:
<profile>
<id>smoke</id>
<properties>
<include.tests>
**/RunSmokeTests.java
</include.tests>
</properties>
</profile>
I came up with another solution to rerun just failed test using maven & cucumber.
1) Record test failures using a RunNotifier
public class RerunningCucumber extends Cucumber {
private final String className;
#SuppressWarnings("rawtypes")
public RerunningCucumber(Class clazz) throws InitializationError, IOException {
super(clazz);
className = clazz.getSimpleName();
}
#Override
public void run(RunNotifier notifier) {
notifier.addListener(new RunListener(){
public void testFailure(Failure failure) throws Exception {
Throwable error = failure.getException();
if (error instanceof AssertionError){
//Nothing. This is a normal failure. Continue
return;
}
//No! A wild exception has appeared!
//Let's run this test again.
RerunningCucumber.addFile(className);
}
});
super.run(notifier);
}
private static final String filename = "target/rerun.properties";
private static final Set<String> addedClasses = new HashSet<String>();
public static synchronized void addFile(String className) throws IOException{
//First find the file
if (addedClasses.contains(className)){
return;
}
File file = new File(filename);
if (!file.exists()){
//Need to create the file
PrintWriter writer = new PrintWriter(file, "UTF-8");
writer.print("retryclasses=**/"+className+".class");
writer.close();
}
else {
PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(file, true)));
out.print(",**/"+className+".class");
out.close();
}
addedClasses.add(className);
}
}
2) Use custom class as a runner for the cucumber tests.
This will run the tests, and whenever there is a failure, output the failed class to a file. Trick is to keep features short and create a lot of test classes to avoid repeating tests.
#RunWith(RerunningCucumber.class)
#CucumberOptions(features = {"classpath:features/testFeature.feature}, format = {
"html:target/cucumber-html-report/testFeature.html",
"json:target/cucumber-json-report/testFeature.json"},
tags = {"#testFeature"})
public class RunTestFeature {
}
3) Add a Rerun profile to maven.
This does three things: 1) it loads the failed classes into memory, 2) cleans JUST the failed classes properties file, and 3) reruns ONLY the failed tests as loaded from the properties file:
<profile>
<id>retry</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>properties-maven-plugin</artifactId>
<version>1.0-alpha-2</version>
<executions>
<!-- Associate the read-project-properties goal with the initialize
phase, to read the properties file. -->
<execution>
<phase>pre-clean</phase>
<goals>
<goal>read-project-properties</goal>
</goals>
<configuration>
<files>
<file>target/rerun.properties</file>
</files>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-clean-plugin</artifactId>
<version>2.6.1</version>
<configuration>
<filesets>
<fileset>
<directory>target</directory>
<includes>
<include>rerun.properties</include>
</includes>
</fileset>
</filesets>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<echo>Retrying the following classes: "${retryclasses}"</echo>
</target>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<includes>
<include>${retryclasses}</include>
</includes>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
4) Usage
First test run:
mvn clean test
Next test runs:
mvn clean test -Pretry
mvn clean test -Pretry
mvn clean test -Pretry
...
You can repeat as many times as you want until there are no errors.
I don't have an executable example at hand, but you can do this also on the jvm. There is a RerunFormatter that writes a text file listing the file and line numbers of failed scenarios:
#CucumberOptions(format = {"rerun:target/rerun.txt"})
You should be able to specify this file as input for another test class by prefixing it with #:
#CucumberOptions(features = {"#target/rerun.txt"})
You can pass cucumber options to mvn as below
mvn clean verify -Dcucumber.options="#rerun.txt"
Note there is a tricky part here. If you are using the same test runner for both first run and rerun (and I believe that's what you want), then the test runner would contains something like
#CucumberOptions(plugin = { "rerun:target/rerun.txt"})
If you fire your rerun with maven using the same rerun file name as below
mvn clean verify -Dcucumber.options="#target/rerun.txt"
then cucumber will complain it could not find the rerun file. Why? Because the plugin "rerun:target/rerun.txt" will delete the file first with this test runner.
Workaround is copy/rename the file first, then kick off the mvn run like
mv target/rerun.txt rerun.txt && mvn clean verify -Dcucumber.options="#rerun.txt"
And this is actually what you want. Because say if there are 5 failed scenarios in file target/rerun.txt. And with the rerun after some fix, 2 of them passed. Now the target/rerun.txt will contain the remaining 3 failed scenarios only, which would be your new start point along the debugging way.
For cucumber + java on maven i found this command:
mvn clean test -Dsurefire.rerunFailingTestsCount=2
You must have an actual version of surefire plugin, mine is 3.0.0-M5.
And nothing else special u even need.
Found solution here Surefire rerun failing tests not working
1)
With junit4 (cucumber-junit engine) it can be done easily with rerun plugin
and features cucumber option. Add another maven profile for runner for failed scenarios, for example RerunCucumber.class.
Run your initial build with main test runner and pass rerun plugin:
#RunWith(Cucumber.class)
#CucumberOptions(tags = "#wip",
monochrome = true,
plugin = {"html:target/cucumber", "json:target/wip.json", "rerun:target/rerun_wip.txt"})
public class RunCucumber {
}
After build is finished, failed scenarios will be written to target/rerun_wip.txt.
Then failed scenarios can be executed via rerunner:
#RunWith(Cucumber.class)
#CucumberOptions(features = {"#features = {"#target/rerun_wip.txt"}"},
monochrome = true,
plugin = {"html:target/rerun/failed_tests", "json:target/rerun/failed_tests.json"})
public class RerunCucumber {
}
Will be executed tests from target/rerun_wip.txt.
2)
With junit5 (cucumber-junit-platform-engine) there is no such approach (no features cucumber option). More read about rerun failed scenarios with junit5: https://github.com/cucumber/cucumber-jvm/tree/main/junit-platform-engine, in 'Rerunning failed scenarios' section
You can use cucumber-jvm-parallel-plugin contributed code as a workaround until it goes live. Hit commands as shown below.
git clone -b tagwiseOutlinewiseIssueRerun https://github.com/sugatmankar/cucumber-jvm-parallel-plugin.git
mvn clean install.
Now edit you project pom file and use as stated here.
Example for using this plugin is here.
Related
I am using OpenApi(3.0) for api definition and the openapi-generator-maven-plugin which generates files for me (api objects + endpoints).
Its however generating a test file in the build folder that I do not want. Its called 'OpenApiGeneratorApplicationTests'. It always blocks my compilation bc in the 'target' (=build) folder I do not have the right Spring Boot setup.
How can I avoid the generation of this test file?
This is my maven config:
<plugin>
<groupId>org.openapitools</groupId>
<artifactId>openapi-generator-maven-plugin</artifactId>
<version>6.0.1</version>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<inputSpec>${project.basedir}/src/main/resources/ApiDef.yaml</inputSpec>
<generatorName>spring</generatorName>
<modelPackage>${clientPackage}.model</modelPackage>
<invokerPackage>${clientPackage}.invoker</invokerPackage>
<apiPackage>${clientPackage}.api</apiPackage>
<generateApis>true</generateApis>
<generateApiTests>false</generateApiTests>
<generateModelTests>false</generateModelTests>
<configOptions>
<delegatePattern>true</delegatePattern>
</configOptions>
</configuration>
</execution>
</executions>
</plugin>
I've encountered today the same issue. So long story short:
By default the generator maven plugin for Spring will generate the 'invoker' - which by implementation is actually a #SpringBootApplication starter class.
#SpringBootApplication
#ComponentScan(basePackages = {"com.mypackage.invoker", "com.mypackage.api" , "org.openapitools.configuration"})
public class OpenApiGeneratorApplication {
public static void main(String[] args) {
SpringApplication.run(OpenApiGeneratorApplication.class, args);
}
#Bean
public Module jsonNullableModule() {
return new JsonNullableModule();
}
}
and will also generate the associated test class OpenApiGeneratorApplicationTests
#SpringBootTest
class OpenApiGeneratorApplicationTests {
#Test
void contextLoads() {
}
}
The problem here is that the src/test/com/mypackage/invoker/OpenApiGeneratorApplicationTests.java, and actually the whole package under test is marked as source, not as test-source and in my case this will result in compile time errors (since the test-scoped dependencies are not found).
I managed to get around it by configuring the maven plugin with
<configOptions>
<interfaceOnly>true</interfaceOnly>
...
</configOptions>
which will skip the generation of the SpringBootApplication and the associated Tests.
I've also submitted a ticket to the OpenApi Generator project's github for marking the
I'm writing integration tests for project and I want to merge all database migration scripts into schema.sql before Spring picks it up to populate the database.
For this I use a small class that searches the project for sql files and merges them into one.
I've created a Suite like so:
#RunWith(Suite.class)
#Suite.SuiteClasses({MyTests.class})
public class SuiteTest {
#BeforeClass
public static void setUp() throws IOException {
RunMigrations.mergeMigrations();//this one merges all sqls into one file, called schema.sql
}
}
Then, here's my test:
#RunWith(SpringJUnit4ClassRunner.class)
#SpringBootTest(classes = App.class)
#ActiveProfiles(resolver = CustomActiveProfileResolver.class)
#ContextConfiguration(classes = App.class)
public class MyTests extends AbstractTransactionalJUnit4SpringContextTests {
#PostConstruct
public void before() {
mvc = MockMvcBuilders.webAppContextSetup(context).addFilter(springSecurityFilterChain).build();
}
#Test
#Transactional
public void Test1(){ //do stuff }
}
But this doesn't work as I thought it would. It looks like Spring tries to run schema.sql faster than I'm creating it and fails like this:
Caused by: org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: missing table [application]
If I just turn off my code that generates schema.sql and let Spring run with already created schema then all is good. But if I delete schema.sql and let my class to generate it, then it fails as described.
I've tried to override run(RunNotifier notifier) method in SpringJUnit4ClassRunner and put my migration merger in there, before it calls super.run(notifier) method, but that still doesn't work.
Is there a way to generate that schema.sql before Spring gets its hands on it?
P.S.
I cannot use flyway for production environment. Perhaps it is possible using it just for tests?
UPD:
After some experimenting, I've set this in test.yml:
spring.jpa.hibernate.ddl-auto: none
Now it loads context, performs one test which just gets Oauth2 token and fails with other tests that perform POST and GET requests because it cannot execute #sql annotations that put additional data before test methods. The database seems to be untouched, that is without any tables whatsoever.
You can enable flyway just for tests either by using #TestPropertySource(properties = {"spring.flyway.enabled=true"}) annotation or by creating a test Spring profile with it's own property file. The latter would look like:
#RunWith(SpringRunner.class)
#SpringBootTest
#ActiveProfiles("test")
public MyTest {
with src/test/resources/application-test.yml file:
spring:
flyway:
enabled: true
and flyway-core as a test scoped dependency.
Do note that Flyway properties in Spring Boot were renamed in Spring Boot 2.0.
Perhaps someone may find this useful.
I managed to solve this by using exec-maven-plugin with failsafe plugin.
The test class setup remains the same, I only removed #BeforeClass annotation from Suite.
Here's the POM:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${org.apache.maven.plugins.maven-surefire-plugin-version}</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${org.apache.maven.plugins.maven-failsafe-plugin-version}</version>
<executions>
<execution>
<id>integration-test-for-postgres</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
<execution>
<id>verify-for-postgres</id>
<phase>verify</phase>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${org.codehaus.mojo.exec-maven-plugin-version}</version>
<executions>
<execution>
<id>build-test-environment</id>
<phase>generate-test-resources</phase>
<goals>
<goal>java</goal>
</goals>
</execution>
</executions>
<configuration>
<!--This class prepares the schema.sql file which is fed to Spring to init DB before tests.-->
<mainClass>...GenerateTestDBSchema</mainClass>
<arguments>
<argument>...</argument><!--the migration folder-->
<argument>...</argument><!--The path where to put schema sql-->
</arguments>
</configuration>
</plugin>
</plugins>
</build>
The GenerateTestDBSchema class has main method and uses args array to accept paths where to find migrations and where to put schema.sql.
public static void main(String[] args) {
try {
mergeMigrations(args[0], args[1]);
} catch (IOException e) {
LOG.error(e.getMessage());
}
}
The mergeMigrations() method is straightforward: just take all files from directory, merge them and write to output path. This way Spring has it's schema.sql before context is launched and it itself decides where to run migrations.
Thanks to #ActiveProfiles(resolver = CustomActiveProfileResolver.class) in integration test, spring resolves profiles and picks up application-{profileName}.yml and sets database address automatically.
I would like to ask for your assistance regarding my issue. I have Regression Suite(.xml) that contains 50 test scripts. We are able to execute the regression suite in Jenkins using Maven, but it suddenly hangs/stuck. For example:
Test Case 1-15 are completed, but when in Test Case 16, the build/execution suddenly stuck/hangs on a certain step. I tried to execute the Test Case 16 in my Eclipse to know what step/code has the issue, but it's working fine.
I would like to ask if there is a maven-surefire functionality that when it detects that a test script is stuck/hangs for a certain number of minutes(for example 5 mins), it will fail that test script then proceed to the next one.
Kindly see below my pom.xml
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<target>1.8</target>
<source>1.8</source>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<inherited>true</inherited>
<configuration>
<testFailureIgnore>false</testFailureIgnore>
<suiteXmlFiles>
<suiteXmlFile>${xmlPath}</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
</plugins>
</build>
You can add a timeout to the test annotation in the test itself:
#Test(timeout=600000) //Fails if the method takes longer than 10 minutes
It's in milliseconds so you just need to decide the maximum length of time the test should take and then calculate it from there
EDIT: If you just want to set it once without having to add the timeout to each individual test then you can set a rule in the base class:
import org.junit.Rule;
import org.junit.rules.TestWatcher;
...
#Rule
public TestWatcher watcher = new TestWatcher() {
#Override
public Statement apply(Statement base, Description description) {
if (description.getMethodName().toLowerCase().contains("thisTestTakesLonger")) {
return new FailOnTimeout(base, 600000); //10 minutes = 600000
} else {
return new FailOnTimeout(base, 300000); //5 minutes = 300000
}
}
};
The else is the global timeout, the if is if you have an individual test that takes longer (you may just want the bit in the else if every test is about the same length)
You can get maven to ignore test failure by setting this property to true in pom.xml
<testFailureIgnore>true</testFailureIgnore>
I have configured the maven-failsafe-plugin for excluding/including some testcategories:
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<groups>my.company.SomeImportantCategory</groups>
</configuration>
</plugin>
</plugins>
Now I have a parameterized test which is NOT annotated with SomeImportantCategory:
#RunWith(Parameterized.class)
#Category(AnotherCategory.class)
public class SomeTestIT {
#Parameters(name = "{0}")
public static Collection<TestData[]> setUp() throws Exception {
// Loading Some Excel-Sheets. I'm using github.com/fhm84/jexunit
return doSomethingVeryTimeConsuming("with/this/excel-file.xls");
}
}
Now I run the integration-tests with this profile. Maven does execute the setUp-Method for collecting test-cases.
Do you know how to skip this?
It would be ok for me to access the setUp-Method and do some Java-magic like reading out the included/excluded Groups (how?!?) and skipping doSomethingVeryTimeConsuming using reflection.
This post on javapapers.com shows how to run a JMH benchmark in Maven by typing mvn exec:exec. Running JMH within Maven is pretty handy, since you can easily run it from an Eclipse Run Configuration or even in a Maven phase.
However, there are two problems with this setup:
When you kill Maven, JMH will continue running in the background, as exec:exec starts it in a separate VM.
Usually, JMH will start yet another VM to run the benchmarks, so you will end up with at least 3 VMs running at the same time.
Fortunately, the Exec Maven Plugin comes with a second goal, exec:java, which executes a main class directly in the VM Maven runs. However, when I tried to configure Maven to run JMH using exec:java, the benchmark crashes because of missing classes:
# JMH 1.11.3 (released 40 days ago)
# VM version: Error: Could not find or load main class org.openjdk.jmh.runner.VersionMain
# VM invoker: C:\Program Files\Java\jdk1.7.0\jre\bin\java.exe
[...]
# Run progress: 0.00% complete, ETA 00:02:40
# Fork: 1 of 1
Error: Could not find or load main class org.openjdk.jmh.runner.ForkedMain
<forked VM failed with exit code 1>
Here is the relevant part of the pom.xml:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.4.0</version>
<configuration>
<mainClass>my.Benchmark</mainClass>
</configuration>
</plugin>
And here is how I run JMH from my.Benchmark:
public static void main(String[] args) throws RunnerException {
Options options = new OptionsBuilder().include(my.Benchmark.class.getSimpleName())
.forks(1).build();
new Runner(options).run();
}
I realize that JMH uses the java.class.path system property to determine the classpath for the forked VMs and that this property does not contain Maven's project dependencies. But what is the preferred way to deal with this?
While my previous answer requires modifying the benchmark program, here is a POM-only solution that sets the java.class.path system property to the runtime classpath with the help of the Dependency Plugin:
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>build-classpath</id>
<goals>
<goal>build-classpath</goal>
</goals>
<configuration>
<includeScope>runtime</includeScope>
<outputProperty>depClasspath</outputProperty>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<configuration>
<mainClass>my.Benchmark</mainClass>
<systemProperties>
<systemProperty>
<key>java.class.path</key>
<value>${project.build.outputDirectory}${path.separator}${depClasspath}</value>
</systemProperty>
</systemProperties>
</configuration>
</plugin>
One way to work around this problem is to extract the "effective" classpath from the class loader of the my.Benchmark class before calling JMH from my main method:
URLClassLoader classLoader = (URLClassLoader) my.Benchmark.class.getClassLoader();
StringBuilder classpath = new StringBuilder();
for(URL url : classLoader.getURLs())
classpath.append(url.getPath()).append(File.pathSeparator);
System.setProperty("java.class.path", classpath.toString());
This seems to work, but it feels a lot like a hack that shouldn't be necessary...