I have a testsuite say JunitTest1 and JunitTest2 as below,
#RunWith(Suite.class)
#SuiteClasses({
com.sample.test1.class,
com.sample.test2.class,
com.sample.test3.class,
})
public class JunitTest1 {
}
#RunWith(Suite.class)
#SuiteClasses({
com.sample.xxx1.class,
com.sample.xxx2.class,
com.sample.xxx3.class,
})
public class JunitTest2 {
}
I want to run both the testsuite in parallel but the test class inside the testsuite should run in the specified order. I have added the below plugin in maven pom.xml,
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20.1</version>
<configuration>
<parallel>suites</parallel>
<threadCount>2</threadCount>
<!-- <forkCount>1</forkCount>
<reuseForks>true</reuseForks> -->
</configuration>
</plugin>
I have followed this link - http://maven.apache.org/surefire/maven-surefire-plugin/examples/junit.html
I have tried all the possible combination for parallel tag like methods, classes, both, suites, suitesAndClasses, suitesAndMethods, classesAndMethods, all but nothing seems to solve my problem. The test suite are still running in sequential if i use suites and for other options the test classes in test suite are running in random order
Is there anything i am doing it wrong? can anyone please guide me to solve this issue ?
junit tests should be independent of one-another. junit does not provide a means to order tests to prevent you from writing tests which are dependent on other tests.
There are other SO threads on this topic:
How to run test methods in order with Junit
and
http://stackoverflow.com/questions/3693626/how-to-run-test-methods-in-specific-order-in-junit4
Related
To tests some scenario i need to run my full application.
i need each test to run in a forked vm in order to have a "clean" vm.
look easy enough with the following:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M7</version>
<configuration>
<forkCount>1</forkCount>
<reuseForks>false</reuseForks>
<parallel>all</parallel>
</configuration>
</plugin>
i added a log to check if the vm is actually forked or not:
log.info("PID:{}", ProcessHandle.current().pid());
for standard junit tests, this is working well.
However for cucumber tests, they are all showing the same PID.
it looks like all the cucumber tests defined from the same runner are considered like a single test for this forked vm feature
this is how i defines the cucumber runner:
#RunWith(Cucumber.class)
#CucumberOptions(
features = "src/test/resources/features/",
plugin = {"json:report.json"},
snippets = CAMELCASE
)
public class RunCucumber {
}
anyone know how to use forked vm with cucumber tests?
if i create a dedicated runner per scenario it works, but this is far from being ideal.
Thanks
I have a Java JUnit Selenium test framework running some tests. There are two classes with two tests each.
I have maven surefire configured like this
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<forkCount>3</forkCount>
<reuseForks>true</reuseForks>
<parallel>methods</parallel>
<threadCount>100</threadCount>
<redirectTestOutputToFile>false</redirectTestOutputToFile>
</configuration>
<version>2.12.4</version>
</plugin>
I'd like it to run 4 tests simultaneously, but no matter what combination of threadCount, parallel and fork settings I use, I can only seem to get 1 class worth of test cases to run at a time. It seems like this should work, can anyone provide a solution?
Do you want to run suites in parallel or methods or tests in parallel?
The only working solution that I found for running Suites in parallel is setting
<property>
<name>suitethreadpoolsize</name>
<value>8</value>
</property>
in the pom.xml. Every other combination did not work as I needed the tests to run on same JVM, not start forked processes.
I use the following configuration for surefire v2.20.1 in maven v3.5.0
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<useUnlimitedThreads>true</useUnlimitedThreads>
<rerunFailingTestsCount>1</rerunFailingTestsCount>
<parallel>methods</parallel>
<forkedProcessExitTimeoutInSeconds>2</forkedProcessExitTimeoutInSeconds>
</configuration>
</plugin>
I believe this is working as our test suite is a lot quicker than it used to be and windows reports the running processes as largely increased specifically when surefire is running.
Probably 1 year too late for you, but just in case it can help someone.
Using parallel=methods like you did will launch all tests (methods) at a time, but 1 class at a time (sequential). So in your example, 2 classes having 2 tests, you will have all tests of of ClassA to execute, then all tests of ClassB.
If you were to use parallel=classes, then all the classes would launch at the same time, but running 1 test (method) at a time (sequential). So in your example, 2 classes having 2 tests, you will have Test1 of ClassA and Test1 of ClassB to start in parallel, and then Test2 of ClassA and Test2 of ClassB to execute afterward.
Since you want all 4 tests to execute in parallel, then use parallel=all.
Both methods and classes will execute in parallel.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
</configuration>
<version>2.22.0</version>
</plugin>
Note : alternatively, you may want to remove the < configuration > block and set them as parameters in your mvn command line.
Ex: mvn clean test -Dparallel=all -DthreadCount=10
Regards,
Using a Spring maven context, I would like to run specific tests based on a maven profile. I would like to have an easy way of tagging the test groups. If possible I would like to use the annotations.
Which options are there, like maven command line parameters, maven profiles specification, etc.
Say I have the following tests:
Example:
// annotation("integration")
public class GeopointFormatterTest {
#Test
public void testIntegration1() { ... }
#Test
public void testIntegration2() { ... }
Annotations like #Profile (which is for creating beans) and #ActiveProfile (which is for selecting specific profiles for creating beans) cannot be used for selecting tests, of course. All tests just run for statements like:
mvn clean install -Pdevelopment
mvn clean install -Pdevelopment -Dspring.profiles.active=acceptance
mvn clean install -Pdevelopment -Dspring.profiles.active=integration
As suggested, I used also #IfProfileValue. This is a good way for selecting tests based on system property values. System property values can be overruled by a CustomProfileValueSource class, like in: #ProfileValueSourceConfiguration(CustomProfileValueSource.class)
EDIT and ALTERNATIVE
The GREAT answers below focus on JUnit's #Category mechanism. Thanks to all!
A different approach is via these steps: [1] set a property within a maven profile and [2] use the property to skip tests via the of the standard surefire test plugin.
[1] Setting the properties via a profile:
<profiles>
<profile>
<id>integrationtests</id>
<properties>
<integration.skip>false</integration.skip>
<acceptance.skip>true</acceptance.skip>
</properties>
</profile>
... other profiles
[2] Using the properties in the surefire test plugin to skip tests.
<build>
<plugins>
<plugin>
<!-- Run the integration test-->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire.plugin.version}</version>
<configuration>
<skipTests>${acceptance.skip}</skipTests>
Start in maven: mvn clean install –Pintegrationtests
Take a look at junit categories.
You'd tag your tests with specific category annotations
public interface FastTests { /* category marker */ }
public interface SlowTests { /* category marker */ }
#Category(SlowTests.class)
public class A {
#Test public void a() {}
}
then form a suite like
#RunWith(Categories.class)
#IncludeCategory({FastTests.class})
#SuiteClasses({A.class, B.class})
public static class FastTestSuite {
//
}
And then run it with
mvn -Dtest=FastTestSuite test
Note also that if you don't want to manually specify your unit test case classes in the suite class, you can also use the help of ClasspathSuite and then just limit based on categories.
You will probably need to categorize your tests using the #Category annotation. A complete example has been provided in the Surefire documentation provided here - search for the string Using JUnit Categories.
Assuming that you have categorized your tests accordingly, you will now be able to setup one or more profiles in your maven build which will trigger these tests as per the category
<profiles>
<profile>
<id>slow-tests</id>
<build>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
<configuration>
<groups>com.mycompany.SlowTests</groups>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>fast-tests</id>
<build>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
<configuration>
<groups>com.mycompany.FastTests</groups>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
You can specify one or more profiles on the command line when running the tests.
mvn test -Pslow-tests,fast-tests
You can specify the profile with this flag:
mvn test -Dspring.profiles.active=acceptance
In my latest project I have an "integration" profile that I use to run the integration tests against an embedded H2 database.
We did solved categorization of junit in the following steps.
I did create a project for you in github.
https://github.com/djaganathan/unit-test-samples
Caveat:- Junit Categorization packages still says as experimental.
1) Created a category of interfaces
/**
* This interface used to categories Junit Test
* those Tests will be executed during bamboo build run
*/
public interface ReleaseTest {
}
2) Tagged the unit testcases with the category you want
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.internal.runners.JUnit4ClassRunner;
import org.junit.runner.RunWith;
import com.github.djaganathan.unit.test.category.ReleaseTest;
#RunWith(JUnit4ClassRunner.class)
public class GeneralTest {
#Test
#Category(value={ReleaseTest.class})
public void doTestForRelease(){}
#Test
public void doTestForDev(){}
}
3) Create profile in maven and attach to it
4) Run the command as mvn test -PreleaseTest
Based on the answers of a number of posts, I created a simple test project that demonstrates a number of code quality & testing features:
Performing either unit tests OR integration tests with plugins Surefire and Failsafe.
Improving code quality via the plugin findbugs.
Determine test coverage stats with plugin Jacoco.
Enjoy!
I have TestNG tests that are grouped with the (groups = "unit") annotation as unit or integration and run using the following Maven config:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.7.2</version>
<configuration>
<groups>${test.group}</groups>
<includes>
<include>**/*Test.java</include>
<include>**/Test*.java</include>
<include>**/When*.java</include>
</includes>
<argLine>${test.args.unit} -Xmx1024m</argLine>
</configuration>
</plugin>
I would like to separate test groups by directory structure instead of annotations and place the unit tests under a directory called unit.
So for example a unit tests in: src/test/java/foo/FooTests/MyFoo.java
I want to move it to: src/test/java/foo/unit/FooTests/MyFoo.java.
Then change the Maven config to pick up Java files under a unit subdir for unit tests.
I tried doing this by changing the plugin config to use the following include:
<groups>${test.group}</groups>
<includes>
<include>**/unit/**/*Test.java</include>
<include>**/unit/**/Test*.java</include>
<include>**/unit/**/When*.java</include>
</includes>
But that doesn't seem to work. Any ideas?
The first thing is that you should use a more up-to-date maven-surefire-plugin (2.12.4) and furthermore dont use includes, cause maven-surefire-plugin has defaults for Unit tests. If you have integration tests better use maven-failsafe-plugin and name them appropriately. If you use TestNG in combination it sounds more like integration tests.
If you name your Unit testes like '*Test' and your Integration Tests as '*IT' then the plugins will pick the right tests.
I'm using JUnit 4.8.1 and Surefire 2.8.1. In my machine I can run one test method only by doing:
$$ mvn clean test -Dtest=TestClass#TestMethod
But when running the same command in the grid configuration, all tests from TestClass are run.
The only difference in the pom is the
<configuration>
<parallel>both</parallel>
</configuration>
configuration in the surefire-plugin.
Any idea, how do I run only one test method in the grid?
possible solution is to use #Categories annotation. But I use simple web driver(not grid).
try step by step set up here
The idea is in following:
you annotate a set of tests you want to exclude and then you show in POM file that you exclude'em in a way somthing like that:
<configuration>
<excludes>
<exclude>blablabla.class</exclude>
</excludes>
</configuration>
Hope this somehow helps you.