I have a Java JUnit Selenium test framework running some tests. There are two classes with two tests each.
I have maven surefire configured like this
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<forkCount>3</forkCount>
<reuseForks>true</reuseForks>
<parallel>methods</parallel>
<threadCount>100</threadCount>
<redirectTestOutputToFile>false</redirectTestOutputToFile>
</configuration>
<version>2.12.4</version>
</plugin>
I'd like it to run 4 tests simultaneously, but no matter what combination of threadCount, parallel and fork settings I use, I can only seem to get 1 class worth of test cases to run at a time. It seems like this should work, can anyone provide a solution?
Do you want to run suites in parallel or methods or tests in parallel?
The only working solution that I found for running Suites in parallel is setting
<property>
<name>suitethreadpoolsize</name>
<value>8</value>
</property>
in the pom.xml. Every other combination did not work as I needed the tests to run on same JVM, not start forked processes.
I use the following configuration for surefire v2.20.1 in maven v3.5.0
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<useUnlimitedThreads>true</useUnlimitedThreads>
<rerunFailingTestsCount>1</rerunFailingTestsCount>
<parallel>methods</parallel>
<forkedProcessExitTimeoutInSeconds>2</forkedProcessExitTimeoutInSeconds>
</configuration>
</plugin>
I believe this is working as our test suite is a lot quicker than it used to be and windows reports the running processes as largely increased specifically when surefire is running.
Probably 1 year too late for you, but just in case it can help someone.
Using parallel=methods like you did will launch all tests (methods) at a time, but 1 class at a time (sequential). So in your example, 2 classes having 2 tests, you will have all tests of of ClassA to execute, then all tests of ClassB.
If you were to use parallel=classes, then all the classes would launch at the same time, but running 1 test (method) at a time (sequential). So in your example, 2 classes having 2 tests, you will have Test1 of ClassA and Test1 of ClassB to start in parallel, and then Test2 of ClassA and Test2 of ClassB to execute afterward.
Since you want all 4 tests to execute in parallel, then use parallel=all.
Both methods and classes will execute in parallel.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
</configuration>
<version>2.22.0</version>
</plugin>
Note : alternatively, you may want to remove the < configuration > block and set them as parameters in your mvn command line.
Ex: mvn clean test -Dparallel=all -DthreadCount=10
Regards,
Related
To tests some scenario i need to run my full application.
i need each test to run in a forked vm in order to have a "clean" vm.
look easy enough with the following:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M7</version>
<configuration>
<forkCount>1</forkCount>
<reuseForks>false</reuseForks>
<parallel>all</parallel>
</configuration>
</plugin>
i added a log to check if the vm is actually forked or not:
log.info("PID:{}", ProcessHandle.current().pid());
for standard junit tests, this is working well.
However for cucumber tests, they are all showing the same PID.
it looks like all the cucumber tests defined from the same runner are considered like a single test for this forked vm feature
this is how i defines the cucumber runner:
#RunWith(Cucumber.class)
#CucumberOptions(
features = "src/test/resources/features/",
plugin = {"json:report.json"},
snippets = CAMELCASE
)
public class RunCucumber {
}
anyone know how to use forked vm with cucumber tests?
if i create a dedicated runner per scenario it works, but this is far from being ideal.
Thanks
I've been able to configure the project to run test cases in parallel using a TestNG runner; however, there are a handful of scenarios that are not very thread safe. If these test cases were to run in parallel, they'd interfere with each other. Now there are a couple of ways I could make these scenarios thread safe, but I was wondering if there was a way to specify these Cucumber scenarios not to run in parallel.
Is there a specific tag I could configure to tag scenarios not to run in parallel? Specify certain feature files not to run in parallel? I believe I might have come across something like that for JUnit 5, but does this exist with TestNG?
Unlike JUnit 5, TestNG does not provide such fine-grained controls. At best you can create multiple runner classes with a different selection of features and different configurations for parallel/serial execution.
Another thing I just realized was I could make the number of threads an argument; with a default of 1 thread.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven.surefire.version}</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<properties>
<property>
<name>dataproviderthreadcount</name>
<value>1</value>
</property>
</properties>
<includes>
<include>api.automation.tests.runners.TestNGRunnerTest</include>
</includes>
</configuration>
</plugin>
I have one Jenkins build that has all the tests tagged with the same name to run in parallel with however many threads needed.
mvn clean test -Dcucumber.filter.tags="${CucumberFilterTag} and not #opentofix" -Dspring.profiles.active=${Environment} -Ddataproviderthreadcount=4
And in the other build, I have a very similar command (with a different tag) just without the added arguement; to use the default of just one thread. This will run all those tagged tests one at a time.
mvn clean test -Dcucumber.filter.tags="${CucumberFilterTag} and not #opentofix" -Dspring.profiles.active=${Environment}
Either way, the first answer still works, but just thought to add this possibility.
No matter what settings I use for Maven + Surefire, I never see more than a single JVM spawned in my process manager. I have a Windows 7 PC with 8 physical cores. I am running the latest Maven / Surefire / JUnit and JDK8 (32-bit).
I tried decomposing all my tests into one test per test class. (I read that JVMs are only spawned for test classes, not test methods.) Still, I never see more than one JVM in process manager.
Ideally, I would like each test class to run a separate JVM -- 8 in parallel (one for each core / do not resuse JVMs).
What are the required Maven Surefire settings?
The following does not work for me:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<includes>
<include>${default.test.suite}</include>
</includes>
<reuseForks>false</reuseForks>
<forkCount>1.0C</forkCount>
<parallel>classes</parallel>
<threadCount>1</threadCount>
<!--
<useUnlimitedThreads>true</useUnlimitedThreads>
<parallelOptimized>false</parallelOptimized>
-->
</configuration>
</plugin>
(I tried various combinations of reuseForks, forkCount, parallel, threadCount, useUnlimitedThreads, and parallelOptimized.)
If you're using JUnit, first of all "parallel" applies only to TestNG, and so do a few other attributes, so they won't be of help / interest to you:
http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html
Here's what works for us in a JUnit setup (doesn't work well for TestNG as just posted here: Running test in parallel with surefire and displaying them properly with the TestNG Jenkins plugin):
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<reuseForks>false</reuseForks>
<forkCount>2.5C</forkCount>
</configuration>
</plugin>
So I think you need to increase your forkCount (with forkCount = 1.0 you won't get parallel stuff going).
For reference, here's a bit more about the small print when it comes to running tests in parallel (incl. forkCount explanation and examples):
http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
I've seen documentation that says you can run TestNG classes/methods/both in parallel. I was curious if there is some configuration (presumably in the maven-surefire-plugin in the pom) where we could something to the effect of
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<parallel>suites</parallel>
</configuration>
</plugin>
If not, are there any clever workarounds to achieve something similar?
Ex:if I have 5 suites (suite1.xml, suite2.xml, suite3.xml etc)
Each of these suites contains 3 classes (suite1 : class1a, class1b, class1c; suite2 : class2a, class2b, etc)
I could technically annotate class1a, class1b, and class1c as a "Group1" group. Then I could do something similar for "Group2" and "Group3". Could I then go ahead try to execute these groups in parallel (achieving a similar effect as running the suites in parallel)?
Alternatively, is there a way to partition the suites in some way to run on different JVM processes by forking the tests?
I'm using JUnit 4.8.1 and Surefire 2.8.1. In my machine I can run one test method only by doing:
$$ mvn clean test -Dtest=TestClass#TestMethod
But when running the same command in the grid configuration, all tests from TestClass are run.
The only difference in the pom is the
<configuration>
<parallel>both</parallel>
</configuration>
configuration in the surefire-plugin.
Any idea, how do I run only one test method in the grid?
possible solution is to use #Categories annotation. But I use simple web driver(not grid).
try step by step set up here
The idea is in following:
you annotate a set of tests you want to exclude and then you show in POM file that you exclude'em in a way somthing like that:
<configuration>
<excludes>
<exclude>blablabla.class</exclude>
</excludes>
</configuration>
Hope this somehow helps you.