Execute multiple TestNG suites in parallel - java

I've seen documentation that says you can run TestNG classes/methods/both in parallel. I was curious if there is some configuration (presumably in the maven-surefire-plugin in the pom) where we could something to the effect of
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<parallel>suites</parallel>
</configuration>
</plugin>
If not, are there any clever workarounds to achieve something similar?
Ex:if I have 5 suites (suite1.xml, suite2.xml, suite3.xml etc)
Each of these suites contains 3 classes (suite1 : class1a, class1b, class1c; suite2 : class2a, class2b, etc)
I could technically annotate class1a, class1b, and class1c as a "Group1" group. Then I could do something similar for "Group2" and "Group3". Could I then go ahead try to execute these groups in parallel (achieving a similar effect as running the suites in parallel)?
Alternatively, is there a way to partition the suites in some way to run on different JVM processes by forking the tests?

Related

How to use a forked vm for each cucumber scenario/feature using maven-surefire-plugin

To tests some scenario i need to run my full application.
i need each test to run in a forked vm in order to have a "clean" vm.
look easy enough with the following:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M7</version>
<configuration>
<forkCount>1</forkCount>
<reuseForks>false</reuseForks>
<parallel>all</parallel>
</configuration>
</plugin>
i added a log to check if the vm is actually forked or not:
log.info("PID:{}", ProcessHandle.current().pid());
for standard junit tests, this is working well.
However for cucumber tests, they are all showing the same PID.
it looks like all the cucumber tests defined from the same runner are considered like a single test for this forked vm feature
this is how i defines the cucumber runner:
#RunWith(Cucumber.class)
#CucumberOptions(
features = "src/test/resources/features/",
plugin = {"json:report.json"},
snippets = CAMELCASE
)
public class RunCucumber {
}
anyone know how to use forked vm with cucumber tests?
if i create a dedicated runner per scenario it works, but this is far from being ideal.
Thanks

Specify test case not to run in parallel with TestNG runner in Spring using Cucumber

I've been able to configure the project to run test cases in parallel using a TestNG runner; however, there are a handful of scenarios that are not very thread safe. If these test cases were to run in parallel, they'd interfere with each other. Now there are a couple of ways I could make these scenarios thread safe, but I was wondering if there was a way to specify these Cucumber scenarios not to run in parallel.
Is there a specific tag I could configure to tag scenarios not to run in parallel? Specify certain feature files not to run in parallel? I believe I might have come across something like that for JUnit 5, but does this exist with TestNG?
Unlike JUnit 5, TestNG does not provide such fine-grained controls. At best you can create multiple runner classes with a different selection of features and different configurations for parallel/serial execution.
Another thing I just realized was I could make the number of threads an argument; with a default of 1 thread.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven.surefire.version}</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<properties>
<property>
<name>dataproviderthreadcount</name>
<value>1</value>
</property>
</properties>
<includes>
<include>api.automation.tests.runners.TestNGRunnerTest</include>
</includes>
</configuration>
</plugin>
I have one Jenkins build that has all the tests tagged with the same name to run in parallel with however many threads needed.
mvn clean test -Dcucumber.filter.tags="${CucumberFilterTag} and not #opentofix" -Dspring.profiles.active=${Environment} -Ddataproviderthreadcount=4
And in the other build, I have a very similar command (with a different tag) just without the added arguement; to use the default of just one thread. This will run all those tagged tests one at a time.
mvn clean test -Dcucumber.filter.tags="${CucumberFilterTag} and not #opentofix" -Dspring.profiles.active=${Environment}
Either way, the first answer still works, but just thought to add this possibility.

How do I get Maven Surefire to run all classes and tests?

I have a Java JUnit Selenium test framework running some tests. There are two classes with two tests each.
I have maven surefire configured like this
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<forkCount>3</forkCount>
<reuseForks>true</reuseForks>
<parallel>methods</parallel>
<threadCount>100</threadCount>
<redirectTestOutputToFile>false</redirectTestOutputToFile>
</configuration>
<version>2.12.4</version>
</plugin>
I'd like it to run 4 tests simultaneously, but no matter what combination of threadCount, parallel and fork settings I use, I can only seem to get 1 class worth of test cases to run at a time. It seems like this should work, can anyone provide a solution?
Do you want to run suites in parallel or methods or tests in parallel?
The only working solution that I found for running Suites in parallel is setting
<property>
<name>suitethreadpoolsize</name>
<value>8</value>
</property>
in the pom.xml. Every other combination did not work as I needed the tests to run on same JVM, not start forked processes.
I use the following configuration for surefire v2.20.1 in maven v3.5.0
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<useUnlimitedThreads>true</useUnlimitedThreads>
<rerunFailingTestsCount>1</rerunFailingTestsCount>
<parallel>methods</parallel>
<forkedProcessExitTimeoutInSeconds>2</forkedProcessExitTimeoutInSeconds>
</configuration>
</plugin>
I believe this is working as our test suite is a lot quicker than it used to be and windows reports the running processes as largely increased specifically when surefire is running.
Probably 1 year too late for you, but just in case it can help someone.
Using parallel=methods like you did will launch all tests (methods) at a time, but 1 class at a time (sequential). So in your example, 2 classes having 2 tests, you will have all tests of of ClassA to execute, then all tests of ClassB.
If you were to use parallel=classes, then all the classes would launch at the same time, but running 1 test (method) at a time (sequential). So in your example, 2 classes having 2 tests, you will have Test1 of ClassA and Test1 of ClassB to start in parallel, and then Test2 of ClassA and Test2 of ClassB to execute afterward.
Since you want all 4 tests to execute in parallel, then use parallel=all.
Both methods and classes will execute in parallel.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
</configuration>
<version>2.22.0</version>
</plugin>
Note : alternatively, you may want to remove the < configuration > block and set them as parameters in your mvn command line.
Ex: mvn clean test -Dparallel=all -DthreadCount=10
Regards,

How to run Maven surefire where each test class has its own parallel JVM?

No matter what settings I use for Maven + Surefire, I never see more than a single JVM spawned in my process manager. I have a Windows 7 PC with 8 physical cores. I am running the latest Maven / Surefire / JUnit and JDK8 (32-bit).
I tried decomposing all my tests into one test per test class. (I read that JVMs are only spawned for test classes, not test methods.) Still, I never see more than one JVM in process manager.
Ideally, I would like each test class to run a separate JVM -- 8 in parallel (one for each core / do not resuse JVMs).
What are the required Maven Surefire settings?
The following does not work for me:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<includes>
<include>${default.test.suite}</include>
</includes>
<reuseForks>false</reuseForks>
<forkCount>1.0C</forkCount>
<parallel>classes</parallel>
<threadCount>1</threadCount>
<!--
<useUnlimitedThreads>true</useUnlimitedThreads>
<parallelOptimized>false</parallelOptimized>
-->
</configuration>
</plugin>
(I tried various combinations of reuseForks, forkCount, parallel, threadCount, useUnlimitedThreads, and parallelOptimized.)
If you're using JUnit, first of all "parallel" applies only to TestNG, and so do a few other attributes, so they won't be of help / interest to you:
http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html
Here's what works for us in a JUnit setup (doesn't work well for TestNG as just posted here: Running test in parallel with surefire and displaying them properly with the TestNG Jenkins plugin):
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<reuseForks>false</reuseForks>
<forkCount>2.5C</forkCount>
</configuration>
</plugin>
So I think you need to increase your forkCount (with forkCount = 1.0 you won't get parallel stuff going).
For reference, here's a bit more about the small print when it comes to running tests in parallel (incl. forkCount explanation and examples):
http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

Maven and code metrics: check for existence of a test case for each class

Is there something that can be used in Maven to automate this kind of check? I'm seeing checkstyle and PMD but I'm not finding this feature.
Basically I'd like the build to fail if there's a class A and there's not an ATestCase. I know, it is not a strict check and can be easily bypassed by creating just the class, but at the moment that would be enough.
What ou are looking for
As Jens Piegsa pointed id out, what you are looking for is a tool that show you the test coverage, in other words the percentage of code which is used by you tests.
It allow you to see how much you code is tested, in a really more reliable way than (at least test by class).
You can use Cobertura, which well integrated in Maven: http://mojo.codehaus.org/cobertura-maven-plugin/
The way to achieve that
POM Configuration
Just put this code snippet in your pom.xml
<project>
...
<reporting>
<plugins>
...
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<version>2.6</version>
</plugin>
</plugins>
</reporting>
</project>
Running coverage
And run
mvn cobertura:cobertura
Or run the report phase (binded with site generation)
mvn site:site
Adding quality Threshold
You can even add failing threshold if you want to invalidate low coverage builds
<plugin>
[...]
<configuration>
<check>
<!-- Fail if code coverage does not respects the goals -->
<haltOnFailure>true</haltOnFailure>
<!-- Per-class thresholds -->
<lineRate>80</lineRate>
<!-- Per-branch thresholds (in a if verify that if and else are covered-->
<branchRate>80</branchRate>
<!-- Project-wide thresholds -->
<totalLineRate>90</totalLineRate>
<totalBranchRate>90</totalBranchRate>
</check>
</configuration>
</plugin>
Short answer: No.
Longer answer: I once wrote a unit test to assert that all VO's had a no-args constructor, and I would think that you could use the same approach here.
Basically, iterate through Package.getPackages() (you'll need to filter out JRE packages, but assuming you're using a sensible namespace, this should be no problem). For each package gather all classes not starting or ending with Test and assert that each one has a matching test.
It's not failsafe, but close enough perhaps?
Cheers,

Categories

Resources