Develop Reference

ScenarioScoped mixes up token with SureFire parallel execution (Java + Maven)

I'm using Cucumber for my API tests and save the shared context in ScenarioContext class which is marked with #ScenarioScoped and #Injected per Steps class. Of course some features use steps from cross-classes as well (which causes the bug described below, but I dont want to change that due to DRY).
TestS run fine when I do them sequentially however they mix up the token Im saving in scenarioContext when I run them parallelly.
Has anyone faced this issue before? I tried with separate runners and 1 runner however the result was the same. Here's my pom.xml corresponding config:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M7</version>
<configuration>
<includes>
<include>**/*Suite.java</include>
</includes>
<forkCount>8C</forkCount>
<reuseForks>false</reuseForks>
</configuration>
</plugin>
</plugins>
</build>
Above configuration should give the highest level of separation but my tests still fail at random (which does NOT happen with sequential run).
I've read following documentations and didnt find corresponding answers:
https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html
https://maven.apache.org/surefire/maven-surefire-plugin/examples/process-communication.html
https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
#Alexey R. - Im adding additional code :)
So my ScenarioContext class looks as follows:
#ScenarioScoped
public class ScenarioContext extends Base {
Map<String, Object> scenarioContext;
public ScenarioContext() throws Exception {
this.scenarioContext = new HashMap<>();
}
public void setContext(Context key, Object value) {
scenarioContext.put(key.toString(), value);
}
public Object getContext(Context key) {
return scenarioContext.get(key.toString());
}
- some other general shortcut methods
}
And in my Steps classes aforementioned scenarioContext is called as follows:
public class myStepsClass extends Base {
#Inject
private ScenarioContext scenarioContext;
public myStepsClass(ScenarioContext scenarioContext) throws Exception {
this.scenarioContext = scenarioContext;
}
+[gherkin steps getting/setting values to scenarioContext]
I have also added a System.out.println("New scenario Context") in Scenario Context constructor and it seems to be printed out once per thread however I still get different results when tests are run in parallel :(

Correctly include jUnit5 #Nested classes when running single class in maven

The #Nested classes that are executed in JUnit 5 they are ordered to run AFTER all the tests in eclosing class. How could I enforce the same behavior using maven, if my goal is to run a single enclosing class and it's nested classes? Is there a commandline or pom.xml modification to make this example test pass?
package example;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
public class SomeTopLevelTest {
private static boolean nestedDone = false;
#Test
void test1() {
Assertions.assertFalse(nestedDone,
"Nested classes should execute after the enclosing tests");
}
#Nested
class SomeNestedTest {
#Test
void test2() {
nestedDone = true;
}
}
#AfterAll
static void recheck() {
Assertions.assertTrue(nestedDone, "Nested class should be executed");
}
}
This does pass in IDE:
But does not in commanline, if I try to specify the name:
mvn test -Dtest=example.SomeTopLevelTest
[ERROR] Failures:
[ERROR] SomeTopLevelTest.recheck:27 Nested class should be executed ==> expected: <true> but was: <false>
mvn test -Dtest=example.SomeTopLevelTest*
[ERROR] Failures:
[ERROR] SomeTopLevelTest.test1:14 Nested classes should execute after the enclosing tests ==> expected: <false> but was: <true>

The problem of #Nested classes not executing is a known issue and it has been reported in both JUnit5 and Surefire issue trackers, but as of now remains unresolved.
The current state of affairs (tested with Maven 3.6.3, Junit5 5.7.2, Surefire 2.22.2 up to 3.0.0-M5):
A. Not selecting a test
mvn test
Results in executing all test classes as expected: methods from enclosing classes first, and then methods from #Nested classes, level by level
B. Selecting test the standard way
mvn test -Dtest=example.SomeTopLevelTest
This apparently triggers the default surefire excludes that use the following pattern:
<excludes>
<exclude>**/*$*</exclude>
</excludes>
Why does it not happen in case A is a mystery, but one can override this behaviour by explicitly clearing the excludes pattern:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludes>
<exclude/>
</excludes>
</configuration>
</plugin>
It does not seem to be possible to do without modyfing the pom.xml.
This DOES NOT solve the issue as posted in this question, because the nested classes are still executed first.
C. Using wildcard with -Dtest parameter
mvn test -Dtest=example.SomeTopLevelTest*
This explicitly selects all the nested classes, but - as stated in the question - results in executing the nested classes first, so it's not a solution.
D. Using includes
mvn test -DtestName=example.SomeTopLevelTest
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>${testName}</include>
</includes>
</configuration>
</plugin>
Apparently include patterns work quite differently than -Dtest parameter, because this is finally the solution to pass the test from the question. With this setup the testName may be a single class, wildcard pattern or regex
example.SomeTopLevelTest to execute all test methods in single class
example/* - all tests (including nested) in package example, but not sub-packages
example/** - all tests in package and subpackages
advanced regex, is supprted too

Junit5 Tag Annotation together with Quarkus QuarkusTestResource Annotation

How can QuarkusTestResource be used in conjunction with Tag Annotation?
Example Test Routine
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
#QuarkusTest
#Tag("integration")
#QuarkusTestResource(DatabaseResource.class)
public class MyTest {
#Test
public void () {
doTests...
}
}
Maven Snippet:
<quarkus-plugin.version>1.12.1.Final</quarkus-plugin.version>
<quarkus.platform.artifact-id>quarkus-universe-bom</quarkus.platform.artifact-id>
<quarkus.platform.group-id>io.quarkus</quarkus.platform.group-id>
<quarkus.platform.version>1.12.1.Final</quarkus.platform.version>
<surefire-plugin.version>2.22.1</surefire-plugin.version>
<testscope>unit</testscope>
...
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire-plugin.version}</version>
<configuration>
<groups>${testscope}</groups>
<systemPropertyVariables>
<java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
<maven.home>${maven.home}</maven.home>
</systemPropertyVariables>
</configuration>
</plugin>
...
Maven Command:
./mvnw clean test
Result: QuarkusTestResource are started even if no QuarkusTest is annotated with "unit", so it seems that Quarkus is not aware of the Tag Annotation?

Quarkus Test Resources are global which means it will run anyway, even if your class is annotated with a tag that shouldn't run.
To prevent this, try annotating your class with
#QuarkusTestResource(restrictToAnnotatedClass = true)
From the Quarkus' website
test resources are global, even if they are defined on a test class or custom profile, which means they will all be activated for all tests, even though we do remove duplicates. If you want to only enable a test resource on a single test class or test profile, you can use #QuarkusTestResource(restrictToAnnotatedClass = true).

What you are looking for is likely the tags methof of QuarkusTestProfile. See this part of the documentation

Parallel test case execution with JUnit 5

Is it possible to execute a test case parallel with JUnit 5?
I'm looking for something like threadPoolSize and invocationCount from TestNG:
#Test(threadPoolSize = 3, invocationCount = 10, timeOut = 10000)

You can write parallel tests with JUnit 5.3.
https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution
#Execution(ExecutionMode.CONCURRENT)
class MyTest {
#Test
void test1() throws InterruptedException {
Thread.sleep(1000);
System.out.println("Test1 " + Thread.currentThread().getName());
}
#Test
void test2() throws InterruptedException {
Thread.sleep(1000);
System.out.println("Test 2! " + Thread.currentThread().getName());
}
}
// should run simultaneously
Remember to add the junit.jupiter.execution.parallel.enabled=true to your JUnit configuration
https://junit.org/junit5/docs/current/user-guide/#running-tests-config-params
Add this to your JUnit configuration if you need a fixed thread pool:
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=4

As of writing, Parallel test execution is an experimental feature in Junit5:
https://junit.org/junit5/docs/snapshot/user-guide/#writing-tests-parallel-execution
Update the junit-platform.properties file:
Configuration parameters to execute top-level classes in parallel but
methods in same thread:
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent
Configuration parameters to execute top-level classes sequentially but
their methods in parallel:
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = same_thread
The default execution mode is applied to all nodes of the test tree
with a few notable exceptions, namely test classes that use the
Lifecycle.PER_CLASS mode or a MethodOrderer (except for Random). In
the former case, test authors have to ensure that the test class is
thread-safe; in the latter, concurrent execution might conflict with
the configured execution order. Thus, in both cases, test methods in
such test classes are only executed concurrently if the
#Execution(CONCURRENT) annotation is present on the test class or
method.

For Maven:
// JUnit 5
#Execution(ExecutionMode.CONCURRENT)
abstract class BaseTest {
}
// pom.xml
<properties>
<!-- CLI parameters -->
<ignore-failure />
<include-tags />
<exclude-tags />
<parallel-enabled />
<thread-count />
</properties>
<build>
<plugins>
<!-- https://mvnrepository.com/artifact/org.apache.maven.plugins/maven-surefire-plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M8</version>
<configuration>
<testFailureIgnore>${ignore-failure}</testFailureIgnore>
<groups>${include-tags}</groups>
<excludedGroups>${exclude-tags}</excludedGroups>
<properties>
<configurationParameters>
junit.jupiter.execution.parallel.enabled=${parallel-enabled}
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=${thread-count}
junit.jupiter.execution.parallel.config.fixed.max-pool-size=${thread-count}
</configurationParameters>
</properties>
</configuration>
</plugin>
</plugins>
</build>

JUnit 4 - How to ignore all package with the tests?

I run my JUnit and Mockito tests in a big project. I use them for testing my server-side components that connect to web-service. All these connections require some time and it is not neccessary for them to be executed during the build.
I would like that my tests would be ignored during the build.
I have about 10 classes with tests. So the obvious way is to annotate all the classes with #Ignore. However I should do this every time I commit my code to the project and then re-annotate all tests. Not the very best solution I think.
So is this possible somehow simply ignore all package (let say com.example.tests) with the tests?
Or what might be the solution to ignore tests in the build in a simple way?

You can mention on your build.gradle what packages to exclude from tests
test {
exclude '**/*IntegrationTest*'
}
same for maven:
must consider this notation:
By default, the Surefire Plugin will automatically include all test classes with the following wildcard patterns:
"**/Test*.java" - includes all of its subdirectories and all Java filenames that start with "Test".
"**/*Test.java" - includes all of its subdirectories and all Java filenames that end with "Test".
"**/*Tests.java" - includes all of its subdirectories and all Java filenames that end with "Tests".
"**/*TestCase.java" - includes all of its subdirectories and all Java filenames that end with "TestCase".
<project>
[...]
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20</version>
<configuration>
<excludes>
<exclude>*com.example.tests*/*Test.java</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
[...]
</project>
Another option is the old
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
or even when call it
mvn install -DskipTests

Using Categories seems to be an option that can come in handy
This is how you may add these to your Gradle script.
test {
useJUnit {
includeCategories 'org.gradle.junit.CategoryA'
excludeCategories 'org.gradle.junit.CategoryB'
}
}
A sample can be found here, adding it for a quick reference.
public interface FastTests
{
/* category marker */
}
public interface SlowTests
{
/* category marker */
}
public class A
{
#Category(SlowTests.class)
#Test public void a()
{
}
}
#Category(FastTests.class})
public class B
{
#Test public void b()
{
}
}
#RunWith(Categories.class)
#IncludeCategory(SlowTests.class)
#ExcludeCategory(FastTests.class)
#SuiteClasses({ A.class, B.class })
public class SlowTestSuite
{
}

I have found the solution for my case.
To disable all the tests during the build or even in any other context that you want the Spring annotation #IfProfileValue can be used. All tests with this annotation will be executed only in wanted context.
The example is this:
#IfProfileValue(name="enableTests", value="true")
public class DemoApplicationTests {
#Test
public void contextLoads() {
...
}
}
In my IDE I can edit the configuration and set the variable by:
-DenableTests=true
This annotation can be used on the level of a class or on the level of a test also.
All classes or tests annotated with such #IfProfileValue will be executed only in my environment and will be ignored during the build.
This approach is the best for me because it is not convenient in my project to change main pom.xml for my own test needs.
Addition.
Also in Spring or Spring Boot you should add Runner.
For example in Spring:
#RunWith(SpringJUnit4ClassRunner.class)
#IfProfileValue(name="enableTests", value="true")
#ContextConfiguration(classes = { YourClassConfig.class })
YourClassConfig might be empty:
#Configuration
public class YourClassConfig {
}