One of our classes uses a system property to load a configuration file. In a particular test case, we set this property to an invalid value to check the behavior of the class under test (CUT) in this situation.
The same class is also used transitively in various integration tests. Since we are using JUnit Jupiter's parallel test execution capabilities, we are witnessing race conditions in those integration tests that are using the CUT under the hood. They are failing because the system property is sometimes still invalid.
The parallelism itself is configured globally via the Maven Surefire plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.0</version>
<configuration>
<properties>
<configurationParameters>
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.dynamic.factor=1
</configurationParameters>
</properties>
</configuration>
</plugin>
Consequently, all tests run in parallel per default. The CUT looks like this:
class AttributesProviderTest {
#Test
#ResourceLock( value = SYSTEM_PROPERTIES, mode = READ_WRITE )
void invalid_attributes_file_should_yield_UncheckedIOException() throws Exception {
final Properties backup = new Properties();
backup.putAll( System.getProperties() );
final String attributesFile = "foo";
System.setProperty( AttributesProvider.ATTRIBUTES_FILE_PROPERTY, attributesFile );
assertThatThrownBy( AttributesProvider::getTestInstance )
.isExactlyInstanceOf( UncheckedIOException.class )
.hasMessage( "Cannot read attributes file '" + attributesFile + "'." );
System.setProperties( backup );
}
// Some other tests...
}
As it can be seen, we are trying to synchronize the system property access. However, if I have understood correctly, this only works for other #ResourceLock-annotated tests, and does not magically synchronize system property access in general?
Is there a way to fix the race condition (without annotating all other tests)? Some ideas:
Ensure the CUT is executed sequentially at the beginning (or some other sort of synchronization).
Refactor the CUT and invoke corresponding file-reading method with a parameter directly.
Throw away the test case.
Related
I have below feature files (Separate feature files) in src/test/resources/feature/ and I would like to run them in parallel. Like: One feature file has to execute in chrome and another one has to execute in firefox as mentioned #Tags name.
Feature: Refund item
#chrome
Scenario: Jeff returns a faulty microwave
Given Jeff has bought a microwave for $100
And he has a receipt
When he returns the microwave
Then Jeff should be refunded $100
Feature: Refund Money
#firefox
Scenario: Jeff returns the money
Given Jeff has bought a microwave for $100
And he has a receipt
When he returns the microwave
Then Jeff should be refunded $100
Can somebody assist me to achieve this.I'm using cucumber-java 1.2.2 version, and AbstractTestNGCucumberTests using as runner. Also, let me know how can I create a Test Runner dynamically by using feature files and make them run in parallel.
Update: 4.0.0 version is available at maven central repository with bunch of changes.for more details go here.
Update: 2.2.0 version is available at maven central repository.
You can use opensource plugin cucumber-jvm-parallel-plugin which has many advantages over existing solutions. Available at maven repository
<dependency>
<groupId>com.github.temyers</groupId>
<artifactId>cucumber-jvm-parallel-plugin</artifactId>
<version>2.1.0</version>
</dependency>
First you need to add this plugin with required configuration in your project pom file.
<plugin>
<groupId>com.github.temyers</groupId>
<artifactId>cucumber-jvm-parallel-plugin</artifactId>
<version>2.1.0</version>
<executions>
<execution>
<id>generateRunners</id>
<phase>generate-test-sources</phase>
<goals>
<goal>generateRunners</goal>
</goals>
<configuration>
<!-- Mandatory -->
<!-- comma separated list of package names to scan for glue code -->
<glue>foo, bar</glue>
<outputDirectory>${project.build.directory}/generated-test-sources/cucumber</outputDirectory>
<!-- The directory, which must be in the root of the runtime classpath, containing your feature files. -->
<featuresDirectory>src/test/resources/features/</featuresDirectory>
<!-- Directory where the cucumber report files shall be written -->
<cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
<!-- comma separated list of output formats json,html,rerun.txt -->
<format>json</format>
<!-- CucumberOptions.strict property -->
<strict>true</strict>
<!-- CucumberOptions.monochrome property -->
<monochrome>true</monochrome>
<!-- The tags to run, maps to CucumberOptions.tags property you can pass ANDed tags like "#tag1","#tag2" and ORed tags like "#tag1,#tag2,#tag3" -->
<tags></tags>
<!-- If set to true, only feature files containing the required tags shall be generated. -->
<filterFeaturesByTags>false</filterFeaturesByTags>
<!-- Generate TestNG runners instead of default JUnit ones. -->
<useTestNG>false</useTestNG>
<!-- The naming scheme to use for the generated test classes. One of 'simple' or 'feature-title' -->
<namingScheme>simple</namingScheme>
<!-- The class naming pattern to use. Only required/used if naming scheme is 'pattern'.-->
<namingPattern>Parallel{c}IT</namingPattern>
<!-- One of [SCENARIO, FEATURE]. SCENARIO generates one runner per scenario. FEATURE generates a runner per feature. -->
<parallelScheme>SCENARIO</parallelScheme>
<!-- This is optional, required only if you want to specify a custom template for the generated sources (this is a relative path) -->
<customVmTemplate>src/test/resources/cucumber-custom-runner.vm</customVmTemplate>
</configuration>
</execution>
</executions>
</plugin>
Now add below plugin just below above plugin which will invoke runner classes generated by above plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19</version>
<configuration>
<forkCount>5</forkCount>
<reuseForks>true</reuseForks>
<includes>
<include>**/*IT.class</include>
</includes>
</configuration>
</plugin>
Above two plugins will do magic for cucumber test running in parallel (provided you machine also have advanced hardware support).
Strictly provided <forkCount>n</forkCount> here 'n' is directly proportional to 1) Advanced Hardware support and 2) you available nodes i.e. registered browser instances to HUB.
One major and most important changes is your WebDriver class must be SHARED and you should not implement driver.quit() method, as closing is take care by shutdown hook.
import cucumber.api.Scenario;
import cucumber.api.java.After;
import cucumber.api.java.Before;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebDriverException;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.events.EventFiringWebDriver;
public class SharedDriver extends EventFiringWebDriver {
private static WebDriver REAL_DRIVER = null;
private static final Thread CLOSE_THREAD = new Thread() {
#Override
public void run() {
REAL_DRIVER.close();
}
};
static {
Runtime.getRuntime().addShutdownHook(CLOSE_THREAD);
}
public SharedDriver() {
super(CreateDriver());
}
public static WebDriver CreateDriver() {
WebDriver webDriver;
if (REAL_DRIVER == null)
webDriver = new FirefoxDriver();
setWebDriver(webDriver);
return webDriver;
}
public static void setWebDriver(WebDriver webDriver) {
this.REAL_DRIVER = webDriver;
}
public static WebDriver getWebDriver() {
return this.REAL_DRIVER;
}
#Override
public void close() {
if (Thread.currentThread() != CLOSE_THREAD) {
throw new UnsupportedOperationException("You shouldn't close this WebDriver. It's shared and will close when the JVM exits.");
}
super.close();
}
#Before
public void deleteAllCookies() {
manage().deleteAllCookies();
}
#After
public void embedScreenshot(Scenario scenario) {
try {
byte[] screenshot = getScreenshotAs(OutputType.BYTES);
scenario.embed(screenshot, "image/png");
} catch (WebDriverException somePlatformsDontSupportScreenshots) {
System.err.println(somePlatformsDontSupportScreenshots.getMessage());
}
}
}
Considering you want to execute more than 50 threads i.e. same no of browser instances are registered to HUB but Hub will die if it doesn't get enough memory therefore to avoid this critical situation you should start hub with -DPOOL_MAX=512 (or larger) as stated in grid2 documentation.
Really large (>50 node) Hub installations may need to increase the jetty threads by setting -DPOOL_MAX=512 (or larger) on the java command line.
java -jar selenium-server-standalone-<version>.jar -role hub -DPOOL_MAX=512
Cucumber does not support parallel execution out of the box.
I've tried, but it is not friendly.
We have to use maven's capability to invoke it in parallel. Refer link
Also there is a github project which uses custom plugin to execute in parallel.
Refer cucumber-jvm-parallel-plugin
If all you are expecting is to be able to run multiple features in parallel, then you can try doing the following :
Duplicate the class AbstractTestNGCucumberTests in your test project and set the attribute parallel=true to the #DataProvider annotated method.
Since the default dataprovider-thread-count from TestNG is 10 and now that you have instructed TestNG to run features in parallel, you should start seeing your feature files get executed in parallel.
But I understand that Cucumber reporting is inherently not thread safe, so your reports may appear garbled.
I achieved cucumber parallelism using courgette-jvm . It worked out of the box and run parallel test at scenario level
Simply inlclude similar runner class in cucumber. My tests are further using RemoteWebdriver to open multiple instances on selenium grid. Make sure grid is up and running and node is registered to the grid.
import courgette.api.CourgetteOptions;
import courgette.api.CourgetteRunLevel;
import courgette.api.CucumberOptions;
import courgette.api.testng.TestNGCourgette;
import io.cucumber.testng.AbstractTestNGCucumberTests;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
#Test
#CourgetteOptions(
threads = 10,
runLevel = CourgetteRunLevel.SCENARIO,
rerunFailedScenarios = true,
rerunAttempts = 1,
showTestOutput = true,
reportTitle = "Courgette-JVM Example",
reportTargetDir = "build",
environmentInfo = "browser=chrome; git_branch=master",
cucumberOptions = #CucumberOptions(
features = "src/test/resources/com/test/",
glue = "com.test.stepdefs",
publish = true,
plugin = {
"pretty",
"json:target/cucumber-report/cucumber.json",
"html:target/cucumber-report/cucumber.html"}
))
class AcceptanceIT extends TestNGCourgette {
}
RemoteWebdriver config is
protected RemoteWebDriver createDriver() throws MalformedURLException , IOException {
Properties properties = new Properties();
ClassLoader loader = Thread.currentThread().getContextClassLoader();
String hubURL = "http://192.168.1.7:65299/wd/hub";
System.setProperty("webdriver.gecko.driver", "/Users/amit/Desktop/amit/projects/misc/geckodriver");
FirefoxProfile profile = new FirefoxProfile();
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(FirefoxDriver.PROFILE, profile);
capabilities.setPlatform(Platform.ANY);
FirefoxOptions options = new FirefoxOptions();
options.merge(capabilities);
driver.set(new RemoteWebDriver(new URL(hubURL),options));
return driver.get();
}
To take maximum advantage of TestNG you can use Testng's third party extension QAF framework. It supports multiple bdd syntax including gherkin using GherkinFactory.
While using BDD with QAF, you can take advantage of each TestNG features, including data-providers, parallel execution configuration in different ways (groups/tests/methods), TestNG listeners.
QAF considers each scenario as TestNG test and Scenario Outline as TestNG data-driven test. As qaf provides driver management and resource management in-built, you don't need to write single line of code for driver management or resource management. All you need to do is create TestNG xml configuration file as per your requirement either to run parallel methods (scenarios) or groups or xml test on one or more browser.
It enables different possible configuration combinations. Below is the xml configuration to address this question which will run scenarios in two browsers and in parallel. You can configure number of threads for each browser as standard TestNG xml configuration as well.
<suite name="AUT Test Automation" verbose="0" parallel="tests">
<test name="Test-on-chrome">
<parameter name="scenario.file.loc" value="resources/features" />
<parameter name="driver.name" value="chromeDriver" />
<classes>
<class name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
</classes>
</test>
<test name="Test-on-FF">
<parameter name="scenario.file.loc" value="resources/features" />
<parameter name="driver.name" value="firefoxDriver" />
<classes>
<class name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
</classes>
</test>
</suite>
More over the latest BDDTestFactory2 supports syntax that is derived from QAF BDD, Jbehave and gherkin. It supports meta-data from qaf bdd as tags and examples from gherkin. You can take benefit of inbuilt data-providers to provide test data in XML/JSON/CSV/EXCEL/DB using meta-data in BDD.
EDIT:
Cucumber-5 users can take benefit of qaf-cucumber support library.
Since v4 of Cucumber you can do this without extensions of plugins.
https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-testng#parallel-execution
Cucumber TestNG supports parallel execution of scenarios. Override the
scenarios method to enable parallel execution.
public class RunCucumberTest extends
AbstractTestNGCucumberTests {
#Override
#DataProvider(parallel = true)
public Object[][] scenarios() {
return super.scenarios();
}
}
Maven Surefire plugin configuration for parallel execution
<plugins> <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<properties>
<property>
<name>dataproviderthreadcount</name>
<value>${threadcount}</value>
</property>
</properties>
</configuration>
</plugin>
</plugins>
Where dataproviderthreadcount is the default number of threads to use
for data providers when running tests in parallel.
I'm using Cucumber for my API tests and save the shared context in ScenarioContext class which is marked with #ScenarioScoped and #Injected per Steps class. Of course some features use steps from cross-classes as well (which causes the bug described below, but I dont want to change that due to DRY).
TestS run fine when I do them sequentially however they mix up the token Im saving in scenarioContext when I run them parallelly.
Has anyone faced this issue before? I tried with separate runners and 1 runner however the result was the same. Here's my pom.xml corresponding config:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M7</version>
<configuration>
<includes>
<include>**/*Suite.java</include>
</includes>
<forkCount>8C</forkCount>
<reuseForks>false</reuseForks>
</configuration>
</plugin>
</plugins>
</build>
Above configuration should give the highest level of separation but my tests still fail at random (which does NOT happen with sequential run).
I've read following documentations and didnt find corresponding answers:
https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html
https://maven.apache.org/surefire/maven-surefire-plugin/examples/process-communication.html
https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
#Alexey R. - Im adding additional code :)
So my ScenarioContext class looks as follows:
#ScenarioScoped
public class ScenarioContext extends Base {
Map<String, Object> scenarioContext;
public ScenarioContext() throws Exception {
this.scenarioContext = new HashMap<>();
}
public void setContext(Context key, Object value) {
scenarioContext.put(key.toString(), value);
}
public Object getContext(Context key) {
return scenarioContext.get(key.toString());
}
- some other general shortcut methods
}
And in my Steps classes aforementioned scenarioContext is called as follows:
public class myStepsClass extends Base {
#Inject
private ScenarioContext scenarioContext;
public myStepsClass(ScenarioContext scenarioContext) throws Exception {
this.scenarioContext = scenarioContext;
}
+[gherkin steps getting/setting values to scenarioContext]
I have also added a System.out.println("New scenario Context") in Scenario Context constructor and it seems to be printed out once per thread however I still get different results when tests are run in parallel :(
I'm trying to reconfigure my test results to not show skip results for configuration methods & is skews my data.
I'm running tests through TestNG where each method has a #beforeMethod and #afterMethod configuration method. In the beforeMethod I check whether a #Test method should run or not & if not I throw a SkipException to skip it.
In my current situation I have 2 test methods I run. One is made to fail and the other is designed to be skipped. So I expect to get a result of 1 fail and 1 skip. In the IDE console this is the result I get, but when I run it through Maven I get 1 fail and 3 skips. Here is my emailable-result.html. The failed test case has no #beforeMethod or #afterMethod showing.
I found out about the IConfigurationListener but I am not sure how to use it to remove a configuration method from the report. I am also using maven surefire
This is what I have so far.
public class MyConfigurationListenerAdapter implements IConfigurationListener {
#Override
public void onConfigurationSkip(ITestResult itr) {
String configName = itr.getName();
if (configName.equals("beforeMethod")||configName.equals("afterMethod")){
//TODO remove itr from test result
}
}
}
pom.xml
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.14.1</version>
<configuration>
<properties>
<property>
<name>listener</name>
<value>java_accelerator.testng.classes.MyConfigurationListenerAdapter</value>
</property>
</properties>
<!-- Suite testng xml file to consider for test execution -->
<suiteXmlFiles>
<suiteXmlFile>src/test/java/testclasses/tests/testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
Anyone able to help me from here?
Is it possible to execute a test case parallel with JUnit 5?
I'm looking for something like threadPoolSize and invocationCount from TestNG:
#Test(threadPoolSize = 3, invocationCount = 10, timeOut = 10000)
You can write parallel tests with JUnit 5.3.
https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution
#Execution(ExecutionMode.CONCURRENT)
class MyTest {
#Test
void test1() throws InterruptedException {
Thread.sleep(1000);
System.out.println("Test1 " + Thread.currentThread().getName());
}
#Test
void test2() throws InterruptedException {
Thread.sleep(1000);
System.out.println("Test 2! " + Thread.currentThread().getName());
}
}
// should run simultaneously
Remember to add the junit.jupiter.execution.parallel.enabled=true to your JUnit configuration
https://junit.org/junit5/docs/current/user-guide/#running-tests-config-params
Add this to your JUnit configuration if you need a fixed thread pool:
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=4
As of writing, Parallel test execution is an experimental feature in Junit5:
https://junit.org/junit5/docs/snapshot/user-guide/#writing-tests-parallel-execution
Update the junit-platform.properties file:
Configuration parameters to execute top-level classes in parallel but
methods in same thread:
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent
Configuration parameters to execute top-level classes sequentially but
their methods in parallel:
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = same_thread
The default execution mode is applied to all nodes of the test tree
with a few notable exceptions, namely test classes that use the
Lifecycle.PER_CLASS mode or a MethodOrderer (except for Random). In
the former case, test authors have to ensure that the test class is
thread-safe; in the latter, concurrent execution might conflict with
the configured execution order. Thus, in both cases, test methods in
such test classes are only executed concurrently if the
#Execution(CONCURRENT) annotation is present on the test class or
method.
For Maven:
// JUnit 5
#Execution(ExecutionMode.CONCURRENT)
abstract class BaseTest {
}
// pom.xml
<properties>
<!-- CLI parameters -->
<ignore-failure />
<include-tags />
<exclude-tags />
<parallel-enabled />
<thread-count />
</properties>
<build>
<plugins>
<!-- https://mvnrepository.com/artifact/org.apache.maven.plugins/maven-surefire-plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M8</version>
<configuration>
<testFailureIgnore>${ignore-failure}</testFailureIgnore>
<groups>${include-tags}</groups>
<excludedGroups>${exclude-tags}</excludedGroups>
<properties>
<configurationParameters>
junit.jupiter.execution.parallel.enabled=${parallel-enabled}
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=${thread-count}
junit.jupiter.execution.parallel.config.fixed.max-pool-size=${thread-count}
</configurationParameters>
</properties>
</configuration>
</plugin>
</plugins>
</build>
Consider this:
#Nullable Object obj = null;
Optional<Object> optional = Optional.ofNullable(obj);
This fails because checker-framework assumes ofNullable cannot accept null values (after all, its parameter is not marked as #Nullable).
Is there a good way to tell checker-framework that this method (or other methods in legacy code that I cannot change), accepts #Nullable types everywhere without having to change code everywhere?
EDIT: this answer was based on #mernst help in the comments and in the Checker Framework's Issue tracker
If you, like me, do not want or cannot use the annotated JDK, you will run into this issue.
Note: In most Java shops I've worked, we simply cannot switch which compiler we use or provide a "custom" JDK (that's really unthinkable). For that to be portable, I would have to add the custom JDK to my source repository, for starters, or distribute it to every machine, including CI servers, where the code compiles, and make sure they are in the exact same path across different OS's. Just not cool.
The solution is to provide stub classes and pass them as an argument to the javac process.
This can be done quite easily with whatever tool you use to compile.
For example, with Maven (using the standard compiler plugin):
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<annotationProcessors>
<annotationProcessor>org.checkerframework.checker.nullness.NullnessChecker</annotationProcessor>
</annotationProcessors>
<compilerArgs>
<arg>-Astubs=checkerframework/stubs</arg>
<arg>-AstubWarnIfNotFound</arg>
</compilerArgs>
</configuration>
</plugin>
You also need to add these dependencies to your project:
<dependency>
<groupId>org.checkerframework</groupId>
<artifactId>checker-qual</artifactId>
<version>1.9.2</version>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.checkerframework</groupId>
<artifactId>checker</artifactId>
<version>1.9.2</version>
<optional>true</optional>
</dependency>
Here, checkerframework/stubs is a directory (relative to the location of the pom), containing the stubs. For Optional, my stub looks like this (strangely, stubs must be named *.astub, so this file is called Optional.astub):
package java.util;
import org.checkerframework.checker.interning.qual.*;
import javax.annotation.Nullable;
class Optional<T> {
static <T> Optional<T> ofNullable(#Nullable T value);
#Nullable T orElse(#Nullable T other);
}
This approach is simple, requires little work, does not mess with which compiler I use or the Java libraries at all, makes sure these definitions are only used with the checkerframework (so I can, for example, add this to a Maven profile and enable it only if I want to by simply passing a Maven argument), will work across machines and OS's without previous setup in the true Java way of doing things.
I'm not sure why you say "its parameter is not marked as #Nullable".
When I look at file
checker-framework/checker/jdk/nullness/src/java/util/Optional.java,
I see the following annotated method:
public static <T> Optional<#NonNull T> ofNullable(#Nullable T value) {
return value == null ? empty() : of(value);
}
Furthermore, when I run the Checker Framework on the following code, it issues no warning.
// run like this:
// javacheck -g TestOptional.java -processor nullness
import java.util.*;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.checkerframework.checker.nullness.qual.NonNull;
public class TestOptional {
void m() {
#Nullable Object obj = null;
Optional<Object> optional1 = Optional.ofNullable(obj);
}
}
I'm not sure what is going on in your case because you didn't provide a complete test case, you didn't say what command you ran, and you didn't give an actual error message. (You did provide a diagnosis, but I'm not sure it is accurate.)
Maybe providing more details would enable better understanding of your problem.