I run my JUnit and Mockito tests in a big project. I use them for testing my server-side components that connect to web-service. All these connections require some time and it is not neccessary for them to be executed during the build.
I would like that my tests would be ignored during the build.
I have about 10 classes with tests. So the obvious way is to annotate all the classes with #Ignore. However I should do this every time I commit my code to the project and then re-annotate all tests. Not the very best solution I think.
So is this possible somehow simply ignore all package (let say com.example.tests) with the tests?
Or what might be the solution to ignore tests in the build in a simple way?
You can mention on your build.gradle what packages to exclude from tests
test {
exclude '**/*IntegrationTest*'
}
same for maven:
must consider this notation:
By default, the Surefire Plugin will automatically include all test classes with the following wildcard patterns:
"**/Test*.java" - includes all of its subdirectories and all Java filenames that start with "Test".
"**/*Test.java" - includes all of its subdirectories and all Java filenames that end with "Test".
"**/*Tests.java" - includes all of its subdirectories and all Java filenames that end with "Tests".
"**/*TestCase.java" - includes all of its subdirectories and all Java filenames that end with "TestCase".
<project>
[...]
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20</version>
<configuration>
<excludes>
<exclude>*com.example.tests*/*Test.java</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
[...]
</project>
Another option is the old
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
or even when call it
mvn install -DskipTests
Using Categories seems to be an option that can come in handy
This is how you may add these to your Gradle script.
test {
useJUnit {
includeCategories 'org.gradle.junit.CategoryA'
excludeCategories 'org.gradle.junit.CategoryB'
}
}
A sample can be found here, adding it for a quick reference.
public interface FastTests
{
/* category marker */
}
public interface SlowTests
{
/* category marker */
}
public class A
{
#Category(SlowTests.class)
#Test public void a()
{
}
}
#Category(FastTests.class})
public class B
{
#Test public void b()
{
}
}
#RunWith(Categories.class)
#IncludeCategory(SlowTests.class)
#ExcludeCategory(FastTests.class)
#SuiteClasses({ A.class, B.class })
public class SlowTestSuite
{
}
I have found the solution for my case.
To disable all the tests during the build or even in any other context that you want the Spring annotation #IfProfileValue can be used. All tests with this annotation will be executed only in wanted context.
The example is this:
#IfProfileValue(name="enableTests", value="true")
public class DemoApplicationTests {
#Test
public void contextLoads() {
...
}
}
In my IDE I can edit the configuration and set the variable by:
-DenableTests=true
This annotation can be used on the level of a class or on the level of a test also.
All classes or tests annotated with such #IfProfileValue will be executed only in my environment and will be ignored during the build.
This approach is the best for me because it is not convenient in my project to change main pom.xml for my own test needs.
Addition.
Also in Spring or Spring Boot you should add Runner.
For example in Spring:
#RunWith(SpringJUnit4ClassRunner.class)
#IfProfileValue(name="enableTests", value="true")
#ContextConfiguration(classes = { YourClassConfig.class })
YourClassConfig might be empty:
#Configuration
public class YourClassConfig {
}
Related
I'm using Cucumber for my API tests and save the shared context in ScenarioContext class which is marked with #ScenarioScoped and #Injected per Steps class. Of course some features use steps from cross-classes as well (which causes the bug described below, but I dont want to change that due to DRY).
TestS run fine when I do them sequentially however they mix up the token Im saving in scenarioContext when I run them parallelly.
Has anyone faced this issue before? I tried with separate runners and 1 runner however the result was the same. Here's my pom.xml corresponding config:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M7</version>
<configuration>
<includes>
<include>**/*Suite.java</include>
</includes>
<forkCount>8C</forkCount>
<reuseForks>false</reuseForks>
</configuration>
</plugin>
</plugins>
</build>
Above configuration should give the highest level of separation but my tests still fail at random (which does NOT happen with sequential run).
I've read following documentations and didnt find corresponding answers:
https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html
https://maven.apache.org/surefire/maven-surefire-plugin/examples/process-communication.html
https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
#Alexey R. - Im adding additional code :)
So my ScenarioContext class looks as follows:
#ScenarioScoped
public class ScenarioContext extends Base {
Map<String, Object> scenarioContext;
public ScenarioContext() throws Exception {
this.scenarioContext = new HashMap<>();
}
public void setContext(Context key, Object value) {
scenarioContext.put(key.toString(), value);
}
public Object getContext(Context key) {
return scenarioContext.get(key.toString());
}
- some other general shortcut methods
}
And in my Steps classes aforementioned scenarioContext is called as follows:
public class myStepsClass extends Base {
#Inject
private ScenarioContext scenarioContext;
public myStepsClass(ScenarioContext scenarioContext) throws Exception {
this.scenarioContext = scenarioContext;
}
+[gherkin steps getting/setting values to scenarioContext]
I have also added a System.out.println("New scenario Context") in Scenario Context constructor and it seems to be printed out once per thread however I still get different results when tests are run in parallel :(
The #Nested classes that are executed in JUnit 5 they are ordered to run AFTER all the tests in eclosing class. How could I enforce the same behavior using maven, if my goal is to run a single enclosing class and it's nested classes? Is there a commandline or pom.xml modification to make this example test pass?
package example;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
public class SomeTopLevelTest {
private static boolean nestedDone = false;
#Test
void test1() {
Assertions.assertFalse(nestedDone,
"Nested classes should execute after the enclosing tests");
}
#Nested
class SomeNestedTest {
#Test
void test2() {
nestedDone = true;
}
}
#AfterAll
static void recheck() {
Assertions.assertTrue(nestedDone, "Nested class should be executed");
}
}
This does pass in IDE:
But does not in commanline, if I try to specify the name:
mvn test -Dtest=example.SomeTopLevelTest
[ERROR] Failures:
[ERROR] SomeTopLevelTest.recheck:27 Nested class should be executed ==> expected: <true> but was: <false>
mvn test -Dtest=example.SomeTopLevelTest*
[ERROR] Failures:
[ERROR] SomeTopLevelTest.test1:14 Nested classes should execute after the enclosing tests ==> expected: <false> but was: <true>
The problem of #Nested classes not executing is a known issue and it has been reported in both JUnit5 and Surefire issue trackers, but as of now remains unresolved.
The current state of affairs (tested with Maven 3.6.3, Junit5 5.7.2, Surefire 2.22.2 up to 3.0.0-M5):
A. Not selecting a test
mvn test
Results in executing all test classes as expected: methods from enclosing classes first, and then methods from #Nested classes, level by level
B. Selecting test the standard way
mvn test -Dtest=example.SomeTopLevelTest
This apparently triggers the default surefire excludes that use the following pattern:
<excludes>
<exclude>**/*$*</exclude>
</excludes>
Why does it not happen in case A is a mystery, but one can override this behaviour by explicitly clearing the excludes pattern:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludes>
<exclude/>
</excludes>
</configuration>
</plugin>
It does not seem to be possible to do without modyfing the pom.xml.
This DOES NOT solve the issue as posted in this question, because the nested classes are still executed first.
C. Using wildcard with -Dtest parameter
mvn test -Dtest=example.SomeTopLevelTest*
This explicitly selects all the nested classes, but - as stated in the question - results in executing the nested classes first, so it's not a solution.
D. Using includes
mvn test -DtestName=example.SomeTopLevelTest
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>${testName}</include>
</includes>
</configuration>
</plugin>
Apparently include patterns work quite differently than -Dtest parameter, because this is finally the solution to pass the test from the question. With this setup the testName may be a single class, wildcard pattern or regex
example.SomeTopLevelTest to execute all test methods in single class
example/* - all tests (including nested) in package example, but not sub-packages
example/** - all tests in package and subpackages
advanced regex, is supprted too
How can QuarkusTestResource be used in conjunction with Tag Annotation?
Example Test Routine
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
#QuarkusTest
#Tag("integration")
#QuarkusTestResource(DatabaseResource.class)
public class MyTest {
#Test
public void () {
doTests...
}
}
Maven Snippet:
<quarkus-plugin.version>1.12.1.Final</quarkus-plugin.version>
<quarkus.platform.artifact-id>quarkus-universe-bom</quarkus.platform.artifact-id>
<quarkus.platform.group-id>io.quarkus</quarkus.platform.group-id>
<quarkus.platform.version>1.12.1.Final</quarkus.platform.version>
<surefire-plugin.version>2.22.1</surefire-plugin.version>
<testscope>unit</testscope>
...
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire-plugin.version}</version>
<configuration>
<groups>${testscope}</groups>
<systemPropertyVariables>
<java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
<maven.home>${maven.home}</maven.home>
</systemPropertyVariables>
</configuration>
</plugin>
...
Maven Command:
./mvnw clean test
Result: QuarkusTestResource are started even if no QuarkusTest is annotated with "unit", so it seems that Quarkus is not aware of the Tag Annotation?
Quarkus Test Resources are global which means it will run anyway, even if your class is annotated with a tag that shouldn't run.
To prevent this, try annotating your class with
#QuarkusTestResource(restrictToAnnotatedClass = true)
From the Quarkus' website
test resources are global, even if they are defined on a test class or custom profile, which means they will all be activated for all tests, even though we do remove duplicates. If you want to only enable a test resource on a single test class or test profile, you can use #QuarkusTestResource(restrictToAnnotatedClass = true).
What you are looking for is likely the tags methof of QuarkusTestProfile. See this part of the documentation
I have looked at a few other SO questions like this and this. But those questions are pretty dated and I'm curious if there exists a new solution.
Here's what my setup looks like:
Category interface:
public interface FastTest{}
Category suite:
#RunWith(Categories.class)
#IncludeCategory(FastTest.class)
public class FastSuite{}
Sample test:
#Category(FastTest.class)
public class FastTests{
#Test public void assertOneFastTest(){}
#Test public void assertTwoFastTest(){}
}
Using maven, let's say I want to only run all my FastTest tests. Ideally, I would use the command
mvn test -Dtest.category=FastTest
or
mvn test -Dtest.suite=FastSuite
But I have not been able to get this working. Does anyone have any suggestions aside from using ClasspathSuite? Thanks.
You can do this from the surefire plugin, using the configuration for groups, but you need to specify the junit47 provider as well. The following works for me:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.11</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>default-test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
<configuration>
<groups>com.xxx.foo.CategoryClass</groups>
</configuration>
</plugin>
I think this was introduced fairly recently, so may not work in earlier versions of the plugin, pre 2.11. You need to specify the provider, otherwise it doesn't work. The groups should be the fully qualified name of the Category class. You can also specify an excludeGroups as well if needs be.
For more information, see Maven Surefire Plugin surefire:test.
Did a little more research and found no options so I ended up using ClasspathSuite. It turns out that it's not any different except that you cannot specify #Category at a class level, you have to annotate every method you want to categorize
Now, each category suite looks like this:
#RunWith(Categories.class)
#Categories.IncludeCategory(FastTest.class)
#Suite.SuiteClasses(AllTests.class)
public class FastSuite{}
You have to modify AllTests to look like this:
#RunWith(ClasspathSuite.class)
public class AllTests{}
For methods, it looks like this:
public class FastTests{
#Categories(FastTest.class)
public void assertOneFastTest(){}
#Categories(FastTest.class)
public void assertTwoFastTest(){}
}
Using maven, you can do the following command:
mvn test -Dtest=FastSuite -Dt3-chrome-path=%CHROME_DRIVER_HOME -Dwebdriver.chrome.driver=%CHROME_DRIVER_HOME
I have following tests structure:
public class WorkerServiceTest {
public class RaiseErrorTest extends AbstractDbUnitTest{
#Test
public void testSomething(){
}
...
}
...
}
It's done because I don't want to create a separate class file for every test case extending AbstractDbUnitTest.
The problem is that mvn test doesn't run test from my inner class. Is there is a way how to configure Maven to run such tests? Of course I can create methods in the parent class calling the inner class methods but I want a clearer way.
Yes, this is possible using the new (well, it's not new anymore) Enclosed runner (since JUnit 4.5) that runs all static inner classes of an outer class.
To use it, just annotate the outer class with #RunWith(Enclosed.class) and make the inner classes static.
#RunWith(Enclosed.class)
public class WorkerServiceTest {
public static class RaiseErrorTest extends AbstractDbUnitTest{
#Test
public void testSomething(){
}
...
}
...
}
And mvn test will run them.
I explain (a little more) the solution that I found...
Maven (AFAIK) uses by default the plugin "maven-surefire-plugin" to run any tests defined at your maven project. According to the documentation of this plugin, by default, it excludes tests that are enclosed at inner static classes (or at least it was with version that i'm using - 2.18.1).
So what i did was to put a empty exclude rule; resulting with a pom's build section like this:
<build>
<plugins>
...
<!-- ~~~~~~~~~~ SUREFIRE -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<excludes>
<exclude></exclude>
</excludes>
</configuration>
</plugin>
...
</plugins>
</build>