Is there any way to group tests in JUnit, so that I can run only some groups?
Or is it possible to annotate some tests and then globally disable them?
I'm using JUnit 4, I can't use TestNG.
edit: #RunWith and #SuiteClasses works great. But is it possible to annotate like this only some tests in test class? Or do I have to annotate whole test class?
JUnit 4.8 supports grouping:
public interface SlowTests {}
public interface IntegrationTests extends SlowTests {}
public interface PerformanceTests extends SlowTests {}
And then...
public class AccountTest {
#Test
#Category(IntegrationTests.class)
public void thisTestWillTakeSomeTime() {
...
}
#Test
#Category(IntegrationTests.class)
public void thisTestWillTakeEvenLonger() {
...
}
#Test
public void thisOneIsRealFast() {
...
}
}
And lastly,
#RunWith(Categories.class)
#ExcludeCategory(SlowTests.class)
#SuiteClasses( { AccountTest.class, ClientTest.class })
public class UnitTestSuite {}
Taken from here: https://community.oracle.com/blogs/johnsmart/2010/04/25/grouping-tests-using-junit-categories-0
Also, Arquillian itself supports grouping:
https://github.com/weld/core/blob/master/tests-arquillian/src/test/java/org/jboss/weld/tests/Categories.java
Do you want to group tests inside a test class or do you want to group test classes? I am going to assume the latter.
It depends on how you are running your tests. If you run them by Maven, it is possible to specify exactly what tests you want to include. See the Maven surefire documentation for this.
More generally, though, what I do is that I have a tree of test suites. A test suite in JUnit 4 looks something like:
#RunWith(Suite.class)
#SuiteClasses({SomeUnitTest1.class, SomeUnitTest2.class})
public class UnitTestsSuite {
}
So, maybe I have a FunctionTestsSuite and a UnitTestsSuite, and then an AllTestsSuite which includes the other two. If you run them in Eclipse you get a very nice hierarchical view.
The problem with this approach is that it's kind of tedious if you want to slice tests in more than one different way. But it's still possible (you can for example have one set of suites that slice based on module, then another slicing on the type of test).
To handle the globally disabling them, JUnit (4.5+) has two ways One is to use the new method assumeThat. If you put that in the #BeforeClass (or the #Before) of a test class, and if the condition fails, it will ignore the test. In the condition you can put a system property or something else that can be globally set on or off.
The other alternative is to create a custom runner which understands the global property and delegates to the appropriate runner. This approach is a lot more brittle (since the JUnit4 internal runners are unstable and can be changed from release to release), but it has the advantage of being able to be inherited down a class hierarchy and be overridden in a subclass. It is also the only realistic way to do this if you have to support legacy JUnit38 classes.
Here is some code to do the custom Runner. Regarding what getAppropriateRunnerForClass might do, the way I implemented it was to have a separate annotation that tells the custom runner what to run with. The only alternative was some very brittle copy paste from the JUnit code.
private class CustomRunner implements Runner
private Runner runner;
public CustomRunner(Class<?> klass, RunnerBuilder builder) throws Throwable {
if (!isRunCustomTests()) {
runner = new IgnoredClassRunner(klass);
} else {
runner = getAppropriateRunnerForClass(klass, builder);
}
public Description getDescription() {
return runner.getDescription();
}
public void run(RunNotifier notifier) {
runner.run(notifier);
}
}
EDIT: The #RunWith tag only works for a whole class. One way to work around that limiation is to move the test methods into a static inner class and annotate that. That way you have the advantage of the annotation with the organization of the class. But, doing that won't help with any #Before or #BeforeClass tags, you will have to recreate those in the inner class. It can call the outer class's method, but it would have to have its own method as a hook.
In JUnit 5 you can declare #Tag for filtering tests, either at the class or method level; analogous to test groups in TestNG or Categories in JUnit 4
From the javadoc :
tags are used to filter which tests are executed for a given test
plan. For example, a development team may tag tests with values such
as "fast", "slow", "ci-server", etc. and then supply a list of tags to
be used for the current test plan, potentially dependent on the
current environment.
For example you could declare a test class with a "slow" #Tag that will be inherited for all methods and override it for some methods if required :
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
#Tag("slow")
public class FooTest{
//
#Test
void loadManyThings(){
...
}
#Test
void loadManyManyThings(){
...
}
#Test
#Tag("fast")
void loadFewThings(){
...
}
}
You could apply the same logic for other test classes.
In this way test classes (and methods too) belongs to a specific tag.
As a good practice instead of copying and pasting #Tag("fast") and #Tag("slow") throughout the test classes, you can create custom composed annotations.
For example :
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
import org.junit.jupiter.api.Tag;
#Target({ ElementType.TYPE, ElementType.METHOD })
#Retention(RetentionPolicy.RUNTIME)
#Tag("slow")
public #interface Slow {
}
and use it as :
#Test
#Slow
void slowProcessing(){
...
}
To enable or disable test marked with a specific tag during the text execution you can rely on the maven-surefire-plugin documentation :
To include tags or tag expressions, use groups.
To exclude tags or tag expressions, use either excludedGroups.
Just configure in your pom.xml the plugin according to your requirement (example of the doc) :
<build>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.0</version>
<configuration>
<groups>acceptance | !feature-a</groups>
<excludedGroups>integration, regression</excludedGroups>
</configuration>
</plugin>
</plugins>
</build>
For information the test goal documentation is not updated.
Try JUnit Test Groups. From documentation :
#TestGroup("integration")
public class MyIntegrationTest {
#ClassRule
public static TestGroupRule rule = new TestGroupRule();
...
}
Execute a simple test group: -Dtestgroup=integration
Execute multiple test groups: -Dtestgroup=group1,group2
Execute all test groups: -Dtestgroup=all
You can create test Suite objects that contain groups of tests. Alternatively, your IDE (like Eclipse) may have support for running all the tests contained in a given package.
You can Use Test Suite(http://qaautomated.blogspot.in/2016/09/junit-test-suits-and-test-execution.html) or you can Junit Categories(http://qaautomated.blogspot.in/2016/09/junit-categories.html) for grouping your test cases effectively.
Related
I am aware that I can make a TestSuite enumerating all the classes that I want, for example:
#RunWith(Suite.class)
#SuiteClasses({SQLServerTests1.class, SQLServerTest2.class, ... })
public class AllSQLServerTests {}
However I have almost 100+ classes and I don't want to have to remember to include any new one in the #SuiteClasses annotation.
As my classes have a naming convention (starting with "SQLServer" for example) I am searching for a way to do something like this:
#RunWith(Suite.class)
#SuiteClasses(prefix="SQLServer")
public class AllSQLServerTests {}
is it possible with plain JUnit? with spring or any other framework?
Tag'em
You can add many tags to each test or test class:
#Test
#Tag("red")
#Tag("production")
public void testWithColour() {...}
#RunWith(JUnitPlatform.class)
#IncludeTags("red & !production")
public class JUnit5Example {
//...
}
You can also use #ExcludeTags but it cannot co-exist with #IncludeTags
Run all in test package
#RunWith(JUnitPlatform.class)
#SelectPackages("com.acme.megaproduct.slowtests")
public class JUnit5Example {
//...
}
Write custom Test Runner
Perhaps none of the above can acommodate your needs, in which case you can add custom filtering by writing your own runner.
See here for step by step how to do it.
Then you just use it like:
#RunWith(MyCustomRunner.class)
public class CustomTestSuite {
//...
}
I have a test class with multiple test methods that I would like to group by some criteria. For this purpose, using JUnit's #Category annotation on a method level seemed like a fine solution:
public class TestClass {
#Test
#Category(AssignmentServiceCategory.class)
public void testMethod1() {}
#Test
#Category(OtherCategory.class)
public void testMethod2() {}
}
I would like to create different run configurations in IntelliJ IDEA for those separate categories so that only the test methods annotated with certain category are executed. My configuration looks like this:
However, when I run this, all of the tests from the class where the method is declared are run, instead of only the ones annotated with specified category. Is my configuration incorrect, or does IDEA allow only class-level #Category annotations?
Versions:
IntelliJ IDEA 2018.1 (181.4203.550)
JRE: 1.8.0_152-release-1136-b20 amd64
JUnit 4.12
UPDATED
I tried to reproduce the issue and could not.
Here's my Test Class
package com.mytests.category;
import org.junit.Test;
import org.junit.experimental.categories.Category;
public class MyTest {
#Test
#Category(PerformanceTests.class)
public void testMethod1() {
System.out.println("method1");
}
#Test
#Category(RegressionTests.class)
public void testMethod2() {
System.out.println("method2");
}
}
Make sure you have the necessary interfaces. In JUnit, you need to create marker interfaces to represent the categories:
package com.mytests.category;
public interface RegressionTests {}
and
package com.mytests.category;
public interface PerformanceTests {}
Then in IntelliJ, I ran the tests once and it creates a configuration for me automatically. Then I edit the configuration
The results were as expected:
Only testMethod1 was executed.
OLDER ANSWER
Or from IntelliJ's Doc (https://www.jetbrains.com/help/idea/run-debug-configuration-junit.html)
Category Select this option if you only want to run test classes and
test methods that are annotated either with the category given with
the #IncludeCategory annotation, or a subtype of this category. Fill in the following fields:
Category Specify the desired category. Type category name, or click
browseButton and select the desired category in the dialog that opens.
Or You could create a TestSuite and specify (in there) which categories the suite is to include.
Something like
package org.mytests.category;
import org.junit.experimental.categories.Categories;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Categories.class)
#Categories.IncludeCategory(RegressionTests.class)
#Suite.SuiteClasses({ClassA.class, ClassB.class, ClassC.class})
public class RegressionTestSuite {
}
I have (had) the same problem with Intellij 2020.1.2
There seems to be a bug related to what setting you select for search for tests.
If i choose in single module all the test are executed, even the ones annotated with other Categories.
If I choose any of the other options (in whole project, across module dependencies) it simply works as expected..
I need to know the suite class that included the currently running JUnit test. For example, if you have
#SuiteClasses({SubSuite.class})
class ParentSuite { }
#SuiteClasses({TestCase.class})
class SubSuite { }
class TestCase {
#Test
public void testMethod() { }
}
and execute ParentSuite with JUnit, then I want to get a reference to ParentSuite.class. Is this possible?
There is the TestWatcher rule that gets you an instance of Description, which is kind of in the right direction, but that does not include the suite class.
I know this is probably not the best way to write unit tests. My original problem is to run a validation on all classes that are in the project under test, but not those of the dependencies. The TestCase will be in a dependency and included by test suites in other projects. The only solution I could think of is filter those classes with the same source location as that of the top-most-suite. To make that more clear:
BaseLibrary
* contains TestCase and TestSuite
* has classes that should not be validated
ConsumerProject
* has a test-scoped and test-classified dependency to BaseLibrary
* contains ParentSuite
* has classes that should be validated
You probably want to use JUnit #Category https://github.com/junit-team/junit/wiki/Categories
Here's a blog on how to make Categories https://weblogs.java.net/blog/johnsmart/archive/2010/04/25/grouping-tests-using-junit-categories-0
This way you can make Categories for each of your projects.
public interface ConsumerProject {}
Then you can use the #Category annotation to mark test classes (or even test methods) as being in particular categories:
#Category(ConsumerProject.class)
public class TestCase { ... }
Then your suites can be set up to run all the tests for particular categories
#RunWith(Categories.class)
#IncludeCategory(ConsumerProject.class)
#SuiteClasses( { ....})
public class ConsumerProjectSuite { }
You can even have the same test get marked with multiple categories
#Category({ConsumerProject.class, OtherProject.class})
public class CommonTests { }
I've made my own class with an #RunWith(AllTests.class) on it for integration tests that I want to execute, but I've put it in a reusable jar. I'm trying to tell failsafe to use it but I'm not sure how to because the documentation for includes says:
A list of elements specifying the tests (by pattern) that should be included in testing. When not specified and when the test parameter is not specified, the default includes will be
<includes>
<include>**/IT*.java</include>
<include>**/*IT.java</include>
<include>**/*ITCase.java</include>
</includes>
Each include item may also contain a comma-separated sublist of items, which will be treated as multiple entries.
This parameter is ignored if the TestNG suiteXmlFiles parameter is specified.
But this test runner is in the classpath, not on the filesystem. How can I tell failsafe to use my class runner and only my class runner?
Can't you use the #RunWith annotation provided by jUnit?
import com.example.YourTestRunner;
#RunWith(YourTestRunner.class)
public class SomeIntegrationTest {
#Test
public void simpleTest() {
// given, when, then
}
}
Update
Based on the comment:
My class isn't a test runner, but a class that uses #RunWith
you can use inheritance to solve the problem:
#RunWith(SomeTestRunner.class)
#Ignore("Not a real test class because it does not contain any #Test methods, but needed to keep surefire happy")
public class ParentTest {
// this is the reusable class that is in the jar file
}
public class SomeIntegrationTest extends ParentTest {
#Test
public void simpleTest() {
// given, when, then
}
}
In our project, we currently have a large number of (junit) tests that are split into three categories: unit, integration, wicket.
I now want to group these tests so I can run only one (or two) of those categories. The only thing I found are junit test suites and categories as described here: http://www.wakaleo.com/component/content/article/267
My problem is, I don't want to declare every single test in the Test Suits with #SuiteClasses.
Is there a way to add the suite classes with wildcards / patterns?
Assuming my understanding of the question is correct, it actually can be done using JUnit. The code below was used with JUnit 4.11 and allowed us to split all tests into 2 categories: "uncategorized" and Integration.
IntegrationTestSuite.java
/**
* A custom JUnit runner that executes all tests from the classpath that
* match the <code>ca.vtesc.portfolio.*Test</code> pattern
* and marked with <code>#Category(IntegrationTestCategory.class)</code>
* annotation.
*/
#RunWith(Categories.class)
#IncludeCategory(IntegrationTestCategory.class)
#Suite.SuiteClasses( { IntegrationTests.class })
public class IntegrationTestSuite {
}
#RunWith(ClasspathSuite.class)
#ClasspathSuite.ClassnameFilters({ "ca.vtesc.portfolio.*Test" })
class IntegrationTests {
}
UnitTestSuite.java
/**
* A custom JUnit runner that executes all tests from the classpath that match
* <code>ca.vtesc.portfolio.*Test</code> pattern.
* <p>
* Classes and methods that are annotated with the
* <code>#Category(IntegrationTestCategory.class)</code> category are
* <strong>excluded</strong>.
*/
#RunWith(Categories.class)
#ExcludeCategory(IntegrationTestCategory.class)
#Suite.SuiteClasses( { UnitTests.class })
public class UnitTestSuite {
}
#RunWith(ClasspathSuite.class)
#ClasspathSuite.ClassnameFilters({ "ca.vtesc.portfolio.*Test" })
class UnitTests {
}
IntegrationTestCategory.java
/**
* A marker interface for running integration tests.
*/
public interface IntegrationTestCategory {
}
The first sample test below is not annotated with any category so all its test methods will be included when running the UnitTestSuite and excluded when running IntegrationTestSuite.
public class OptionsServiceImplTest {
#Test
public void testOptionAssignment() {
// actual test code
}
}
Next sample is marked as Integration test on the class level which means both its test methods will be excluded when running the UnitTestSuite and included into IntegrationTestSuite:
#Category(IntegrationTestCategory.class)
public class PortfolioServiceImplTest {
#Test
public void testTransfer() {
// actual test code
}
#Test
public void testQuote() {
}
}
And the third sample demos a test class with one method not annotated and the other marked with the Integration category.
public class MarginServiceImplTest {
#Test
public void testPayment() {
}
#Test
#Category(IntegrationTestCategory.class)
public void testCall() {
}
}
Try using ClasspathSuite
I also had the same problem where I had more then 5500 jUnit tests. I categorised then into 3 groups and created 3 suites using the above jUnit extension. Its great.
You could put them in different packages. Most IDEs have a way to run all the tests in a given package. It's also pretty simple to find all the test classes in a package with a shell script for running tests as part of a build or whatever. I don't know if there's a way to do it with ant, but I would imagine so.
TestNG lets you tag tests as being in particular groups, then run those groups. That sound like exactly what you want, apart from the fact that it's not JUnit!
You could abuse JUnit's assumption mechanism to do what you want: have a system property for each group of tests, and then start each test by assuming that the appropriate property is set. Running all tests will run everything, but everything you don't want will be ignored.
Even if you use JUnit categories, you still won't be able to use wildcards/patterns since categories are annotations, which are Java types.
As pointed out by other commenters, this is exactly why TestNG uses strings to define groups instead of annotations:
#Test(groups = { "database.ACCOUNTS", "fast-test" })
public void newAccountsShouldBeCreated() { ... }
Once you have defined your groups this way, you can include and exclude groups using regular expressions (e.g. "database.*", "front-end.*", etc...).
TestNG is indeed not based on JUnit, but it's very easy to convert all your JUnit tests to TestNG. Here are two blog posts that give an overview of the process:
http://beust.com/weblog/2011/01/04/one-click-test-conversions/
http://beust.com/weblog/2011/02/07/are-your-unit-tests-talking-to-each-other-behind-your-back/
See Junit category or How to run all tests belonging to a certain Category in JUnit 4
have you considered using TestNG?
This is built on JUnit but a lot more powerfull: See comparison.
Grouping is easy.
Tranforming your tests from JUnit to TestNG should be straightforward.
Alternatively, you could create 3 ant scripts that will each run their unit tests but this is less flexible.