I converted a dynamically generated JUnit test suite to TestNG using a #Factory annotation. The tests are generated by scanning a directory that contains several test specification files (written in a DSL) and by loading them in an test class that knows how to execute them. The test class has a single #Test method named test, implements org.testng.ITest and overrides getTestName() as recommended.
However, in the TestNG Eclipse UI, only the factory and a single execution of test is shown (although it is executed several times, as expected). With JUnit, I had the name of each spec listed as a separate test. This is very important because several test specs could fail and I would need to see all failures (which does not work in the TestNG Eclipse UI for me).
How can I achieve something similar with TestNG? I use the Eclipse TestNG plugin 6.7.0 and testng 6.7.
Try extending XmlTest in your test class and call setName(). I extended XmlSuite and used setName to name my "test suite" which is a class with multiple tests defined in it and that worked for me. In fact I just tried what I suggested on the same class and now it's called a test with the correct name. I'm not sure why the ITest interface is being ignored as I also tried that approach without success.
As carlin.scott suggested, extending XmlTest works for this problem. However, I overrode toString() to return the test name (instead of calling setName). This has the advantage that the test name in the TestNG view is easier to read. By default, it contains additional information such as parameters and metagroups.
Related
I have a bunch of JUnit tests that extend my base test class called BaseTest which in turn extends Assert. Some of my tests have a #Category(SlowTests.class) annotation.
My BaseTest class is annotated with the following annotation #RunWith(MyJUnitRunner.class).
I've set up a Gradle task that is expected to run only SlowTests. Here's my Gradle task:
task integrationTests(type: Test) {
minHeapSize = "768m"
maxHeapSize = "1024m"
testLogging {
events "passed", "skipped", "failed"
outputs.upToDateWhen {false}
}
reports.junitXml.destination = "$buildDir/test-result"
useJUnit {
includeCategories 'testutils.SlowTests'
}
}
When I run the task, my tests aren't run. I've pinpointed this issue to be related to the custom runner MyJUnitRunner on the BaseTest. How can I set up my Gradle or test structure so that I can use a custom runner while using the Suite.
The solution to this turned out to smaller and trickier than I thought. Gradle was using my custom test runner and correctly invoking the filter method. However, my runner reloads all test classes through its own classloader for Javaassist enhancements.
This lead to the issue that SlowTest annotation was loaded through the Gradle classloader but when passed to my custom runner, the runner checked if the class was annotated with that annotation. This check never resolved correctly as the equality of the SlowTest annotation loaded through two different classloaders was different.
--
Since I've already done the research, I'll just leave this here. After days of digging through the Gradle and the (cryptic) JUnit sources, here's what I got.
Gradle simply doesn't handle any advanced JUnit functionality except the test categorization. When you create a Gradle task with the include-categories or the exclude-categories conditions, it builds a CategoryFilter. If you don't know, a Filter is what JUnit gives to the test-runner to decide whether a test or a test method should be filtered out. The test runner must implement the Filterable interface.
JUnit comes with multiple runners, the Categories is just another one of them. It extends a family of test runners called Suite. These suite based runners are designed to run a "suite" of tests. A suite of tests could be built by annotation introspection, by explicitly defining tests in a suite or any other method that builds a suite of tests.
In the case of the Categories runner, JUnit has it's own CategoryFilter but Gradle doesn't use that, it uses it's own CategoryFilter. Both provide more or less the same functionality and are JUnit filters so that can be used by any suite that implements Filterable.
The actual class in the Gradle responsible for running the JUnit tests is called JUnitTestClassExecuter. Once it has parsed the command line options it requests JUnit to check the runner should be used for a test. This method is invoked for every test as seen here.
The rest is simply up to JUnit. Gradle just created a custom RunNotifier to generate the standard XML files representing test results.
I hope someone finds this useful and saved themselves countless hours of debugging.
TLDR: You can use any runner in Gradle. Gradle has no specifics pertaining to runners. It is JUnit that decided the runners. If you'd like to know what runner will be used for your test, you can debug this by calling
Request.aClass(testClass).getRunner(). Hack this somewhere into your codebase and print it to the console. (I wasn't very successful in attaching a debugger to Gradle.)
A similar question has already been asked here.
One (unaccepted) answer states:
the test class will always be started directly and then through the
"link" in the suite. This is as expected.
Can someone explain what this actually means and whether or not it is possible to prevent the tests running twice.
When I run the tests from the command line using mvn test they only run once.
UPDATE
I have a test suite defined as follows:
#RunWith(Suite.class)
#SuiteClasses({ TestCase1.class, TestCase2.class })
public class MyTestSuite
{
}
When you run tests in Eclipse on project level (or package level), Eclipse searches all project's source folders for JUnit classes (or selected package). These are all classes with #Test annotations and all classes with #RunWith (probably some more too). Then for all these classes it runs them as tests.
As a result of this behavior, if you have a suite class that references tests classes in the same project, these tests will run twice. If you had another suite that did the same, they would run three times and so on. To understand this behavior try running a suite that contains one test case twice, for instance:
#RunWith(Suite.class)
#SuiteClasses({ TestCase1.class, TestCase1.class })
public class TestSuite {}
Accepted strategy here is to define a suite or suites for a project an run them exclusively. Do not start tests on a project level but run selected suites only.
As far as Maven is concerned, I suspect that its default configuration only picks out suite class and omits test cases. Had it been configured differently, it would behave the same as Eclipse.
Elipse tests 2 classes and give you 2 results.
Maven tests 2 classes and give you one result with 2 sub results.
I think is somethink like this, but still most important thing is that result are
positive! :)
Regards!
Same as this question https://github.com/spring-projects/spring-boot/issues/13750
Just exclude individual test cases and include the suite test cases.
Our test suite is growing quickly and we have reached a point where our more functional tests are dependent on other systems.
We use gradle test tasks to run these tests using the include and exclude filters but this is becoming cumbersome because we are having to name our tests in a particular way.
Our current approach is to name our tests in the following way:
class AppleSingleServiceTest {}
class BananaMultiServiceTest {}
class KiwiIntegrationTest {}
and then include tests in the relevant task using
include '**/*SingleServiceTest.class'
include '**/*MultiServiceTest.class'
include '**/*IntegrationTest.class'
Is it possible find test classes in gradle by looking at annotations?
#SingleServiceTest
public class AppleTest {}
I think any tests that are not annotated would then be run as normal unit tests, so if you forget to annotate a more functional test it will fail
An example of a single service test is a selenium test where any external dependencies of the SUT are stubbed
An example of a multi service test is one where some but maybe not all external dependencies are not stubbed
As of Gradle 1.6, Gradle supports selecting tests with JUnit #Category, like this:
test {
useJUnit {
includeCategories 'org.gradle.junit.CategoryA'
excludeCategories 'org.gradle.junit.CategoryB'
}
}
More details can be found in the docs.
The feature you are asking for doesn't currently exist, but you can make a feature request at http://forums.gradle.org. Or you can use the (cumbersome) JUnit #Category, which requires you to define test suites.
I had a similar need to filter tests with annotations. I eventually managed to create a solution. It is posted here.
How do I run a Junit 4.8.1 Test suite from command line ?
Also I want to use the categories introduces with JUnit 4.8 , is there a way where
I can specify from command line the category which I want to run.
Using java run JUnitCore class (also see here).
Categories are supposed to be used with test suites with #RunWith(Categories.class)
, #IncludeCategory and #ExcludeCategory. I am not aware of any dynamic way to use categories to run tests but I'd like to know of such it it exists. You can have pre-defined test suites for certain categories to run them.
There is no way (as of 4.8) to specify categories from the command line.
I can suggest two approaches:
1. Create Ant file with junit target and then invoke this target from commend line.
2. Implement test suite class, in it in some class with main() method. So you will be able to run it.
In 4.10, we do this:
mvn verify -p(your profiles) -Dit.test=(SuiteClass)
where SuiteClass is an empty class (no methods or fields) that is annotated with #RunWith(Categories.class) and #Suite.SuiteClasses({FooIT.class, BarIT.class, ...}). FooIT and BarIT are the integration tests.
Hi – I wanted to automatically sweep JUnit tests into suites as part of my continuous builds, so I derived a runner from JUnit's Suite which finds all test classes in a package. The runner works just fine, but the results display is less than expected.
I have one class in my testing support package with a #RunWith annotation for my runner. The runner works by reading a property to get the package under test. Set the property and tell JUnit to run the annotated class, and all tests in that package are executed. The name of the suite is reported as the name of the class which has the #RunWith annotation, in both Ant and IntelliJ. My runner has an override for ParentRunner.getName() which returns the name of the package under test. I verified that the string gets into the runner's Description object. What am I missing?
Environment:
JUnit: 4.5
Ant: 1.7.0
IntelliJ IDEA: 8.1
Thanks for whatever direction you can provide.
This is because ANT and IntelliJ use their own runners, so they are building the name based on the test, and not getting the name from your runner. In other words, the runner is delegated to for the purpose of running the test, but not for the purpose of describing it.
I had a similar problem just a few weeks ago and created an open source project because of it.
You may include it via maven
<dependency>
<groupId>com.github.cschoell</groupId>
<artifactId>junit-dynamicsuite</artifactId>
<version>0.2.0</version>
<scope>test</scope>
</dependency>
or download it from the github page, where you will find its documentation as well.
https://github.com/cschoell/Junit-DynamicSuite
There is a Junit runner included which allows to scan either a directory or the classpath for unit tests and filter them by implementing a simple interface.