Our test suite is growing quickly and we have reached a point where our more functional tests are dependent on other systems.
We use gradle test tasks to run these tests using the include and exclude filters but this is becoming cumbersome because we are having to name our tests in a particular way.
Our current approach is to name our tests in the following way:
class AppleSingleServiceTest {}
class BananaMultiServiceTest {}
class KiwiIntegrationTest {}
and then include tests in the relevant task using
include '**/*SingleServiceTest.class'
include '**/*MultiServiceTest.class'
include '**/*IntegrationTest.class'
Is it possible find test classes in gradle by looking at annotations?
#SingleServiceTest
public class AppleTest {}
I think any tests that are not annotated would then be run as normal unit tests, so if you forget to annotate a more functional test it will fail
An example of a single service test is a selenium test where any external dependencies of the SUT are stubbed
An example of a multi service test is one where some but maybe not all external dependencies are not stubbed
As of Gradle 1.6, Gradle supports selecting tests with JUnit #Category, like this:
test {
useJUnit {
includeCategories 'org.gradle.junit.CategoryA'
excludeCategories 'org.gradle.junit.CategoryB'
}
}
More details can be found in the docs.
The feature you are asking for doesn't currently exist, but you can make a feature request at http://forums.gradle.org. Or you can use the (cumbersome) JUnit #Category, which requires you to define test suites.
I had a similar need to filter tests with annotations. I eventually managed to create a solution. It is posted here.
Related
This question is more of a best practices approach. The application we have is Spring Boot 1.5.4 and builds using Gradle. I'm in the process of creating nightly builds with Jenkins, and want to make sure all the unit tests pass in the project.
The project has a number of "tests" like this however:
#SpringBootTest(classes = {Application.class})
#RunWith(SpringRunner.class)
public class DatabaseCreationProcessImplTest {
This particular class creates a sample database image for developers to work off of. Granted we could make straight SQL scripts, but a Java process is useful since there's code that also queries for data from outside sources (e.g. Liferay.)
The reason we're using a unit test for this is because developers can easily run it in IntelliJ to load a new database image. However this isn't really a "test", it's using the test runner as a quick way to run a Java process.
I'm working on setting up nightly builds and I don't want this test to be included in the builds. I can do something like the following in the build script:
test {
exclude 'com/mydomain/service/util/impl/DatabaseCreationProcessImplTest.class'
}
However by doing this, if running the unit test individually in the IDE with the Spring test runner, it is unable to find any tests. I thought about passing in a Boolean value in the Jenkins task for doing this, e.g.
test {
systemProperties 'property': 'value'
testLogging {
events "passed", "skipped", "failed", "standardOut", "standardError"
}
// Exclusions go here
if (Boolean.getBoolean('exclude.tests')) {
exclude 'com/mydomain/service/util/impl/DatabaseCreationProcessImplTest.class'
}
}
However this seems like a hack/kludge... any ways looking for some "best practices" approach for handling this. Is JUnit the right way for quickly running Java processes? Are there other alternatives? Is it possible to create a Gradle script which developers can use to invoke common Java (Spring Boot) process as well?
I think you could group your not-really-tests in a test suite with JUnit's #SuiteClasses annotation:
#Suite.SuiteClasses(DatabaseCreationProcessImplTest.class)
public class NotReallyTests {}
And then use a condition that you pass from your Jenkins command line to exclude the not-really-tests suite:
test {
if (project.hasProperty('excludeNotReallyTests')) {
useJunit {
excludeCategories 'fully.qualified.name.of.your.NotReallyTests'
}
}
}
Your Jenkins command line would then be
$ gradle -PexcludeNotReallyTests=true
It's a little less hacky than your solution in that it keeps track of the grouping of tests that are not really tests in the codebase instead of the build.gradle file.
The Testing API for Android provides several annotations that are used to group tests together. Then you can specify which tests to run by giving one of the annotations on the command line. I do not know the details of how to implement this. It is just a suggestion for you to explore on your own, if you are interested.
I have a bunch of JUnit tests that extend my base test class called BaseTest which in turn extends Assert. Some of my tests have a #Category(SlowTests.class) annotation.
My BaseTest class is annotated with the following annotation #RunWith(MyJUnitRunner.class).
I've set up a Gradle task that is expected to run only SlowTests. Here's my Gradle task:
task integrationTests(type: Test) {
minHeapSize = "768m"
maxHeapSize = "1024m"
testLogging {
events "passed", "skipped", "failed"
outputs.upToDateWhen {false}
}
reports.junitXml.destination = "$buildDir/test-result"
useJUnit {
includeCategories 'testutils.SlowTests'
}
}
When I run the task, my tests aren't run. I've pinpointed this issue to be related to the custom runner MyJUnitRunner on the BaseTest. How can I set up my Gradle or test structure so that I can use a custom runner while using the Suite.
The solution to this turned out to smaller and trickier than I thought. Gradle was using my custom test runner and correctly invoking the filter method. However, my runner reloads all test classes through its own classloader for Javaassist enhancements.
This lead to the issue that SlowTest annotation was loaded through the Gradle classloader but when passed to my custom runner, the runner checked if the class was annotated with that annotation. This check never resolved correctly as the equality of the SlowTest annotation loaded through two different classloaders was different.
--
Since I've already done the research, I'll just leave this here. After days of digging through the Gradle and the (cryptic) JUnit sources, here's what I got.
Gradle simply doesn't handle any advanced JUnit functionality except the test categorization. When you create a Gradle task with the include-categories or the exclude-categories conditions, it builds a CategoryFilter. If you don't know, a Filter is what JUnit gives to the test-runner to decide whether a test or a test method should be filtered out. The test runner must implement the Filterable interface.
JUnit comes with multiple runners, the Categories is just another one of them. It extends a family of test runners called Suite. These suite based runners are designed to run a "suite" of tests. A suite of tests could be built by annotation introspection, by explicitly defining tests in a suite or any other method that builds a suite of tests.
In the case of the Categories runner, JUnit has it's own CategoryFilter but Gradle doesn't use that, it uses it's own CategoryFilter. Both provide more or less the same functionality and are JUnit filters so that can be used by any suite that implements Filterable.
The actual class in the Gradle responsible for running the JUnit tests is called JUnitTestClassExecuter. Once it has parsed the command line options it requests JUnit to check the runner should be used for a test. This method is invoked for every test as seen here.
The rest is simply up to JUnit. Gradle just created a custom RunNotifier to generate the standard XML files representing test results.
I hope someone finds this useful and saved themselves countless hours of debugging.
TLDR: You can use any runner in Gradle. Gradle has no specifics pertaining to runners. It is JUnit that decided the runners. If you'd like to know what runner will be used for your test, you can debug this by calling
Request.aClass(testClass).getRunner(). Hack this somewhere into your codebase and print it to the console. (I wasn't very successful in attaching a debugger to Gradle.)
A similar question has already been asked here.
One (unaccepted) answer states:
the test class will always be started directly and then through the
"link" in the suite. This is as expected.
Can someone explain what this actually means and whether or not it is possible to prevent the tests running twice.
When I run the tests from the command line using mvn test they only run once.
UPDATE
I have a test suite defined as follows:
#RunWith(Suite.class)
#SuiteClasses({ TestCase1.class, TestCase2.class })
public class MyTestSuite
{
}
When you run tests in Eclipse on project level (or package level), Eclipse searches all project's source folders for JUnit classes (or selected package). These are all classes with #Test annotations and all classes with #RunWith (probably some more too). Then for all these classes it runs them as tests.
As a result of this behavior, if you have a suite class that references tests classes in the same project, these tests will run twice. If you had another suite that did the same, they would run three times and so on. To understand this behavior try running a suite that contains one test case twice, for instance:
#RunWith(Suite.class)
#SuiteClasses({ TestCase1.class, TestCase1.class })
public class TestSuite {}
Accepted strategy here is to define a suite or suites for a project an run them exclusively. Do not start tests on a project level but run selected suites only.
As far as Maven is concerned, I suspect that its default configuration only picks out suite class and omits test cases. Had it been configured differently, it would behave the same as Eclipse.
Elipse tests 2 classes and give you 2 results.
Maven tests 2 classes and give you one result with 2 sub results.
I think is somethink like this, but still most important thing is that result are
positive! :)
Regards!
Same as this question https://github.com/spring-projects/spring-boot/issues/13750
Just exclude individual test cases and include the suite test cases.
I have extracted all my integration tests out of my multi-module setup and put them all into a separate project. These integration tests are based on spring and a use a real database. I am using dbmaintain which is a database versioning tool, it automatically tracks which SQL files need to be applied and keeps the database in a correct state.
What I would like is to be able to run the code that ensures the database is up to date before any test is run. So if you run all the tests (from Eclipse or Maven in my case) that it will first perform the db check once, or if you run a single test it will first perform the db check. No matter how many tests are run, it should always run the db check.
Right now I am thinking that I will use #BeforeClass in the base test class (all tests ultimately extend from this class) which will instantiate a singleton to do it's work. That singleton will control everything to make sure things only get run once.
I am hoping there is a cleaner way.
By default, the Maven runner for JUnit reserves the right to reorder tests. This is actually a Good Thing(tm), because you can tell the Maven JUnit plugin to run tests in parallel, which means you wouldn't know the order anyways. In addition, other tools (like TeamCity) can be set to run failing tests first.
I think your best bet would be to add your DB update code as part of the test suite setup (not part of your JUnit framework). Use the Exec Maven Plugin to call your DB code, binding it to the generate-test-resources phase. You'll want to make sure that when you run your tests, you actually call Maven to run the test.
JUnit does have the concept of an ExternalResource, which is a more explicit way of declaring the database dependency. It would be a few more lines of code than the base class, but depending on your perpective it may be more elegant.
Within Maven:
(1) Add the dbmaintain plugin: http://source.mysema.com/display/maven/Maven+Plugins
(2a) Call the appropriate goal (e.g. updateDatabase) explicitly before calling test
(2b) Or, if you want the dependency to be executed during a specific phase, then maven supports this, too: http://maven.apache.org/plugins/maven-dependency-plugin/usage.html
Then, you can connect Eclipse to these Maven changes:
How do I start Maven "compile" goal on save in Eclipse?
JUnit doesn't support test ordering. You will need to use TestNG for this. For example:
#Test(groups = "init")
public void initDatabase() { ... }
#Test(dependsOnGroups = "init")
public void test1() { ... }
#Test(dependsOnGroups = "init")
public void test2() { ... }
In this example, initDatabase() will be run first, and only if it succeeds will test1() and test2() be run. If initDatabase() fails, test1() and test2() will not run and they will be marked as "skipped" in the report.
Note also that you can add methods to any group at any time and the dependencies will keep working the way you expect them.
I was wondering how different surefire is when executing TestNG than TestNG ant tasks? The reason is that I am seeing consistent difference in behavior when trying to run a TestNG test that extends a JUnit test base (this is a workaround to run JBehave tests in TestNG described here: http://jbehave.org/documentation/faq/). Surefire detects my test as a JUnit test incorrectly (probably because its base is TestCase), while the Ant tasks run perfectly. Can anyone provide an insight into how TestNG handle both cases?
The test looks as follows:
public class YourScenario extends JUnitScenario {
#org.testng.annotations.Test
public void runScenario() throws Throwable {
super.runScenario();
}
}
The short answer is that the ant task is part of the TestNG distribution, so it's part of our tests and I always make sure that it remains up to date with TestNG.
Surefire is developed as part of the Maven project, and as such, it sometimes lags behind (and just like you, I have sometimes encountered bugs when running my tests with Surefire that didn't happen when running from the command line/ant/Eclipse).
I'll bring this question to the Maven team's attention, maybe they'll have more to say.
This looks to be a known bug: http://jira.codehaus.org/browse/SUREFIRE-575.
Have you tried using a TestNG XML suite definition instead of Surefire's automatic test case detection?