Develop Reference

Gradle test - print stdout/stderror for failed tests

I'm introducing a Github actions pipeline to an existing project to run ./gradlew test. Unsurprisingly, I've run into cases where tests pass locally but not on the build machine, due to various things like mismatching timezones.
By default, gradle doesn't print the stdout for these tests. I am aware that it will do so if passed --info, however the test suite is some 1500 tests in size which makes the pipeline output extremely verbose (it actually makes my browser lag if I turn it on for the full suite and try to view the resulting output in Github).
To fix the initial teething problems, I've resorted to also targeting the suites that are failing (e.g. ./gradlew test --tests "foo.bar.AppTest" --info). This is a bit of a faff, though. Is there a way to tell gradle to print the stdout contents just for tests that have failed? This would put me in a much better position going forward!

This page contains what you are looking for.
It boils down to configuring the test task like so:
test {
testLogging {
// set options for log level LIFECYCLE
events "failed"
}
}
There are more options to finely control logging if you read that page.
Since you probably only need this for github actions, you can use the CI environmental variable to enable your configurations on CI environments only:
test {
doFirst {
if (System.getenv('CI')) {
testLogging {
// set options for log level LIFECYCLE
events "failed"
}
}
}
}
Other CI providers also set this environmental variable

As mentioned in this related answer when dealing with a multi-module android app the following can be used (root build.gradle)
// Call from root build.gradle
setupTestLogging()
fun Project.setupTestLogging() {
for (sub in subprojects) {
sub.tasks.withType<Test> {
testLogging {
exceptionFormat = TestExceptionFormat.FULL
}
}
}
}
(note that while exceptionFormat alone should be enough to get the wanted outcome, the events("standardOut" ...) mentioned above can be specified in the same way).
For mono-module android projects the same solution will work by dropping the part that iterates on the submodules

How can I use a custom runner when using categories in Junit?

I have a bunch of JUnit tests that extend my base test class called BaseTest which in turn extends Assert. Some of my tests have a #Category(SlowTests.class) annotation.
My BaseTest class is annotated with the following annotation #RunWith(MyJUnitRunner.class).
I've set up a Gradle task that is expected to run only SlowTests. Here's my Gradle task:
task integrationTests(type: Test) {
minHeapSize = "768m"
maxHeapSize = "1024m"
testLogging {
events "passed", "skipped", "failed"
outputs.upToDateWhen {false}
}
reports.junitXml.destination = "$buildDir/test-result"
useJUnit {
includeCategories 'testutils.SlowTests'
}
}
When I run the task, my tests aren't run. I've pinpointed this issue to be related to the custom runner MyJUnitRunner on the BaseTest. How can I set up my Gradle or test structure so that I can use a custom runner while using the Suite.

The solution to this turned out to smaller and trickier than I thought. Gradle was using my custom test runner and correctly invoking the filter method. However, my runner reloads all test classes through its own classloader for Javaassist enhancements.
This lead to the issue that SlowTest annotation was loaded through the Gradle classloader but when passed to my custom runner, the runner checked if the class was annotated with that annotation. This check never resolved correctly as the equality of the SlowTest annotation loaded through two different classloaders was different.
--
Since I've already done the research, I'll just leave this here. After days of digging through the Gradle and the (cryptic) JUnit sources, here's what I got.
Gradle simply doesn't handle any advanced JUnit functionality except the test categorization. When you create a Gradle task with the include-categories or the exclude-categories conditions, it builds a CategoryFilter. If you don't know, a Filter is what JUnit gives to the test-runner to decide whether a test or a test method should be filtered out. The test runner must implement the Filterable interface.
JUnit comes with multiple runners, the Categories is just another one of them. It extends a family of test runners called Suite. These suite based runners are designed to run a "suite" of tests. A suite of tests could be built by annotation introspection, by explicitly defining tests in a suite or any other method that builds a suite of tests.
In the case of the Categories runner, JUnit has it's own CategoryFilter but Gradle doesn't use that, it uses it's own CategoryFilter. Both provide more or less the same functionality and are JUnit filters so that can be used by any suite that implements Filterable.
The actual class in the Gradle responsible for running the JUnit tests is called JUnitTestClassExecuter. Once it has parsed the command line options it requests JUnit to check the runner should be used for a test. This method is invoked for every test as seen here.
The rest is simply up to JUnit. Gradle just created a custom RunNotifier to generate the standard XML files representing test results.
I hope someone finds this useful and saved themselves countless hours of debugging.
TLDR: You can use any runner in Gradle. Gradle has no specifics pertaining to runners. It is JUnit that decided the runners. If you'd like to know what runner will be used for your test, you can debug this by calling
Request.aClass(testClass).getRunner(). Hack this somewhere into your codebase and print it to the console. (I wasn't very successful in attaching a debugger to Gradle.)

Spock unit tests in grails for individual test does not work in intellij

I am trying to run individual spock unit tests using intellij idea.
Consider:
// rest of code
def "Test Something"() {
// test code below
}
In above test, when I goto the test body and right context menu, I get two kinds of tests for Test Something. One is the grails test and other is the junit test.
Referring to this question, the accepted answer recommends using the jUnit runner. But using it, the code simply does not compile(probably because certain plugins and other classes are not available).
(I am not sure though as this is the desired behavior because I am just running a single test and not all tests. So wonder why is it compiling all classes ,including plugin classes not required by the test target class.)
Using the grails runner, I check the configuration and here is the screenshot:
So nothing looks wrong with the command there.
But the test on running gives Test framework quit unexpectedly error.
I try running same command from grails console(CMD windows) and it runs without any error message.
But on checking the output html files(in target/test-reports) I see that none of the tests actually ran!
So what is going on here and why are not individual tests running?
PS:
When I run All tests using test-app command, tests run as expected. Only individual (unit)tests are not running.

Part of the price paid for Spock's nice test naming, is that you can't specify an individual test to run anymore.
Here are some articles about it. The first seems pretty on-point:
Run a specific test in a single test class with Spock and Maven
This one isn't about running a single test, but has some relevance and talks about Spock's test-name conversions, plus Peter Niederwieser chimes in with comments:
Can TestNG see my Spock (JUnit) test results?
A workaround for this could be the #IgnoreRest annotation. Simply annotate the test you want to run with #IgnoreRest, and then specify that test class to run, and only the annotated test will run. http://spockframework.github.io/spock/javadoc/1.0/spock/lang/IgnoreRest.html

Try using the grails unit test and add the following in the command line part:
-Dgrails.env=development
This will run the test as we change the running environment to development . Hope this will help to everyone facing such problems.

Custom JUnit test detection using gradle

Our test suite is growing quickly and we have reached a point where our more functional tests are dependent on other systems.
We use gradle test tasks to run these tests using the include and exclude filters but this is becoming cumbersome because we are having to name our tests in a particular way.
Our current approach is to name our tests in the following way:
class AppleSingleServiceTest {}
class BananaMultiServiceTest {}
class KiwiIntegrationTest {}
and then include tests in the relevant task using
include '**/*SingleServiceTest.class'
include '**/*MultiServiceTest.class'
include '**/*IntegrationTest.class'
Is it possible find test classes in gradle by looking at annotations?
#SingleServiceTest
public class AppleTest {}
I think any tests that are not annotated would then be run as normal unit tests, so if you forget to annotate a more functional test it will fail
An example of a single service test is a selenium test where any external dependencies of the SUT are stubbed
An example of a multi service test is one where some but maybe not all external dependencies are not stubbed

As of Gradle 1.6, Gradle supports selecting tests with JUnit #Category, like this:
test {
useJUnit {
includeCategories 'org.gradle.junit.CategoryA'
excludeCategories 'org.gradle.junit.CategoryB'
}
}
More details can be found in the docs.

The feature you are asking for doesn't currently exist, but you can make a feature request at http://forums.gradle.org. Or you can use the (cumbersome) JUnit #Category, which requires you to define test suites.

I had a similar need to filter tests with annotations. I eventually managed to create a solution. It is posted here.

JUnit & Integration tests - Is it possible to run one ahead of any test that is run

I have extracted all my integration tests out of my multi-module setup and put them all into a separate project. These integration tests are based on spring and a use a real database. I am using dbmaintain which is a database versioning tool, it automatically tracks which SQL files need to be applied and keeps the database in a correct state.
What I would like is to be able to run the code that ensures the database is up to date before any test is run. So if you run all the tests (from Eclipse or Maven in my case) that it will first perform the db check once, or if you run a single test it will first perform the db check. No matter how many tests are run, it should always run the db check.
Right now I am thinking that I will use #BeforeClass in the base test class (all tests ultimately extend from this class) which will instantiate a singleton to do it's work. That singleton will control everything to make sure things only get run once.
I am hoping there is a cleaner way.

By default, the Maven runner for JUnit reserves the right to reorder tests. This is actually a Good Thing(tm), because you can tell the Maven JUnit plugin to run tests in parallel, which means you wouldn't know the order anyways. In addition, other tools (like TeamCity) can be set to run failing tests first.
I think your best bet would be to add your DB update code as part of the test suite setup (not part of your JUnit framework). Use the Exec Maven Plugin to call your DB code, binding it to the generate-test-resources phase. You'll want to make sure that when you run your tests, you actually call Maven to run the test.
JUnit does have the concept of an ExternalResource, which is a more explicit way of declaring the database dependency. It would be a few more lines of code than the base class, but depending on your perpective it may be more elegant.

Within Maven:
(1) Add the dbmaintain plugin: http://source.mysema.com/display/maven/Maven+Plugins
(2a) Call the appropriate goal (e.g. updateDatabase) explicitly before calling test
(2b) Or, if you want the dependency to be executed during a specific phase, then maven supports this, too: http://maven.apache.org/plugins/maven-dependency-plugin/usage.html
Then, you can connect Eclipse to these Maven changes:
How do I start Maven "compile" goal on save in Eclipse?

JUnit doesn't support test ordering. You will need to use TestNG for this. For example:
#Test(groups = "init")
public void initDatabase() { ... }
#Test(dependsOnGroups = "init")
public void test1() { ... }
#Test(dependsOnGroups = "init")
public void test2() { ... }
In this example, initDatabase() will be run first, and only if it succeeds will test1() and test2() be run. If initDatabase() fails, test1() and test2() will not run and they will be marked as "skipped" in the report.
Note also that you can add methods to any group at any time and the dependencies will keep working the way you expect them.