Develop Reference

Run an initializer in the very beginning of `gradle test` for JUnit 4

In our project, we have a lot of JUnit 4 tests, and they are executed from Gradle.
Now, we wanted to insert an initializer for the entire "gradle test" execution. The initializer is a class-loading tweak, so it should run just once in the very beginning.
I have looked into some:
JUnit's RunListener
RunListener does not run on Gradle.
https://github.com/gradle/gradle/issues/1330
Similar with junit-foundation
I was trying this, but it didn't work for unsure reasons yet.
It might be because of its complicated classloading structure.
Still worth going deeper into it.
JUnit's RunWith
We need to specify all the test class names.
We already have a lot of test classes.
And, when we add a new test class, we have to add the test class there, too. We easily forget.
JUnit 4: Set up things in a test suite before tests are run (like a test's #BeforeClass method, just for a test suite)
We wanted all the test classed to be looked up automatically (like normal gradle test).
How do I Dynamically create a Test Suite in JUnit 4?
ClasspathSuite may help to auto-lookup test classes?
https://github.com/takari/takari-cpsuite
But, gradle tests --tests="..." would not work as intended.
Static initializer in test classes, to run just once through the entire test run?
Need to add so many static {} blocks in so many existing tests?
Easily to forget, too.
Does anyone have any more ideas to solve this? (If we have a way in JUnit 5, we'll consider that.)

How can I use a custom runner when using categories in Junit?

I have a bunch of JUnit tests that extend my base test class called BaseTest which in turn extends Assert. Some of my tests have a #Category(SlowTests.class) annotation.
My BaseTest class is annotated with the following annotation #RunWith(MyJUnitRunner.class).
I've set up a Gradle task that is expected to run only SlowTests. Here's my Gradle task:
task integrationTests(type: Test) {
minHeapSize = "768m"
maxHeapSize = "1024m"
testLogging {
events "passed", "skipped", "failed"
outputs.upToDateWhen {false}
}
reports.junitXml.destination = "$buildDir/test-result"
useJUnit {
includeCategories 'testutils.SlowTests'
}
}
When I run the task, my tests aren't run. I've pinpointed this issue to be related to the custom runner MyJUnitRunner on the BaseTest. How can I set up my Gradle or test structure so that I can use a custom runner while using the Suite.

The solution to this turned out to smaller and trickier than I thought. Gradle was using my custom test runner and correctly invoking the filter method. However, my runner reloads all test classes through its own classloader for Javaassist enhancements.
This lead to the issue that SlowTest annotation was loaded through the Gradle classloader but when passed to my custom runner, the runner checked if the class was annotated with that annotation. This check never resolved correctly as the equality of the SlowTest annotation loaded through two different classloaders was different.
--
Since I've already done the research, I'll just leave this here. After days of digging through the Gradle and the (cryptic) JUnit sources, here's what I got.
Gradle simply doesn't handle any advanced JUnit functionality except the test categorization. When you create a Gradle task with the include-categories or the exclude-categories conditions, it builds a CategoryFilter. If you don't know, a Filter is what JUnit gives to the test-runner to decide whether a test or a test method should be filtered out. The test runner must implement the Filterable interface.
JUnit comes with multiple runners, the Categories is just another one of them. It extends a family of test runners called Suite. These suite based runners are designed to run a "suite" of tests. A suite of tests could be built by annotation introspection, by explicitly defining tests in a suite or any other method that builds a suite of tests.
In the case of the Categories runner, JUnit has it's own CategoryFilter but Gradle doesn't use that, it uses it's own CategoryFilter. Both provide more or less the same functionality and are JUnit filters so that can be used by any suite that implements Filterable.
The actual class in the Gradle responsible for running the JUnit tests is called JUnitTestClassExecuter. Once it has parsed the command line options it requests JUnit to check the runner should be used for a test. This method is invoked for every test as seen here.
The rest is simply up to JUnit. Gradle just created a custom RunNotifier to generate the standard XML files representing test results.
I hope someone finds this useful and saved themselves countless hours of debugging.
TLDR: You can use any runner in Gradle. Gradle has no specifics pertaining to runners. It is JUnit that decided the runners. If you'd like to know what runner will be used for your test, you can debug this by calling
Request.aClass(testClass).getRunner(). Hack this somewhere into your codebase and print it to the console. (I wasn't very successful in attaching a debugger to Gradle.)

Spock unit tests in grails for individual test does not work in intellij

I am trying to run individual spock unit tests using intellij idea.
Consider:
// rest of code
def "Test Something"() {
// test code below
}
In above test, when I goto the test body and right context menu, I get two kinds of tests for Test Something. One is the grails test and other is the junit test.
Referring to this question, the accepted answer recommends using the jUnit runner. But using it, the code simply does not compile(probably because certain plugins and other classes are not available).
(I am not sure though as this is the desired behavior because I am just running a single test and not all tests. So wonder why is it compiling all classes ,including plugin classes not required by the test target class.)
Using the grails runner, I check the configuration and here is the screenshot:
So nothing looks wrong with the command there.
But the test on running gives Test framework quit unexpectedly error.
I try running same command from grails console(CMD windows) and it runs without any error message.
But on checking the output html files(in target/test-reports) I see that none of the tests actually ran!
So what is going on here and why are not individual tests running?
PS:
When I run All tests using test-app command, tests run as expected. Only individual (unit)tests are not running.

Part of the price paid for Spock's nice test naming, is that you can't specify an individual test to run anymore.
Here are some articles about it. The first seems pretty on-point:
Run a specific test in a single test class with Spock and Maven
This one isn't about running a single test, but has some relevance and talks about Spock's test-name conversions, plus Peter Niederwieser chimes in with comments:
Can TestNG see my Spock (JUnit) test results?
A workaround for this could be the #IgnoreRest annotation. Simply annotate the test you want to run with #IgnoreRest, and then specify that test class to run, and only the annotated test will run. http://spockframework.github.io/spock/javadoc/1.0/spock/lang/IgnoreRest.html

Try using the grails unit test and add the following in the command line part:
-Dgrails.env=development
This will run the test as we change the running environment to development . Hope this will help to everyone facing such problems.

automatic inclusion of classes in a Java test suite

Test suites in Junit4 run nicely, but there's a snag here:
#RunWith(Suite.class)
#Suite.SuiteClasses({ A.class, B.class, ...})
If somebody develops a unit test and forgets to include it in Suite.SuiteClasses, that's obviously a problem.
(that's not a burning problem since Ant will catch that later but still)
So I wondered: if you have say "test" folder in Eclipse project and there are some packages with classes in it - is there a way to include them all automatically in junit4 test suite somehow?
(yes you can right-click "test" folder and do Run as Junit but that sometimes fails individual tests for some reason while they individually pass so I do not have much trust in this solution, plus test suites are nice toys to play with ;-)).

I suggest ClasspathSuite
import org.junit.extensions.cpsuite.ClasspathSuite;
import org.junit.runner.RunWith;
#RunWith(ClasspathSuite.class)
public class MySuite {}

Not an answer but more then comment:
[...] Run as Junit but that sometimes fails individual tests for some reason while they individually pass [...]
The reason is that some tests don't clean up correctly. This should always put you on alert. Try to identify a pair of tests that can't be executed in one "run" and have a close look at the first test. From my own experience: fix those issues as soon as possible (aka: NOW!), otherwise you may run in very deep problems later (typically: complaints from QA guys, like "the tests fail on my environment")

i don't agree with Andreas_D. it's not because tests don't clean after themselves. good unit tests don't have to clean after themselves. it's because your some of your tests depends on result of another. you need better tests and/or fixtures
however i agree with part 'fix them now!'. you have a serious problem when results of your tests are not reproducible

What can cause JUnit to disregard #Ignore annotations?

I just used MyEclipse to automatically generate some JUnit test cases. One of the generated methods looks like this:
#Ignore("Ignored") #Test
public void testCreateRevision()
{
fail("Not yet implemented"); // TODO
}
I added the #Ignore annotation manually. However, when I run the test, JUnit lists that method, and others like it, under "failures," rather than ignoring them (related: What's the difference between failure and error in JUnit?). And it displays the "Not yet implemented" message instead of the "Ignored" message. Clearly, fail() must be getting called, and therefore, the #Ignore assertion is not working.
What's going on here? Is there a setting I need to enable for this to work?
EDIT :
Things I have considered/tried so far:
I am using JUnit 4, so it's not a version problem.
I am importing org.junit.Ignore, so it's not a case of the wrong Ignore being used.
I have tried using #Ignore alone, #Ignore #Test and #Ignore("message") #Test; all fail.
EDIT 2 :
I created the tests with MyEclipse, via New > Other; Java > JUnit > JUnit Test Case; New JUnit 4 test, and the library in my build path is JUnit 4. I'm building with ant and actually running the case with MyEclipse.

Make sure you are importing the right #Ignore. To be sure use #org.junit.Ignore explicitly.
Double check if your test is being executed by JUnit 4, not 3. The easiest way to do this is to either change the test name so it is not prefixed by test (now it shouldn't be executed at all and JUnit 4 does not need this prefix anyway) or examine your test case inheritance hierarchy: it shouldn't extend directly or indirectly from junit.framework.TestCase (Junit 3 requirement).

I had this problem also even though JUnit 3 was not on my classpath. I believe that compatibility mode on Junit 4 picks up on the 'test' prefix in your testname and thus operates as JUnit 3 would rather than picking up the #Ignore. The solution is to rename your test.

In JUnit5 use #Disabled or #Disabled("Reason text")

Are you sure the test classes were recompiled?
It's a quite common problem, that the recompilation fails because there was typo somewhere in the sources (like a missing semicolon), and the IDE does not tell you that compiling failed.
Try deleting the target/test-classes folder.

In my case I found my IDE was executing the test disregarding the #Ignore annotation. When I ran mvn install (or any other maven phase) the test was skipped and that's what I was actually aiming for (see attached ilustration).
Ilustration of ways of execution

I think its just #Ignore that will skip the test
JUnit Ignore

Ensuring the test class is also imported fixed the issue for me i.e.
import org.junit.Ignore;
import org.junit.Test;