Develop Reference

How does Gradle's java plugin run JUnit tests in parallel: one queue per fork or a single queue and multiple forks?

I have been recently trying Gradle, I didn't have any prior experience with it, and so far I have been able to do the things I wanted to and am satisfied with the results. However, in my case I have to run Selenium tests with JUnit and some of them are disproportionately larger than others (i.e.: 25min vs 4min).
When using the maxParallelForks option, sometimes it takes longer than I would expect, as the tests seem to be assigned beforehand to the forks and sometimes I end up with iddle forks, while one of them is stuck with a long test, and when it finishes other shorter tests run after it (which could have run in any of the other available forks).
TL;DR:
When running tests in parallel, Gradle seems to assign tests as if there were multiple queues (one per fork) and I would like it to be like a single queue where the forks take the next test in the queue.
As an off-topic example, my situation is like being stuck in a queue at the supermarket, when the ones next to you are empty, but you can't change.

it's the latter. gradle uses one queue and distributes each entry clockwise to the running processes. Say you have 4 tests:
Test1 taking 10s
Test2 taking 1s
Test3 taking 10s
Test4 taking 1s
and using maxParallelForks = 2 the overall test task execution would be around 20s. I guess we need to discuss if this can be improved by getting notified about "free" processes to assign Test3 directly to test worker process 2 after Test2 comes back after 1s.

As of July 2021, Your question (and my experience) matches the issue described in this issue which
"has been automatically closed due to inactivity".
The issue is not resolved, and yes, the system assigns tasks early and does not rebalance to use idle workers.

Can I skip a JUnit test while the test is running?

Suppose I want to manually run from my IDE (Intellij IDEA, or eclipse) 4000 JUnit tests; the first 1000 tests run pretty smoothly (say they take 3 minutes all 1000) but the test 1001 takes alone over 30 minutes.
Is there a way I can skip the test 1001 (while it's still running) and to let the test 1002 (and the others) keep going. I do not want to #Ignore the test 1001 and rerun the suite because I already have the answer for tests 1-1000; also I do not want to select tests 1001-4000 because it takes too much time.
I would some kind of button - Skip Current Test - which can be pressed when the test is running.
In case such feature does not exist, an enhancement for it needs to be done by the IDE developers or by JUnit developers?

This is actually pretty simple with JUnit 4 using Assume. Assume is a helper class like Assert. The difference is that Assert will make the test fail while Assume will skip it.
The common use case is Assume.assumeTrue( isWindows() ) for tests that only work on, say, a Windows file system.
So what you can do is define a system property skipSlowTests and add
Assume.assumeTrue( Boolean.getBoolean("skipSlowTests") )
at the beginning of slow tests that you usually want to skip. Create an Eclipse launch configuration which defines the property to true and you have a convenient way to switch between the two.
If you want to run a slow test, select the method in Eclipse (or the whole class) and use "Run as JUnit Test" from the context menu. Since the property is false by default, the tests will be run.

No, you cannot skip tests if they are already running.
What I suggest you do is use Categories to separate your slow tests from the rest of your tests.
For example:
public interface SlowTests {
}
public class MyTest {
#Test
public void test1{
}
#Category(SlowTests.class)
#Test
public void test1001{
// this is a slow test
}
}
Create a test suite for the fast tests.
#RunWith(Categories.class)
#ExcludeCategory(SlowTests.class)
#SuiteClasses(MyTest.class)
public class FastTestSuite {
}
Now execute the FastTestSuite if you don't want to run the slow tests (e.g. test1001). Execute MyTest as normal if you want to run all the tests.

What you're asking for is to stop executing your code while it is in mid test. You can't stop executing a current test without having hooks in your code to allow it. Your best solution is to use Categories as others have suggested.
Basically, JUnit executes all of the #Before methods (including #Rules), then your #Test method, then the #After methods (again, including #Rules). Even assuming that JUnit had a mechanism for stopping execution of it's bits of the code (which it doesn't), most of the time is spent in your code. So to 'skip' a test which has already started requires you to modify your test code (and potentially the code that it's testing) in order that you can cleanly stop it. Cleanly stopping an executing thread is a question in itself [*].
So what are your options?
Run the tests in parallel, then you don't have to wait as long for the tests to finish. This may work, but parallelizing the tests may well be a lot of work.
Stop execution of the tests, and fix the one that's you're working on. Most IDEs have an option to kill the JVM in which the tests are running. This is definitely the easiest option.
Implement your own test runner, which runs the test in a separate thread. This test runner then either waits for the thread to finish executing, or checks a flag somewhere which would be a signal for it to stop. This sounds complicated, because you need t manage your threads but also to set the flag in a running jvm. Maybe creating a file somewhere? This runner would then fail the currently running test, and you could move on to the next. Please note that 'stopping' a test midway may leave stuff in an inconsistent state, or you may end up executing stuff in parallel.
There are parallel JUnit runners out there, and I don't think you're going to get much help from IDE developers (at least in the short term). Also, look at TestNG, which allows stuff to be run in parallel.
For using categories, one solution I use is to run the long running tests separately using maven surefire or similar, not through the IDE. This involves checking out the source code somewhere else on my machine and building there.
[*]: Java, how to stop threads, Regarding stopping of a thread

I think a more common solution is to have two test suites: one for the fast tests and another for the slow ones. This is typically the way you divide unit tests (fast) and integration tests (slow).
It's highly unlikely that you'll get modifications to JUnit or IntelliJ for something like this. Better to change the way you use them - it'll get you to an answer faster.

You can modify your thest and do something like
public void theTest(){
if (System.getProperty("skipMyTest") == null){
//execute the test
}
}
and pass the environment variable if you want to skip the test

JUnit Tests: how to test results from a Thread

I call a method passing a parameter. If this parameter is equal to something particular then a thread is started doing something repeatedly until it is stopped. In every repetition some values are changed.
Is there any way to check these values from JUnit?

If you are spawning threads you are not unit testing anymore - you are integration testing. Refactor your code so that the logic that changes this 'value' can be tested without the thread spawning. If it works without spawning a thread then it will work when spawning threads (I know I've set myself up for a lecture on that one... You will need to make sure you are properly synchronizing any potentially shared variables and don't have any code that could cause a deadlock).
Without seeing the code it is difficult to try to suggest ways to test it. However, you are definitely not unit testing if you are spawning threads.

If you are trying to test to see if each iteration modified the values appropriately, then call the iteration code with the expected inputs and test the expected outputs. Test each peice in isolation:
pseudo java code:
for each (file : files) {
doSomething(file); // this updates some running totals or something
}
Then you want to write some unit tests that call your doSomething() on each input you want to test and see if the values update appropriately (mock where necessary). Then do an integration test where you let the thread spawn and check the resulting values.

How to write multi-threaded unit tests?

I'd like to know if there are some unit testing frameworks which are capable of writing multi-threaded tests easily?
I would imagine something like:
invoke a special test method by n threads at the same time for m times. After all test threads finished, an assertion method where some constraints should be validated would be invoked.
My current approach is to create Thread objects inside a junit test method, loop manually the real test cases inside each run() method, wait for all threads and then validate the assertions. But using this, I have a large boilerplate code block for each test.
What are your experiences?

There is ConTest, and also GroboUtils.
I've used GroboUtils many years ago, and it did the job. ConTest is newer, and would be my preferred starting point now, since rather than just relying on trial and error, the instrumentation forces specific interleavings of the threads, providing a deterministic test. In contrast, GroboUtils MultiThreadedTestRunner simply runs the tests and hopes the scheduler produces an interleaving that causes the thread bug to appear.
EDIT: See also ConcuTest which also forces interleavings and is free.

There is also MultithreadedTC by Bill Pugh of FindBugs fame.

Just using the concurrency libraries would simplify your code. You can turn your boiler plate code into one method.
Something like
public static void runAll(int times, Runnable... tests) {
}

Timing out tests in TestNG

Thanks to a library upgrade (easymock 2.2 -> 2.4), we're having tests that have started locking up. I'd like to have a time out on individual tests, all of them. The idea is to identify the locked up tests - we're currently guessing - and fix them.
Is this possible, preferably on a suite-wide level? We have 400 tests, doing this each method or even each class will be time consuming.

The suite tag can have the time-out attribute. This time-out will be used as default for all test methods.
This default time-out can than be overridden on a per test method basis.

If the Suite level turns out to be the wrong approach (i.e. "too wide a net", because you end up marking too much methods with a timeout limit), you need to define a custom IAnnotationTransformer which, for each illegible function, will give you the opportunity to modify a #Test annotation (with, for instance the setTimout() method).
(setTimout(0) cancels a timeout directive)

Very late, but: running jstack -l <PID> will give you the stack dump, which you can inspect to find which calls are stuck. You might want to sample a few times to be sure they're stuck.

You can do a search and replace for "#Test" with "#Test(timeout=)"
Should work to find the locked up test and can be undone after that.