Suppose I want to manually run from my IDE (Intellij IDEA, or eclipse) 4000 JUnit tests; the first 1000 tests run pretty smoothly (say they take 3 minutes all 1000) but the test 1001 takes alone over 30 minutes.
Is there a way I can skip the test 1001 (while it's still running) and to let the test 1002 (and the others) keep going. I do not want to #Ignore the test 1001 and rerun the suite because I already have the answer for tests 1-1000; also I do not want to select tests 1001-4000 because it takes too much time.
I would some kind of button - Skip Current Test - which can be pressed when the test is running.
In case such feature does not exist, an enhancement for it needs to be done by the IDE developers or by JUnit developers?
This is actually pretty simple with JUnit 4 using Assume. Assume is a helper class like Assert. The difference is that Assert will make the test fail while Assume will skip it.
The common use case is Assume.assumeTrue( isWindows() ) for tests that only work on, say, a Windows file system.
So what you can do is define a system property skipSlowTests and add
Assume.assumeTrue( Boolean.getBoolean("skipSlowTests") )
at the beginning of slow tests that you usually want to skip. Create an Eclipse launch configuration which defines the property to true and you have a convenient way to switch between the two.
If you want to run a slow test, select the method in Eclipse (or the whole class) and use "Run as JUnit Test" from the context menu. Since the property is false by default, the tests will be run.
No, you cannot skip tests if they are already running.
What I suggest you do is use Categories to separate your slow tests from the rest of your tests.
For example:
public interface SlowTests {
}
public class MyTest {
#Test
public void test1{
}
#Category(SlowTests.class)
#Test
public void test1001{
// this is a slow test
}
}
Create a test suite for the fast tests.
#RunWith(Categories.class)
#ExcludeCategory(SlowTests.class)
#SuiteClasses(MyTest.class)
public class FastTestSuite {
}
Now execute the FastTestSuite if you don't want to run the slow tests (e.g. test1001). Execute MyTest as normal if you want to run all the tests.
What you're asking for is to stop executing your code while it is in mid test. You can't stop executing a current test without having hooks in your code to allow it. Your best solution is to use Categories as others have suggested.
Basically, JUnit executes all of the #Before methods (including #Rules), then your #Test method, then the #After methods (again, including #Rules). Even assuming that JUnit had a mechanism for stopping execution of it's bits of the code (which it doesn't), most of the time is spent in your code. So to 'skip' a test which has already started requires you to modify your test code (and potentially the code that it's testing) in order that you can cleanly stop it. Cleanly stopping an executing thread is a question in itself [*].
So what are your options?
Run the tests in parallel, then you don't have to wait as long for the tests to finish. This may work, but parallelizing the tests may well be a lot of work.
Stop execution of the tests, and fix the one that's you're working on. Most IDEs have an option to kill the JVM in which the tests are running. This is definitely the easiest option.
Implement your own test runner, which runs the test in a separate thread. This test runner then either waits for the thread to finish executing, or checks a flag somewhere which would be a signal for it to stop. This sounds complicated, because you need t manage your threads but also to set the flag in a running jvm. Maybe creating a file somewhere? This runner would then fail the currently running test, and you could move on to the next. Please note that 'stopping' a test midway may leave stuff in an inconsistent state, or you may end up executing stuff in parallel.
There are parallel JUnit runners out there, and I don't think you're going to get much help from IDE developers (at least in the short term). Also, look at TestNG, which allows stuff to be run in parallel.
For using categories, one solution I use is to run the long running tests separately using maven surefire or similar, not through the IDE. This involves checking out the source code somewhere else on my machine and building there.
[*]: Java, how to stop threads, Regarding stopping of a thread
I think a more common solution is to have two test suites: one for the fast tests and another for the slow ones. This is typically the way you divide unit tests (fast) and integration tests (slow).
It's highly unlikely that you'll get modifications to JUnit or IntelliJ for something like this. Better to change the way you use them - it'll get you to an answer faster.
You can modify your thest and do something like
public void theTest(){
if (System.getProperty("skipMyTest") == null){
//execute the test
}
}
and pass the environment variable if you want to skip the test
Related
I have implemented a method that executes some logic, after a certain amount of time, using a TimerTask and Timer.schedule.
I want to verify this behaviour using Junit, however, I would like to know if there are better ways to test it, without using thread sleeping, or measuring time.
Thanks.
You can use a "own thread" excecutor service to get around the "multiple threads" complications.
You can further test that some class A pushes tasks into such a service; and you can also use unit tests to ensure that the parameters used when pushing tasks are what you expect them to be.
In other words: you really don't want to use unit tests to prove that scheduling is working (assuming that you didn't completely re-invent the wheel and you implemented your own scheduling ... which is something that you simply should not do). You want use unit tests to prove that your code is using existing (well tested) frameworks with the arguments you expect to see.
I was writting JUnit test. I would like to know if the tests within a test class can run in parallel.
class TestMyClass {
#Test
public void test1() {
}
#Test
public void test2() {
}
}
Will Junit ever run test1() and test2() in parallel?
Consider TestNG if you are looking for Parallel tests execution.
Yes, you can. Take a look at this question for details on how to set that up. The correctness of your tests should not really on this behaviour though. Your tests should run correctly if they're run concurrently or not.
I cannot directly answer on whether jUnit will run them in parallel or not, but theoretically that shouldn't matter. The only thing you should keep in mind is the sequence of execution you can bet on, like
setup
execution of test
teardown
This should be enough, as each single test should be completely independent from each other. If your tests depend on the order they're executed or whether they run in parallel, then you probably have some wrong dependencies.
No, because a fixture is setUp before each test. Running test in parallel could change the fixture state. I guess you could write a test executor to run tests in parallel.
I want to stop/destroy a running JUnitCore, which is started with
JUnitCore.run(Request.aClass(ClassToRun));
Like pleaseStop() on the RunNotifier.
Any ideas?
http://junit.sourceforge.net/javadoc/org/junit/runner/package-summary.html
Option 1:
the best option is to write your own Runner implementation, inherited from org.junit.runners.BlockJUnit4ClassRunner and declare it in your execution context, for instance as Main Class of a raw Java command line.
Get inspired of JUnit source code (it is really small), mainly org.junit.runners.ParentRunner and override the runChildren method by your own to get the opportunity to exit the execution loop when a stop command has been triggered.
The factory for Runner is Request. To start, you invoke
(new JUnitCore()).run(Request.runner(new MyStoppableRunner().aClass(ClassToRun))
Option 2:
If using your own runner is not possible in your context (launched within Eclipse for instance), thanks to a RunListener implementation registered in the used runner, you can get a reference to the thread running your test case.
If stop command has been triggered, your listener may throw a RuntimeException or even an Error in the hope it will make the original test runner collapse.
Bonus
These two options are basic as the aim is to check a stop condition and do not go on looping on methods or tests classes.
You may want to try to interrupt the test thread if stuck in sleep or wait state. To do so a watch dog thread should be created to invoke interrupt on the test thread after an inactivity timeout.
as far as I know there 's no such method available.
So the first answer should be : impossible...
But with Java bytecode enhancement frameworks nothing is really impossible...
So I could advise you to write a Java Interface Closeable or something like this... use the Java bytecode enhancement framework of your choice (asm, bcel, javaassist or any other) and enhance the JunitCore class to implement this interface.Once done you will be able to stop this facade for your tests...
public interface Closeable{
public void stopMe();
}
Hi,bytecode enhancement is not the silver bullet of course....But forking and patching an Open Source project requires huge changes into your project management... Who will remeber this little patch 5 years and 3 releases later ? Adding a small class to enhance the bytecode to fulfill your needs is a pragmatic but as always not a perfect answer ....I am Ok with Yves that trying to add the feature into JUnit would be the best solution but it requires far more than technicaal knowledge... Of course you may encounter classloading weird problems while using such technique.... For integration testing I would suggest using TestNG rather than JUnit it provides many enhancements while providing a compatibility layer....
HTH
Jerome
I want to provide another simple solution to stop a JUnitCore:
JUnitCore jUnitCore = new JUnitCore();
Field field = JUnitCore.class.getDeclaredField("fNotifier");
field.setAccessible(true);
RunNotifier runNotifier = (RunNotifier) field.get(jUnitCore);
runNotifier.pleaseStop();
Credits to Matthew Farwell who transfered my idea into code.
I needed to stop all running processes/threads as I was running executing my test suite from a main method using java -jar test.jar within a Docker image. I couldn't then extract the correct exit code after the tests had finished. I went with this:
final JUnitCore engine = new JUnitCore();
engine.addListener(new TextListener(System.out));
final Result testsResult = engine.run(AllTestSuite.class);
if (testsResult.wasSuccessful()) {
System.out.println("Tests complete with success!!!");
System.exit(0);
}
System.out.println("Tests complete with "+ result.getFailureCount() + " failures!!!");
System.exit(1);
I call a method passing a parameter. If this parameter is equal to something particular then a thread is started doing something repeatedly until it is stopped. In every repetition some values are changed.
Is there any way to check these values from JUnit?
If you are spawning threads you are not unit testing anymore - you are integration testing. Refactor your code so that the logic that changes this 'value' can be tested without the thread spawning. If it works without spawning a thread then it will work when spawning threads (I know I've set myself up for a lecture on that one... You will need to make sure you are properly synchronizing any potentially shared variables and don't have any code that could cause a deadlock).
Without seeing the code it is difficult to try to suggest ways to test it. However, you are definitely not unit testing if you are spawning threads.
If you are trying to test to see if each iteration modified the values appropriately, then call the iteration code with the expected inputs and test the expected outputs. Test each peice in isolation:
pseudo java code:
for each (file : files) {
doSomething(file); // this updates some running totals or something
}
Then you want to write some unit tests that call your doSomething() on each input you want to test and see if the values update appropriately (mock where necessary). Then do an integration test where you let the thread spawn and check the resulting values.
I'd like to know if there are some unit testing frameworks which are capable of writing multi-threaded tests easily?
I would imagine something like:
invoke a special test method by n threads at the same time for m times. After all test threads finished, an assertion method where some constraints should be validated would be invoked.
My current approach is to create Thread objects inside a junit test method, loop manually the real test cases inside each run() method, wait for all threads and then validate the assertions. But using this, I have a large boilerplate code block for each test.
What are your experiences?
There is ConTest, and also GroboUtils.
I've used GroboUtils many years ago, and it did the job. ConTest is newer, and would be my preferred starting point now, since rather than just relying on trial and error, the instrumentation forces specific interleavings of the threads, providing a deterministic test. In contrast, GroboUtils MultiThreadedTestRunner simply runs the tests and hopes the scheduler produces an interleaving that causes the thread bug to appear.
EDIT: See also ConcuTest which also forces interleavings and is free.
There is also MultithreadedTC by Bill Pugh of FindBugs fame.
Just using the concurrency libraries would simplify your code. You can turn your boiler plate code into one method.
Something like
public static void runAll(int times, Runnable... tests) {
}