I have a JUnit class which deals with testing the clock in my system. The clock has a method jump(long milliSeconds which basically makes the clock jump to the specified time and thus sets the instance field of clock's currentTime to the parameter passed to the jump method.
So I have three JUnit methods. In the first I'm simply testing that the current time of the clock is 0, because I haven't invoked anything on the clock. Then I'm just testing that making the clock jump to a specified time once is reflected on the current time correctly. Lastly, I'm invoking the jump method a few time and after each jump I'm testing whether the current time is correct.
The problem I'm having is that sometimes my JUnit tests pass sometimes they fail. Assume I run the above three JUnit methods once, and it passes. That's fine. Then if I run the three again, then the first one fails because the current time of the clock is NOT 0 anymore but instead the current time is the last jump that was invoked in last test.
I'm confused about this because I thought that after executing all three JUnit methods sequentially it doesn't "remember" what it did if I run the tests again.
So does that require me to initialize the current time to be 0 in the #Before setUp() ? The thing is it's only sometimes that the above occurs. If I wait 5 minutes and run it again. It runs fine. Then if I immediately run it after again, I get the same error.
Has it perhaps something to do with the fact that I have declared the Clock class as final? Or that I have enforced the Singleton design pattern on it?
You designed the Clock as a singleton: there is only one instance of Clock per classloader: Clock.INSTANCE. So obviously, if a method of a test affects the state of the clock, the next method will find this clock with this new state.
You've just rediscovered one of the reasons why singleton is an anti-pattern: it's hard to unit-test. Every test should not assume anything about the clock, and put it in a well-known initial state before testing.
Or you could simply design the Clock as a plain old Java object that you can instantiate in the setup of your test, and use an IOC container to inject a unique instance of Clock in your beans at runtime.
Also, note that unit tests are supposed to be independant from each other. You might want to execute all tests, or only one of them, and they should be able to run in any order.
In JUnit the ordering of test-method invocations is not guaranteed, this is explained in the Junit FAQ section.
Your implementation is using a singleton, so as exposed before the order in which JUnit runs the test is going to affect the test results.
It's not a good practice that tests depends from each others, but if you need to ensure that you could use TestNG.
To fix that use a simple POJO class and let some DI framework handle that. If it's a constraint and you should preserve your class as it, test it with TestNG or rewrite your tests to ensure the order that you need. I.e: I think that a good test could be check that a lower specified time could throws an Exception.
Related
I have implemented a method that executes some logic, after a certain amount of time, using a TimerTask and Timer.schedule.
I want to verify this behaviour using Junit, however, I would like to know if there are better ways to test it, without using thread sleeping, or measuring time.
Thanks.
You can use a "own thread" excecutor service to get around the "multiple threads" complications.
You can further test that some class A pushes tasks into such a service; and you can also use unit tests to ensure that the parameters used when pushing tasks are what you expect them to be.
In other words: you really don't want to use unit tests to prove that scheduling is working (assuming that you didn't completely re-invent the wheel and you implemented your own scheduling ... which is something that you simply should not do). You want use unit tests to prove that your code is using existing (well tested) frameworks with the arguments you expect to see.
Suppose I want to manually run from my IDE (Intellij IDEA, or eclipse) 4000 JUnit tests; the first 1000 tests run pretty smoothly (say they take 3 minutes all 1000) but the test 1001 takes alone over 30 minutes.
Is there a way I can skip the test 1001 (while it's still running) and to let the test 1002 (and the others) keep going. I do not want to #Ignore the test 1001 and rerun the suite because I already have the answer for tests 1-1000; also I do not want to select tests 1001-4000 because it takes too much time.
I would some kind of button - Skip Current Test - which can be pressed when the test is running.
In case such feature does not exist, an enhancement for it needs to be done by the IDE developers or by JUnit developers?
This is actually pretty simple with JUnit 4 using Assume. Assume is a helper class like Assert. The difference is that Assert will make the test fail while Assume will skip it.
The common use case is Assume.assumeTrue( isWindows() ) for tests that only work on, say, a Windows file system.
So what you can do is define a system property skipSlowTests and add
Assume.assumeTrue( Boolean.getBoolean("skipSlowTests") )
at the beginning of slow tests that you usually want to skip. Create an Eclipse launch configuration which defines the property to true and you have a convenient way to switch between the two.
If you want to run a slow test, select the method in Eclipse (or the whole class) and use "Run as JUnit Test" from the context menu. Since the property is false by default, the tests will be run.
No, you cannot skip tests if they are already running.
What I suggest you do is use Categories to separate your slow tests from the rest of your tests.
For example:
public interface SlowTests {
}
public class MyTest {
#Test
public void test1{
}
#Category(SlowTests.class)
#Test
public void test1001{
// this is a slow test
}
}
Create a test suite for the fast tests.
#RunWith(Categories.class)
#ExcludeCategory(SlowTests.class)
#SuiteClasses(MyTest.class)
public class FastTestSuite {
}
Now execute the FastTestSuite if you don't want to run the slow tests (e.g. test1001). Execute MyTest as normal if you want to run all the tests.
What you're asking for is to stop executing your code while it is in mid test. You can't stop executing a current test without having hooks in your code to allow it. Your best solution is to use Categories as others have suggested.
Basically, JUnit executes all of the #Before methods (including #Rules), then your #Test method, then the #After methods (again, including #Rules). Even assuming that JUnit had a mechanism for stopping execution of it's bits of the code (which it doesn't), most of the time is spent in your code. So to 'skip' a test which has already started requires you to modify your test code (and potentially the code that it's testing) in order that you can cleanly stop it. Cleanly stopping an executing thread is a question in itself [*].
So what are your options?
Run the tests in parallel, then you don't have to wait as long for the tests to finish. This may work, but parallelizing the tests may well be a lot of work.
Stop execution of the tests, and fix the one that's you're working on. Most IDEs have an option to kill the JVM in which the tests are running. This is definitely the easiest option.
Implement your own test runner, which runs the test in a separate thread. This test runner then either waits for the thread to finish executing, or checks a flag somewhere which would be a signal for it to stop. This sounds complicated, because you need t manage your threads but also to set the flag in a running jvm. Maybe creating a file somewhere? This runner would then fail the currently running test, and you could move on to the next. Please note that 'stopping' a test midway may leave stuff in an inconsistent state, or you may end up executing stuff in parallel.
There are parallel JUnit runners out there, and I don't think you're going to get much help from IDE developers (at least in the short term). Also, look at TestNG, which allows stuff to be run in parallel.
For using categories, one solution I use is to run the long running tests separately using maven surefire or similar, not through the IDE. This involves checking out the source code somewhere else on my machine and building there.
[*]: Java, how to stop threads, Regarding stopping of a thread
I think a more common solution is to have two test suites: one for the fast tests and another for the slow ones. This is typically the way you divide unit tests (fast) and integration tests (slow).
It's highly unlikely that you'll get modifications to JUnit or IntelliJ for something like this. Better to change the way you use them - it'll get you to an answer faster.
You can modify your thest and do something like
public void theTest(){
if (System.getProperty("skipMyTest") == null){
//execute the test
}
}
and pass the environment variable if you want to skip the test
I have recently joined a group with some severe JUnit testing issues. One problem is an 8 minute long test! The test has several sections; each makes calls to org.springframework.context.ApplicationEventPublisher.publishEvent()
followed by Thread.sleep() of various amounts of time, then tests conditions.
There are several obvious problems with this approach, the timing of the Thread.sleep() calls is fragile:
tests occasionally fail on busy machines; and
tests take far too long when they do not fail.
Is the pool upon which these events are handled accessible for testing and is there a call to see if the event cascade has quiesced?
Worth mentioning is that test code that actually calls external services are integration tests and not unit tests. If you're truly unit testing here you should replace those calls with mocks. That way you can better control the values returned to your business logic and test for specific conditions. Also, as you've seen, this all but eliminates false positives due to external (non-code) situations. Obviously these tests aren't failing, the facility they expect to use is.
You can overwrite the default applicationEventMulticaster by adding this bean id to your application context.
Instead of the default SimpleApplicationEventMulticaster, you could set a TaskExecutor on this bean to perform the event publishing asynchronously in multiple threads.
Or you could implement your own multicaster, which prints out which event listener took so long or was blocking, for how long and on which events. That could help you to track down the real problem of the 8-Minute-Testcase.
Interestingly, the JavaDoc of the SimpleApplicationEventMulticaster, which is used by default by Spring when you are using ApplicationContext, states the following:
By default, all listeners are invoked in the calling thread. This allows the danger of a rogue listener blocking the entire application, but adds minimal overhead. Specify an alternative TaskExecutor to have listeners executed in different threads, for example from a thread pool.
I (intentionally) avoid Spring so I'm not sure I can help with the specifics but just looking at the sleep issue, you can use something like WaitFor in tempus-fugit (shameless plug) to poll for a Condition rather than "sleep and hope". It's not ideal and usually a change in the way you test (as suggested before) is preferable but it does mean you get finer grained "waits" which are more likely to avoid race-conditions / flaky tests and generally speed up the test.
See the project's documentation for details and post back if you find it useful!
I'd like to know if there are some unit testing frameworks which are capable of writing multi-threaded tests easily?
I would imagine something like:
invoke a special test method by n threads at the same time for m times. After all test threads finished, an assertion method where some constraints should be validated would be invoked.
My current approach is to create Thread objects inside a junit test method, loop manually the real test cases inside each run() method, wait for all threads and then validate the assertions. But using this, I have a large boilerplate code block for each test.
What are your experiences?
There is ConTest, and also GroboUtils.
I've used GroboUtils many years ago, and it did the job. ConTest is newer, and would be my preferred starting point now, since rather than just relying on trial and error, the instrumentation forces specific interleavings of the threads, providing a deterministic test. In contrast, GroboUtils MultiThreadedTestRunner simply runs the tests and hopes the scheduler produces an interleaving that causes the thread bug to appear.
EDIT: See also ConcuTest which also forces interleavings and is free.
There is also MultithreadedTC by Bill Pugh of FindBugs fame.
Just using the concurrency libraries would simplify your code. You can turn your boiler plate code into one method.
Something like
public static void runAll(int times, Runnable... tests) {
}
This post started out as "What are some common patterns in unit testing multi-threaded code ?", but I found some other discussions on SO that generally agreed that "It is Hard (TM)" and "It Depends (TM)". So I thought that reducing the scope of the question would be more useful.
Background : We are implementing a simple scheduler that gives you a way to register callbacks when starting and stopping jobs and of course configure the frequency of scheduling. Currently, we're making a lightweight wrapper around java.util.Timer.
Aspects:
I haven't found a way to test this scheduler by relying on only public interfaces (something like addJob(jobSchedule, jobArgs,jobListener) , removeJob(jobId)).
How do I time the fact that the the job was called according to the schedule specified ?
you could use a recorder object that record the order, timings and other useful stuff in each unit test of your scheduler. The test is simple:
create a recorder object
configure the schedule
execute a unit test
check that recorder object is "compatible" with the schedule
One thing also to remember is that you don't need to test that Timer works. You can write a mock version of Timer (by extending the class or using EasyMock) that simply checks that you are calling it correctly, possibly even replacing enough that you don't need threads at all. In this case that might be more work than needed if your job listener has enough callbacks to track the scheduler.
The other important thing to remember is that when testing the scheduler, use custom jobs that track how the scheduler is working; when testing scheduled jobs, call the callbacks directly and not through the scheduler. You may have a higher level integration test that checks both together, depending on the system.
There are many failure modes that such a scheduler could exhibit, and each would most likely require its own test case. These test cases are likely to be very different, so "it depends."
For testing concurrent software in Java in general, I recommend this presentation from JavaOne 2007: Testing Concurrent Software.
For testing that a scheduler must execute jobs in accurate accordance to their schedule, I'd create an abstraction of time itself. I've done something similar in one of my projects, where I have a Time or Clock interface. The default implementation will be MillisecondTime, but during testing I will switch it out with a TickTime. This implementation will allow my unit test to control when the time advances and by how much.
This way, you could write a test where a job is scheduled to run once every 10 tick. Then your test just advances the tick counter and checks to make sure that the jobs run at the correct ticks.
A couple of ways to test concurrent code.
run the same code many times under load, some bugs appear only occasionally, but can show up consistently if performed repeatedly.
Store the results of different threads/jobs in a collection such as a BlockingQueue. This will allow you to check the results in the current thread and finish in a timely manner (without ugly arbitrary sleep statements)
If you are finding testing concurrency difficult consider refactoring your objects/components to make them easier to test.
If the scheduler delegates to an Executor or ExecutorService to run the tasks, you could use Dependency Injection to remove a direct dependency on the type of Executor, and use a simple single threaded Executor to test much of the functionality of your scheduler without the complication of truly multi-threaded code. Once you'd got those tests debugged, you could move on the the harder, but now substantially reduced in magnitude, task of testing thread-safety.