Develop Reference

unit testing asynchronous code and code without externally visible effects

Actually, I have two questions, although a bit related:
I know that unit tests should test the public API. However, if I have a close method that closes sockets and shuts down executors, however, neither sockets nor executors are exposed to users of this API, should I test if this is done, or only that the method executed without error? Where is the borderline between public api/behavior and impl details?
if I test a method that performs some checks, then queues a task on executor service, then returns a future to monitor operation progress, should I test both this method and the task, even if the task itself is a private method or otherwise not exposed code? Or should I instead test only the public method, but arrange for the task to be executed in the same thread by mocking an executor? In the latter case, the fact that task is submitted to an executor using the execute() method would be an implementation detail, but tests would wait for tasks to complete to be able to check if the method along with it's async part works properly.

The only question you should ask yourself is this: will I or my colleagues be able to change the code with confidence without these tests frequently executed. If the answer is no - write and maintain the tests as they provide value.
With this in mind you may consider refactoring your code so that the "low level plumbing" (e.g. socket and thread management) lives in a separate module where you treat it explicitly as part of the contract that module provides.

Junit, testing timer.schedule without relying on thread sleeping or time

I have implemented a method that executes some logic, after a certain amount of time, using a TimerTask and Timer.schedule.
I want to verify this behaviour using Junit, however, I would like to know if there are better ways to test it, without using thread sleeping, or measuring time.
Thanks.

You can use a "own thread" excecutor service to get around the "multiple threads" complications.
You can further test that some class A pushes tasks into such a service; and you can also use unit tests to ensure that the parameters used when pushing tasks are what you expect them to be.
In other words: you really don't want to use unit tests to prove that scheduling is working (assuming that you didn't completely re-invent the wheel and you implemented your own scheduling ... which is something that you simply should not do). You want use unit tests to prove that your code is using existing (well tested) frameworks with the arguments you expect to see.

Unit testing parts of the application that use Thread Local

I am trying to implement unit testing in a web application and certain parts of it use ThreadLocal.
I cannot figure out how to go about testing it.
It looks like Junit runs all its tests using a single thread, namely the main thread.
I need to be able to assign different values to my ThreadLocal variable.
Has anyone come across such a scenario ? What do you guys recommend.

Groboutils has support for running multi-threaded tests, which will allow you to test your ThreadLocal variables.
http://groboutils.sourceforge.net/testing-junit/using_mtt.html

I would simply start threads within my unit test.
I recommend you use Futures and execute them using a ThreadPoolExecutor.

It might be enough to adorn the respective test methods with a timeout, i.e. #Test(timeout=100). Please note the "THREAD SAFETY WARNING" in Test::timeout , especially when you use #BeforeClass and #After annotations.

How to write multi-threaded unit tests?

I'd like to know if there are some unit testing frameworks which are capable of writing multi-threaded tests easily?
I would imagine something like:
invoke a special test method by n threads at the same time for m times. After all test threads finished, an assertion method where some constraints should be validated would be invoked.
My current approach is to create Thread objects inside a junit test method, loop manually the real test cases inside each run() method, wait for all threads and then validate the assertions. But using this, I have a large boilerplate code block for each test.
What are your experiences?

There is ConTest, and also GroboUtils.
I've used GroboUtils many years ago, and it did the job. ConTest is newer, and would be my preferred starting point now, since rather than just relying on trial and error, the instrumentation forces specific interleavings of the threads, providing a deterministic test. In contrast, GroboUtils MultiThreadedTestRunner simply runs the tests and hopes the scheduler produces an interleaving that causes the thread bug to appear.
EDIT: See also ConcuTest which also forces interleavings and is free.

There is also MultithreadedTC by Bill Pugh of FindBugs fame.

Just using the concurrency libraries would simplify your code. You can turn your boiler plate code into one method.
Something like
public static void runAll(int times, Runnable... tests) {
}

What are some strategies to unit test a scheduler?

This post started out as "What are some common patterns in unit testing multi-threaded code ?", but I found some other discussions on SO that generally agreed that "It is Hard (TM)" and "It Depends (TM)". So I thought that reducing the scope of the question would be more useful.
Background : We are implementing a simple scheduler that gives you a way to register callbacks when starting and stopping jobs and of course configure the frequency of scheduling. Currently, we're making a lightweight wrapper around java.util.Timer.
Aspects:
I haven't found a way to test this scheduler by relying on only public interfaces (something like addJob(jobSchedule, jobArgs,jobListener) , removeJob(jobId)).
How do I time the fact that the the job was called according to the schedule specified ?

you could use a recorder object that record the order, timings and other useful stuff in each unit test of your scheduler. The test is simple:
create a recorder object
configure the schedule
execute a unit test
check that recorder object is "compatible" with the schedule

One thing also to remember is that you don't need to test that Timer works. You can write a mock version of Timer (by extending the class or using EasyMock) that simply checks that you are calling it correctly, possibly even replacing enough that you don't need threads at all. In this case that might be more work than needed if your job listener has enough callbacks to track the scheduler.
The other important thing to remember is that when testing the scheduler, use custom jobs that track how the scheduler is working; when testing scheduled jobs, call the callbacks directly and not through the scheduler. You may have a higher level integration test that checks both together, depending on the system.

There are many failure modes that such a scheduler could exhibit, and each would most likely require its own test case. These test cases are likely to be very different, so "it depends."
For testing concurrent software in Java in general, I recommend this presentation from JavaOne 2007: Testing Concurrent Software.
For testing that a scheduler must execute jobs in accurate accordance to their schedule, I'd create an abstraction of time itself. I've done something similar in one of my projects, where I have a Time or Clock interface. The default implementation will be MillisecondTime, but during testing I will switch it out with a TickTime. This implementation will allow my unit test to control when the time advances and by how much.
This way, you could write a test where a job is scheduled to run once every 10 tick. Then your test just advances the tick counter and checks to make sure that the jobs run at the correct ticks.

A couple of ways to test concurrent code.
run the same code many times under load, some bugs appear only occasionally, but can show up consistently if performed repeatedly.
Store the results of different threads/jobs in a collection such as a BlockingQueue. This will allow you to check the results in the current thread and finish in a timely manner (without ugly arbitrary sleep statements)
If you are finding testing concurrency difficult consider refactoring your objects/components to make them easier to test.

If the scheduler delegates to an Executor or ExecutorService to run the tasks, you could use Dependency Injection to remove a direct dependency on the type of Executor, and use a simple single threaded Executor to test much of the functionality of your scheduler without the complication of truly multi-threaded code. Once you'd got those tests debugged, you could move on the the harder, but now substantially reduced in magnitude, task of testing thread-safety.