Develop Reference

Why is performance of executing Mockito mocks so erratic?

Would anyone have an explanation, or even better a suggested fix, for why the time taken to execute Mockito mocks is so erratic? The simplest SSCCE I could come up with for this is below:
import static org.mockito.Mockito.mock;
public class TestSimpleMockTiming
{
public static final void main (final String args [])
{
final Runnable theMock = mock (Runnable.class);
int tookShort = 0;
int tookMedium = 0;
int tookLong = 0;
int tookRidiculouslyLong = 0;
long longest = 0;
for (int n = 0; n < 2000000; n++)
{
final long startTime = System.nanoTime ();
theMock.run ();
final long duration = System.nanoTime () - startTime;
if (duration < 1000000) // 0.001 seconds
tookShort++;
else if (duration < 100000000) // 0.1 seconds
tookMedium++;
else if (duration < 1000000000) // 1 second !!!
tookLong++;
else
tookRidiculouslyLong++;
longest = Math.max (longest, duration);
}
System.out.println (tookShort + ", " + tookMedium + ", " + tookLong + ", " + tookRidiculouslyLong);
System.out.println ("Longest duration was " + longest + " ns");
}
}
If I run this (from within Eclipse, using JDK 1.7.45 on Win 7 x64) typical output looks like:
1999983, 4, 9, 4
Longest duration was 5227445252 ns
So, while in the majority of situations the mock executes very fast, there's several executions that take even longer than 1 second. That's an eternity for a method that does nothing. From my experimenting with this, I don't believe the issue is the accuracy of System.nanoTime (), I think the mock really does take that long to execute. Is there anything I can do to improve on this and make the timing behave more consistently?
(FYI, why this is an issue is that I have a Swing app which contains various frames, and I try to write JUnit tests for the frames so that I can test that the layoutManagers behave correctly without having to fire up the whole app and navigate to the correct screen. In one such test, the screen uses a javax.swing.Timer to implement scrolling, so the display will pan around an area when the mouse is held near the end of the frame. I noticed the behaviour of this was very erratic, and the scrolling while usually fine would periodically freeze for up to a second and it looked dreadful. I wrote an SSCCE around this, thinking the problem was that Swing Timers can't be depended on to fire at a consistent rate, and in the SSCCE it worked perfectly.
After hours of tearing my hair out then trying to spot differences between my real code and the scrolling demo SSCCE, I started putting nano timers around blocks of code that ran repeatedly, noticed the time taken by my paintComponent method to be very erratic and eventually narrowed it down to a mock call. Testing the screen from running the real app, the scrolling behaves smoothly, its only a problem from the JUnit test because of the mock call, which led to me testing a simple mock in isolation with the SSCCE posted above.)
Many thanks!

This test is flawed in multiple ways. If you want to benchmark properly I strongly suggest that using JMH, it is done by someone Alexey Shipilev that is much smarter than us and definitely more knowledgeable on the JVM than most people doing Java on our beloved planet.
Here's the most notable way the test is flawed.
The test ignores what the JVM is doing, like the warmup phase, compilation C1 and C2 thread, GC, threading issues (even though this code is not multi-threaded, the JVM/OS may have to do something else) etc...
The test do seem to ignore if the actual OS/JVM/CPU combination offer a proper resolution up to the nanosecond.
Even though there's a System.nanoTime() are you sure the JVM and the OS have the proper resolution. On windows for example, there's the JVM don't have access to the the real nanosecond, but instead to some counter, not a wall-clock time. The javadoc states this, here's snippet :
This method can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time. The value returned represents nanoseconds since some fixed but arbitrary origin time (perhaps in the future, so values may be negative). The same origin is used by all invocations of this method in an instance of a Java virtual machine; other virtual machine instances are likely to use a different origin.
This method provides nanosecond precision, but not necessarily nanosecond resolution (that is, how frequently the value changes) - no guarantees are made except that the resolution is at least as good as that of currentTimeMillis().
The test also ignores how Mockito works.
Mockito stores every invocation in its own model in order to be able to verify these calls after executing the scenario. So on every iteration of the loop Mockito stores another invocation up to 2M invocations, which will impact the JVM (maybe the mock instance will hold several generations and promoted to the tenured which is definitely more costly for the GC). That means that the more the iterations the more this code stresses the JVM and not Mockito.
I believe it's not released (there's dev binaries on jcentral however), but Mockito will offer a setting to allow mockito to stub only hence it will not store invocations, which may allow Mockito to fit well in a scenario like this one.
The test lacks proper statistical analysis.
Interestingly enough the code of the test have a pseudo percentile approach. Which is good! Although it doesn't work like that and in this case it cannot work to catch the big issue. Instead it should record every measure in order to extract the tendencies of the evolution of the time mockito spent as the iteration count advances.
And if you want, it's good idea to store every recorded measure, so it would be possible to feed them to a proper statistical analysis tool like R in order extract a graph, percentile data, etc.
On that statistical matter it would certainly be interesting to use the HDRHistogram. Outside a microbenchmark of course as it will impact the memory and alter the result of the microbenchmark. Let's keep that for JMH.
Both point 1 and 2 can be addressed if you change the code to use JMH.
Hope that helps.

A JVM is a very complex thing that does a lot of optimization at runtime (including caching and byte code optimization). Thus, measuring execution time of Java programs, first of all you should do a warmup phase before doing your actual benchmark.
I expect that your first four runs took your longest profilling time and afterwards, the execution time became better and better.
Execute your benchmark a few hundreds or thousands times before you actually start profiling. Afterwards, I expect your measurement results should become more stable.

Measuring method time

I want to optimize a method so it runs in less time. I was using System.currentTimeMillis() to calculate the time it lasted.
However, I just read the System.currentTimeMillis() Javadoc and it says this:
This method shouldn't be used for measuring timeouts or other elapsed
time measurements, as changing the system time can affect the results.
So, if I shouldn't use it to measure the elapsed time, how should I measure it?

Android native Traceview will help you measuring the time and also will give you more information.
Using it is as simple as:
// start tracing to "/sdcard/calc.trace"
Debug.startMethodTracing("calc");
// ...
// stop tracing
Debug.stopMethodTracing();
A post with more information in Android Developers Blog
Also take #Rajesh J Advani post into account.

There are a few issues with System.currentTimeMillis().
if you are not in control of the system clock, you may be reading the elapsed time wrong.
For server code or other long running java programs, your code is likely going to be called in over a few thousand iterations. By the end of this time, the JVM will have optimized the bytecode to the extent where the time taken is actually a lot lesser than what you measured as part of your testing.
It doesn't take into account the fact that there might be other processes on your computer or other threads in the JVM that compete for CPU time.
You can still use the method, but you need to keep the above points in mind. As others have mentioned, a profiler is a much better way of measuring system performance.

Welcome to the world of benchmarking.
As others point out - techniques based on timing methods like currentTimeMillis will only tell you elapsed time, rather than how long the method spent in the CPU.
I'm not aware of a way in Java to isolate timings of a method to how long it spent on the CPU - the answer is to either:
1) If the method is long running (or you run it many times, while using benchmarking rules like do not discard every result), use something like the "time" tool on Linux (http://linux.die.net/man/1/time) who will tell you how long the app spent on the CPU (obviously you have to take away the overhead of the application startup etc).
2) Use a profiler as others pointed out. This has dangers such as adding too much overhead using tracing techniques - if it uses stack sampling, it won't be 100% accurate
3) Am not sure how feasible this is on android - but you could get your bechmark running on a quiet multicore system and isolate a core (or ideally whole socket) to only be able to run your application.

You can use something called System.nanoTime(). As given here
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/System.html#nanoTime()
As the document says
This method can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time.
Hope this will help.

SystemClock.elapsedRealtime()
Quoting words in the linked page: elapsedRealtime() and elapsedRealtimeNanos() return the time since the system was booted, and include deep sleep. This clock is guaranteed to be monotonic, and continues to tick even when the CPU is in power saving modes, so is the recommend basis for general purpose interval timing.

Java - Measuring Method Execution Time

I am trying to measure the complexity of an algorithm using a timer to measure the execution time, whilst changing the size of the input array.
The code I have at the moment is rather simple:
public void start() {
start = System.nanoTime();
}
public long stop() {
long time = System.nanoTime() - start;
start = 0;
return time;
}
It appears to work fine, up until the size of the array becomes very large, and what I expect to be an O(n) complexity algorithm turns out appearing to be O(n^2). I believe that this is due to the threading on the CPU, with other processes cutting in for more time during the runs with larger values for n.
Basically, I want to measure how much time my process has been running for, rather than how long it has been since I invoked the algorithm. Is there an easy way to do this in Java?

Measuring execution time is a really interesting, but also complicated topic. To do it right in Java, you have to know a little bit about how the JVM works. Here is a good article from developerWorks about benchmarking and measuring. Read it, it will help you a lot.
The author also provides a small framework for doing benchmarks. You can use this framework. It will give you exaclty what you needs - the CPU consuming time, instead of just two time stamps from before and after. The framework will also handle the JVM warm-up and will keep track of just-in-time-compilings.
You can also use a performance monitor like this one for Eclipse. The problem by such a performance monitor is, that it doesn't perform a benchmark. It just tracks the time, memory and such things, that your application currently uses. But that's not a real measurement - it's just a snapshot at a specific time.

Benchmarking in Java is a hard problem, not least because the JIT can have weird effects as your method gets more and more heavily optimized. Consider using a purpose-built tool like Caliper. Examples of how to use it and to measure performance on different input sizes are here.

If you want the actual CPU time of the current thread (or indeed, any arbitrary thread) rather than the wall clock time then you can get this via ThreadMXBean. Basically, do this at the start:
ThreadMXBean thx = ManagementFactory.getThreadMXBean();
thx.setThreadCpuTimeEnabled(true);
Then, whenever you want to get the elapsed CPU time for the current thread:
long cpuTime = thx.getCurrentThreadCpuTime();
You'll see that ThreadMXBean has calls to get CPU time and other info for arbitrary threads too.
Other comments about the complexities of timing also apply. The timing of the individual invocation of a piece of code can depend among other things on the state of the CPU and on what the JIT compiler decides to do at that particular moment. The overall scalability behaviour of an algorithm is generally a trend that emerges across a number of invocations and you will always need to be prepared for some "outliers" in your timings.
Also, remember that just because a particular timing is expressed in nanoseconds (or indeed milliseconds) does not mean that the timing actually has that granularity.

Create quick/reliable benchmark with java?

I'm trying to create a benchmark test with java. Currently I have the following simple method:
public static long runTest(int times){
long start = System.nanoTime();
String str = "str";
for(int i=0; i<times; i++){
str = "str"+i;
}
return System.nanoTime()-start;
}
I'm currently having this loop multiple times within another loop that is happening multiple times and getting the min/max/avg time it takes to run this method through. Then I am starting some activity on another thread and testing again. Basically I am just wanting to get consistent results... It seems pretty consistent if I have the runTest loop 10 million times:
Number of times ran: 5
The max time was: 1231419504 (102.85% of the average)
The min time was: 1177508466 (98.35% of the average)
The average time was: 1197291937
The difference between the max and min is: 4.58%
Activated thread activity.
Number of times ran: 5
The max time was: 3872724739 (100.82% of the average)
The min time was: 3804827995 (99.05% of the average)
The average time was: 3841216849
The difference between the max and min is: 1.78%
Running with thread activity took 320.83% as much time as running without.
But this seems a bit much, and takes some time... if I try a lower number (100000) in the runTest loop... it starts to become very inconsistent:
Number of times ran: 5
The max time was: 34726168 (143.01% of the average)
The min time was: 20889055 (86.02% of the average)
The average time was: 24283026
The difference between the max and min is: 66.24%
Activated thread activity.
Number of times ran: 5
The max time was: 143950627 (148.83% of the average)
The min time was: 64780554 (66.98% of the average)
The average time was: 96719589
The difference between the max and min is: 122.21%
Running with thread activity took 398.3% as much time as running without.
Is there a way that I can do a benchmark like this that is both consistent and efficient/fast?
I'm not testing the code that is between the start and end times by the way. I'm testing the CPU load in a way (see how I'm starting some thread activity and retesting). So I think that what I'm looking for it something to substitute for the code I have in "runTest" that will yield quicker and more consistent results.
Thanks

In short:
(Micro-)benchmarking is very complex, so use a tool like the Benchmarking framework http://www.ellipticgroup.com/misc/projectLibrary.zip - and still be skeptical about the results ("Put micro-trust in a micro-benchmark", Dr. Cliff Click).
In detail:
There are a lot of factors that can strongly influence the results:
The accuracy and precision of System.nanoTime: it is in the worst case as bad as of System.currentTimeMillis.
code warmup and class loading
mixed mode: JVMs JIT compile (see Edwin Buck's answer) only after a code block is called sufficiently often (1500 or 1000 times)
dynamic optimizations: deoptimization, on-stack replacement, dead code elimination (you should use the result you computed in your loop, e.g. print it)
resource reclamation: garbace collection (see Michael Borgwardt's answer) and object finalization
caching: I/O and CPU
your operating system on the whole: screen saver, power management, other processes (indexer, virus scan, ...)
Brent Boyer's article "Robust Java benchmarking, Part 1: Issues" ( http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html) is a good description of all those issues and whether/what you can do against them (e.g. use JVM options or call ProcessIdleTask beforehand).
You won't be able to eliminate all these factors, so doing statistics is a good idea. But:
instead of computing the difference between the max and min, you should put in the effort to compute the standard deviation (the results {1, 1000 times 2, 3} is different from {501 times 1, 501 times 3}).
The reliability is taken into account by producing confidence intervals (e.g. via bootstrapping).
The above mentioned Benchmark framework ( http://www.ellipticgroup.com/misc/projectLibrary.zip) uses these techniques. You can read about them in Brent Boyer's article "Robust Java benchmarking, Part 2: Statistics and solutions" ( https://www.ibm.com/developerworks/java/library/j-benchmark2/).

Your code ends up testing mainly garbage collection performance because appending to a String in a loop ends up creating and immediately discarding a large number of increasingly large String objects.
This is something that inherently leads to wildly varying measurements and is influenced strongy by multi-thread activity.
I suggest you do something else in your loop that has more predictable performance, like mathematical calculations.

In the 10 million times run, odds are good the HotSpot compiler detected a "heavily used" piece of code and compiled it into machine native code.
JVM bytecode is interpreted, which leads it susceptible to more interrupts from other background processes occurring in the JVM (like garbage collection).
Generally speaking, these kinds of benchmarks are rife with assumptions that don't hold. You cannot believe that a micro benchmark really proves what it set out to prove without a lot of evidence proving that the initial measurement (time) isn't actually measuring your task and possibly some other background tasks. If you don't attempt to control for background tasks, then the measurement is much less useful.

Java task runtime

First of all I have to admit that these are very basic and primitive questions... I want to demonstrate different Algorithms in Java for sorting and searching, and to get a value for the runtime. There're issues I cannot solve:
there's Hotspot compiling - which is a runtime optimization I need to deactivate (I guess).
How do I get time-values (seconds) for runtimes? Starting a timer before the execution and stopping it afterwards... seems a little primitive. And the timer-object itself consumes runtime... I need to avoid that.
Is there anything in the Java API one could utilize to solve these problems?
Thanks,
Klaus

You can disable HotSpot with -Xint on the command line, to get an order of magnitude decrease in performance. However, why don't you want to measure real world performance? Different things can become bottlenecks when you compile.
Generally for microbenchmarks:
use System.nanoTime to get a time measurement at the start and end
run for a reasonable length of time
do the measurement a number of times over (there's some "warm up")
don't interleave measurements of different algorithms
don't do any I/O in the measured segment
use the result (HotSpot can completely optimise away trivial operations)
do it in a real world situation (or a cloae as possible)
remember dual core is the norm, and more cores will become normal

Use -Xint JVM flag. Other options can be seen here.
Use the ThreadMXBean API to get CPU/User times for your thread. An example can be seen here.

Using System.nanoTime() twice consumes less than 1 micro-second. I suggest you run any benchmark for a number of second and take an average so a micro-second error won't be significant.
Overall, I would suggest not making things more complicated than you need it to be.
To have a built in warm up I often ignore the first 10%-20% of iterations. Something like
long start;
int count;
for(int i = -count / 5; i < count; i++) {
if (count == 0) start = System.nanoTime();
// do tested code
}
long time = System.nanoTime() - start;
long average = time / count;
System.out.printf("Average time was %,d micro-seconds%n", average / 1000);