I want to measure the execution time of for loops on various platforms like php, c, python, Java, javascript... How can i measure it?
I know these platforms so i am talking about these:
for (i = 0; i < 1000000; i++)
{
}
I don't want to measure anything within the loop.
Little bit modification:
#all Some of the friends of mine are saying compiler will optimize this code making this loop a useless loop. I agree with this. we can add a small statement like some incremental statement, but the fact is I just want to calculate the execution time of per iteration in a loop in various languages. By adding a incremental statement will add up the execution time and that will effect the results, cause on various platforms, execution time for incrementing a value also differ and that will make a result useless.
In short, in better way I should ask:
I WANT TO CALCULATE THE EXECUTION TIME OF PER ITERATION IN A LOOP on Various PLATFORMS..HOW CAN DO THIS???
edit---
I came to know about Python Profilers
Profiler modules ...which evaluate cpu time... absolute time.. Any suggestions???Meanwhile i am working on this...
Although an answer has been given for C++, it looks from your description ("[You] don't want to measure anything within the loop") like you're trying to measure the time which it takes a program to iterate over an empty loop.
Please take care here: not only will it take varying times from different platforms and processors, but many compilers will optimise away such loops, effectively rendering the answer as "0" for any loop size.
Note that it also depends on what exactly you want to achieve: do you care about the time your program waits due to it being preempted by the system scheduler? All the solutions above take actual time elapsed into consideration, but that also involves the time when other processes run instead of your own.
If you don't care about this, all of the solutions above are good. If you do care, you probably need some profiling software to actually see how long the loop takes.
I'd start with a program that does nothing but your loop and (in a linux environment at least) do time you-prg-executable.
Then I'd investigate if there are tools which work like time. Not sure, but I'd look at JRat for java, and gcc's gcov for C and C++. No doubt there are similar tools for the other languages. But, of course, you need to see if they give actual time or not.
javascript
start = new Date;
for(var i = 0; i < 1000000; i++) {}
time = new Date - start;
The right way to do it in python is to run timeit from the command line:
$ python -m timeit "for i in xrange(100): pass"
100000 loops, best of 3: 2.5 usec per loop
Other version in PHP that doesn't require any extra stuff:
$start = microtime(true);
for (...) {
....
}
$end = microtime(true);
echo ($end - $start).' seconds';
For compiled languages such as C and C++, make sure that your compiler flags are set such that the loop isn't optimized away. With optimization switched on I would expect most compilers to detect that nothing is going on in the loop and optimize it away.
If you're using Python, you can use a module specifically built for timing things. It's called Timeit.
Here are a couple of references I found (just Googled it):
Dive Into Python: Using the Timeit
Module
Python Documentation:
Timeit Module
And here's some example code to get you started quickly:
import timeit
t = timeit.Timer("for i in range(100): pass", "")
# Timeit will run the statement 1,000,000 times by default, and return the time it took for all the runs together (it doesn't try to average them out or anything).
t.timeit()
2.9035916423318398 # This is the result. Don't forget (like I did in an earlier edit) that this is the result of running the code 1,000,000 times!
in php: (code timer)
$timer = new timer();
$timer->start();
for(i=0;i<1000000;i++)
{
}
$timer->stop();
echo $timer->getTime();
I asked the same question a while back that was specifically for the c++ language.
Heres the answer I ended up using:
#include <omp.h>
// Starting the time measurement
double start = omp_get_wtime();
// Computations to be measured
...
// Measuring the elapsed time
double end = omp_get_wtime();
// Time calculation (in seconds)
The result wont make sense in an empty loop, sine most compilers will optimize it at compiling time, before the runtime.
To compare languages speed, you will need a real algorithm, like "Merge Sort", "Binary Search" or maybe "Dijkstra" if you want something complicated.
Implement the same algorithm in all languages then compare.
Here is a benchmark on a Bio-Informatics algorithm. link text Check the Results page
To the point: just get the current time before doing something (this is the start time) and get the current time after doing something (this is the end time) and then just do the primary school math to get the elapsed time. Every API provides ways to get the current time. In Java for example it's System.currentTimeMillis() and System.nanoTime().
But: especially in Java, the elapsed time isn't always that reliable. There can be microdifferences and it also depends much on how you do the tests. I've seen circumstances where in test2() is faster than test1() because it is executed a bit later and that it become slower when you rearrange the execution to test2() and then test1().
Last but not least, micro-optimization is root of all evil.
For Java, both Apache Commons Lang and the Spring Framework have StopWatch (see the Java doc for Apache's here) classes that you can use as a way to measure execution time. Under the covers though it's just subtracting System.currentTimeMillis() and it doesn't save you that much code to use this utility.
Related
Is there a library that allows measuring Java Stream steps?
As in "when consuming this stream, X amount of time is spent in that filter, Y amount of time is spent in this map, etc..."?
Or do I have to tinker around a bit?
By definition, there is no 'between' time for steps. The method chain that you define only defines the steps to take (the methods build an internal structure containing the steps).
Once you call the terminal operator, all steps are executed in indeterminate order. If you never call a terminal operator, no steps are executed either.
What I mean by indeterminate order is this:
intstream.peek(operation1).peek(operation2).sum();
You don't know if operation1 is called, then operation2 called, then operation1 again, or if operation1 is called a bunch of times, then operation2 a bunch of times, or something else yet. (I hope I'm correct in saying this. [edit] correct) It's probably up to the implementation of the VM to trade off storage with time complexity.
The best you can to is measure the time each operation takes as you have full control over every operation. The time the stream engine takes is hard to gauge. However, know that when you do this, you're breaking the stream API requirement that operations should be side-effect-free. So don't use it in production code.
The only tool I can think of is JMH, which is a general benchmarking tool for Java. However note that it might be hard to understand the effect that different Stream operators have on the benchmark due to JIT. The fact that your code contains a Stream with a bunch of different operators doesn't mean that's exactly the way JIT compiles it.
Would anyone have an explanation, or even better a suggested fix, for why the time taken to execute Mockito mocks is so erratic? The simplest SSCCE I could come up with for this is below:
import static org.mockito.Mockito.mock;
public class TestSimpleMockTiming
{
public static final void main (final String args [])
{
final Runnable theMock = mock (Runnable.class);
int tookShort = 0;
int tookMedium = 0;
int tookLong = 0;
int tookRidiculouslyLong = 0;
long longest = 0;
for (int n = 0; n < 2000000; n++)
{
final long startTime = System.nanoTime ();
theMock.run ();
final long duration = System.nanoTime () - startTime;
if (duration < 1000000) // 0.001 seconds
tookShort++;
else if (duration < 100000000) // 0.1 seconds
tookMedium++;
else if (duration < 1000000000) // 1 second !!!
tookLong++;
else
tookRidiculouslyLong++;
longest = Math.max (longest, duration);
}
System.out.println (tookShort + ", " + tookMedium + ", " + tookLong + ", " + tookRidiculouslyLong);
System.out.println ("Longest duration was " + longest + " ns");
}
}
If I run this (from within Eclipse, using JDK 1.7.45 on Win 7 x64) typical output looks like:
1999983, 4, 9, 4
Longest duration was 5227445252 ns
So, while in the majority of situations the mock executes very fast, there's several executions that take even longer than 1 second. That's an eternity for a method that does nothing. From my experimenting with this, I don't believe the issue is the accuracy of System.nanoTime (), I think the mock really does take that long to execute. Is there anything I can do to improve on this and make the timing behave more consistently?
(FYI, why this is an issue is that I have a Swing app which contains various frames, and I try to write JUnit tests for the frames so that I can test that the layoutManagers behave correctly without having to fire up the whole app and navigate to the correct screen. In one such test, the screen uses a javax.swing.Timer to implement scrolling, so the display will pan around an area when the mouse is held near the end of the frame. I noticed the behaviour of this was very erratic, and the scrolling while usually fine would periodically freeze for up to a second and it looked dreadful. I wrote an SSCCE around this, thinking the problem was that Swing Timers can't be depended on to fire at a consistent rate, and in the SSCCE it worked perfectly.
After hours of tearing my hair out then trying to spot differences between my real code and the scrolling demo SSCCE, I started putting nano timers around blocks of code that ran repeatedly, noticed the time taken by my paintComponent method to be very erratic and eventually narrowed it down to a mock call. Testing the screen from running the real app, the scrolling behaves smoothly, its only a problem from the JUnit test because of the mock call, which led to me testing a simple mock in isolation with the SSCCE posted above.)
Many thanks!
This test is flawed in multiple ways. If you want to benchmark properly I strongly suggest that using JMH, it is done by someone Alexey Shipilev that is much smarter than us and definitely more knowledgeable on the JVM than most people doing Java on our beloved planet.
Here's the most notable way the test is flawed.
The test ignores what the JVM is doing, like the warmup phase, compilation C1 and C2 thread, GC, threading issues (even though this code is not multi-threaded, the JVM/OS may have to do something else) etc...
The test do seem to ignore if the actual OS/JVM/CPU combination offer a proper resolution up to the nanosecond.
Even though there's a System.nanoTime() are you sure the JVM and the OS have the proper resolution. On windows for example, there's the JVM don't have access to the the real nanosecond, but instead to some counter, not a wall-clock time. The javadoc states this, here's snippet :
This method can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time. The value returned represents nanoseconds since some fixed but arbitrary origin time (perhaps in the future, so values may be negative). The same origin is used by all invocations of this method in an instance of a Java virtual machine; other virtual machine instances are likely to use a different origin.
This method provides nanosecond precision, but not necessarily nanosecond resolution (that is, how frequently the value changes) - no guarantees are made except that the resolution is at least as good as that of currentTimeMillis().
The test also ignores how Mockito works.
Mockito stores every invocation in its own model in order to be able to verify these calls after executing the scenario. So on every iteration of the loop Mockito stores another invocation up to 2M invocations, which will impact the JVM (maybe the mock instance will hold several generations and promoted to the tenured which is definitely more costly for the GC). That means that the more the iterations the more this code stresses the JVM and not Mockito.
I believe it's not released (there's dev binaries on jcentral however), but Mockito will offer a setting to allow mockito to stub only hence it will not store invocations, which may allow Mockito to fit well in a scenario like this one.
The test lacks proper statistical analysis.
Interestingly enough the code of the test have a pseudo percentile approach. Which is good! Although it doesn't work like that and in this case it cannot work to catch the big issue. Instead it should record every measure in order to extract the tendencies of the evolution of the time mockito spent as the iteration count advances.
And if you want, it's good idea to store every recorded measure, so it would be possible to feed them to a proper statistical analysis tool like R in order extract a graph, percentile data, etc.
On that statistical matter it would certainly be interesting to use the HDRHistogram. Outside a microbenchmark of course as it will impact the memory and alter the result of the microbenchmark. Let's keep that for JMH.
Both point 1 and 2 can be addressed if you change the code to use JMH.
Hope that helps.
A JVM is a very complex thing that does a lot of optimization at runtime (including caching and byte code optimization). Thus, measuring execution time of Java programs, first of all you should do a warmup phase before doing your actual benchmark.
I expect that your first four runs took your longest profilling time and afterwards, the execution time became better and better.
Execute your benchmark a few hundreds or thousands times before you actually start profiling. Afterwards, I expect your measurement results should become more stable.
I have developed an image processing algorithm in core java (without using any third party API), Now I have to calculate execution time of that algorithm, for this i have used System.currentTimeMillis() like that,
public class MyAlgo {
public MyAlgo(String imagePath){
long stTime = System.currentTimeMillis();
// ..........................
// My Algorithm
// ..........................
long endTime = System.currentTimeMillis();
System.out.println("Time ==> " + (endTime - stTime));
}
public static void main(String args[]){
new MyAlgo("d:\\myImage.bmp");
}
}
But the problem is that each time I am running this program I am getting different execution time. Can anyone please suggest me that how can I do this?
If you don't want to use external profiling libraries just wrap your algorithm in a for() loop that executes it 1000 times and divide the total time by 1000. The result will be much more accurate since all the other tasks/processes will even out.
Note: The overall measure time will reflect the expected time of the algorithm to finish and not the total time that algorithms code instruction require.
For example if your algorithm uses a lot of memory and on average java VM calls garbage collector twice per each execution of algorithm - than you should take into account also the time of the garbage collector.
That is exactly what a for() loop does, so you will get good results.
You cannot get a reliable result from one execution alone; Java (well, JVMs) does runtime optimizations, plus there are other processes competing for CPU time/resource access. Also, are you sure your algorithm runs in constant time whatever the inputs?
Your best bet to have a calculation as reliable as possible is to use a library dedicated to performance measurements; one of them is caliper.
Set up a benchmark with different inputs/outputs etc and run it.
You need to apply some statistical analysis over multiple executions of your algorithm. For instance, execute it 1000 times and analyze min, max and average time.
Multiple executions in different scenarios might provide insights too, for instance, in different hardware or with images with different resolution.
I suppose your algorithm can be divided in multiple steps. You can monitor the steps independently to understand the impact of each one.
Marvin Image Processing Framework, for instance, provides methods to monitor and analyze the time and the number of executions of each algorithm step.
First of all I have to admit that these are very basic and primitive questions... I want to demonstrate different Algorithms in Java for sorting and searching, and to get a value for the runtime. There're issues I cannot solve:
there's Hotspot compiling - which is a runtime optimization I need to deactivate (I guess).
How do I get time-values (seconds) for runtimes? Starting a timer before the execution and stopping it afterwards... seems a little primitive. And the timer-object itself consumes runtime... I need to avoid that.
Is there anything in the Java API one could utilize to solve these problems?
Thanks,
Klaus
You can disable HotSpot with -Xint on the command line, to get an order of magnitude decrease in performance. However, why don't you want to measure real world performance? Different things can become bottlenecks when you compile.
Generally for microbenchmarks:
use System.nanoTime to get a time measurement at the start and end
run for a reasonable length of time
do the measurement a number of times over (there's some "warm up")
don't interleave measurements of different algorithms
don't do any I/O in the measured segment
use the result (HotSpot can completely optimise away trivial operations)
do it in a real world situation (or a cloae as possible)
remember dual core is the norm, and more cores will become normal
Use -Xint JVM flag. Other options can be seen here.
Use the ThreadMXBean API to get CPU/User times for your thread. An example can be seen here.
Using System.nanoTime() twice consumes less than 1 micro-second. I suggest you run any benchmark for a number of second and take an average so a micro-second error won't be significant.
Overall, I would suggest not making things more complicated than you need it to be.
To have a built in warm up I often ignore the first 10%-20% of iterations. Something like
long start;
int count;
for(int i = -count / 5; i < count; i++) {
if (count == 0) start = System.nanoTime();
// do tested code
}
long time = System.nanoTime() - start;
long average = time / count;
System.out.printf("Average time was %,d micro-seconds%n", average / 1000);
This question already has answers here:
Is stopwatch benchmarking acceptable?
(13 answers)
Closed 7 years ago.
I want to do some timing tests on a Java application. This is what I am currently doing:
long startTime = System.currentTimeMillis();
doSomething();
long finishTime = System.currentTimeMillis();
System.out.println("That took: " + (finishTime - startTime) + " ms");
Is there anything "wrong" with performance testing like this? What is a better way?
Duplicate: Is stopwatch benchmarking acceptable?
The one flaw in that approach is that the "real" time doSomething() takes to execute can vary wildly depending on what other programs are running on the system and what its load is. This makes the performance measurement somewhat imprecise.
One more accurate way of tracking the time it takes to execute code, assuming the code is single-threaded, is to look at the CPU time consumed by the thread during the call. You can do this with the JMX classes; in particular, with ThreadMXBean. You can retrieve an instance of ThreadMXBean from java.lang.management.ManagementFactory, and, if your platform supports it (most do), use the getCurrentThreadCpuTime method in place of System.currentTimeMillis to do a similar test. Bear in mind that getCurrentThreadCpuTime reports time in nanoseconds, not milliseconds.
Here's a sample (Scala) method that could be used to perform a measurement:
def measureCpuTime(f: => Unit): java.time.Duration = {
import java.lang.management.ManagementFactory.getThreadMXBean
if (!getThreadMXBean.isThreadCpuTimeSupported)
throw new UnsupportedOperationException(
"JVM does not support measuring thread CPU-time")
var finalCpuTime: Option[Long] = None
val thread = new Thread {
override def run(): Unit = {
f
finalCpuTime = Some(getThreadMXBean.getThreadCpuTime(
Thread.currentThread.getId))
}
}
thread.start()
while (finalCpuTime.isEmpty && thread.isAlive) {
Thread.sleep(100)
}
java.time.Duration.ofNanos(finalCpuTime.getOrElse {
throw new Exception("Operation never returned, and the thread is dead " +
"(perhaps an unhandled exception occurred)")
})
}
(Feel free to translate the above to Java!)
This strategy isn't perfect, but it's less subject to variations in system load.
The code shown in the question is not a good performance measuring code:
The compiler might choose to optimize your code by reordering statements. Yes, it can do that. That means your entire test might fail. It can even choose to inline the method under test and reorder the measuring statements into the now-inlined code.
The hotspot might choose to reorder your statements, inline code, cache results, delay execution...
Even assuming the compiler/hotspot didn't trick you, what you measure is "wall time". What you should be measuring is CPU time (unless you use OS resources and want to include these as well or you measure lock contestation in a multi-threaded environment).
The solution? Use a real profiler. There are plenty around, both free profilers and demos / time-locked trials of commercials strength ones.
Using a Java Profiler is the best option and it will give you all the insight that you need into the code. viz Response Times, Thread CallTraces, Memory Utilisations, etc
I will suggest you JENSOR, an open source Java Profiler, for its ease-of-use and no overheads on CPU. You can download it, instrument the code and will get all the info you need about your code.
You can download it from: http://jensor.sourceforge.net/
Keep in mind that the resolution of System.currentTimeMillis() varies between different operating systems. I believe Windows is around 15 msec. So if your doSomething() runs faster than the time resolution, you'll get a delta of 0. You could run doSomething() in a loop multiple times, but then the JVM may optimize it.
Have you looked at the profiling tools in netbeans and eclipse. These tools give you a better handle on what is REALLY taking up all the time in your code. I have found problems that I did not realize by using these tools.
Well that is just one part of performance testing. Depending on the thing you are testing you may have to look at heap size, thread count, network traffic or a whole host of other things. Otherwise I use that technique for simple things that I just want to see how long they take to run.
That's good when you are comparing one implementation to another or trying to find a slow part in your code (although it can be tedious). It's a really good technique to know and you'll probably use it more than any other, but be familiar with a profiling tool as well.
I'd imagine you'd want to doSomething() before you start timing too, so that the code is JITted and "warmed up".
Japex may be useful to you, either as a way to quickly create benchmarks, or as a way to study benchmarking issues in Java through the source code.