Develop Reference

Best method to count byte codes executed for a Java code

I was trying to get timing data for various Java programs. Then I had to perform some regression analysis based on this timing data. Here are the two methods I used to get the timing data:
System.currentTimeMillis(): I used this initially, but I wanted the timing data to be constant when the same program was run multiple
times. The variation was huge in this case. When two instances of the
same code were executed in parallel, the variation was even more. So
I dropped this and started looking for some profilers.
-XX countBytecodes Flag in Hotspot JVM: Since the variation in timing data was huge, I thought of measuring the number of byte codes executed, when this code was executed. This should have given a more static count, when the same program was executed multiple times. But This also had variations. When the programs were executed sequentially, the variations were small, but during parellel runs of the same code, the variations were huge. I also tried compiling using -Xint, but the results were similar.
So I am looking for some profiler that could give me the count of byte codes executed when a code is executed. The count should remain constant (or correlation close to 1) across runs of the same program. Or if there could be some other metric based on which I could get timing data, which should stay almost constant across multiple runs.

I wanted the timing data to be constant when the same program was run multiple times
That is not possible on a real machine unless it is designed for hard real time system which your machine will almost certainly be not.
I am looking for some profiler that could give me the count of byte codes executed when a code is executed.
Assuming you could do this, it wouldn't prove anything. You wouldn't be able to see for example that ++ is 90x cheaper than % depending on the hardware you run it on. You won't be able to see that a branch miss of an if is up to 100x more expensive than a speculative branch. You wouldn't be able to see that a memory access to an area of memory which triggers a TLB miss can be more expensive than copying 4 KB of data.
if there could be some other metric based on which I could get timing data, which should stay almost constant across multiple runs.
You can run it many times and take the average. This will hide any high results/outliers and give you a favourable idea of throughput. It can be a reproducible number for a given machine, if run long enough.

Is execution time of same program for multiple time give different result?

I have measure the execution time of my program using System.nanoTime() function. For every execution it is giving different execution time. Also i have measure the no of clock cycles by multiplying it with the processor speed. And due to different execution time the no of clock cycles coming is also different. I don't know whether it is correct or i am doing wrong somewhere. Plz suggest the answer.

What you are doing is wrong. Your line of questioning indicates you do not understand how processors and clock cycles work, even at a rudimentary level.
There is no way to measure processor speed by timing how long it takes to execute a java program.

You have several errors in your assumption:
First the value returned by System.nanoTime() has a resolution of nanoseconds but not necessarily the precision. It is possible that the value is only updated once per millisecond (this is implementation dependent).
Next is that you're meassuring wall clock time. The time elapsed depends highly on your system load. You never know which processes also take processing time or not. Therefore you cannot calculate from the wall time to any execution speed directly.
Third you assume that a certain number of Java commands equal always to the same number of processor instructions. This is wrong. E.g. Hotspot can reorder your commands or even compile your code in the meantime and optimize the compiled code several times.

Measuring method time

I want to optimize a method so it runs in less time. I was using System.currentTimeMillis() to calculate the time it lasted.
However, I just read the System.currentTimeMillis() Javadoc and it says this:
This method shouldn't be used for measuring timeouts or other elapsed
time measurements, as changing the system time can affect the results.
So, if I shouldn't use it to measure the elapsed time, how should I measure it?

Android native Traceview will help you measuring the time and also will give you more information.
Using it is as simple as:
// start tracing to "/sdcard/calc.trace"
Debug.startMethodTracing("calc");
// ...
// stop tracing
Debug.stopMethodTracing();
A post with more information in Android Developers Blog
Also take #Rajesh J Advani post into account.

There are a few issues with System.currentTimeMillis().
if you are not in control of the system clock, you may be reading the elapsed time wrong.
For server code or other long running java programs, your code is likely going to be called in over a few thousand iterations. By the end of this time, the JVM will have optimized the bytecode to the extent where the time taken is actually a lot lesser than what you measured as part of your testing.
It doesn't take into account the fact that there might be other processes on your computer or other threads in the JVM that compete for CPU time.
You can still use the method, but you need to keep the above points in mind. As others have mentioned, a profiler is a much better way of measuring system performance.

Welcome to the world of benchmarking.
As others point out - techniques based on timing methods like currentTimeMillis will only tell you elapsed time, rather than how long the method spent in the CPU.
I'm not aware of a way in Java to isolate timings of a method to how long it spent on the CPU - the answer is to either:
1) If the method is long running (or you run it many times, while using benchmarking rules like do not discard every result), use something like the "time" tool on Linux (http://linux.die.net/man/1/time) who will tell you how long the app spent on the CPU (obviously you have to take away the overhead of the application startup etc).
2) Use a profiler as others pointed out. This has dangers such as adding too much overhead using tracing techniques - if it uses stack sampling, it won't be 100% accurate
3) Am not sure how feasible this is on android - but you could get your bechmark running on a quiet multicore system and isolate a core (or ideally whole socket) to only be able to run your application.

You can use something called System.nanoTime(). As given here
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/System.html#nanoTime()
As the document says
This method can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time.
Hope this will help.

SystemClock.elapsedRealtime()
Quoting words in the linked page: elapsedRealtime() and elapsedRealtimeNanos() return the time since the system was booted, and include deep sleep. This clock is guaranteed to be monotonic, and continues to tick even when the CPU is in power saving modes, so is the recommend basis for general purpose interval timing.

Change the thread count of test plan in JMeter, at run time

I want to change the number of threads for a JMeter test plan at runtime.
I have Googled my problem and found a proposed solution to use JMeter plugins. But in this solution I would have to schedule the thread group before running the test plan, which I don't want. I also found another potential solution which changes the property, but doesn't affect test plan behavior at run time.
Ultimately, what I am trying to do is change the thread number given in a thread group and have it immediately increase or decrease the number of threads in the current running test plan.
Is this possible?

IMHO that's just a fancy feature that has no real benefit when doing proper performance testing.
In order to generate relevant test output (report), you need repeatability, and clearly defined test methodology and scenarios. In order to compare impact of any application/server/infrastructure changes, you need repeatability.
What do you mean by
We can't predict the user of our site
That's why we do performance testing at the first place. To find out what is our application/infrastructure limit.
I.e. the most significant metric you can produce is how your application response time changes when number of parallel users change. But not change erratically, in run time.
With jMeter plugins' Ultimate thread group you can cover any imaginable scenario.

The short answer is: no, you cannot change the number of threads dynamically during runtime. Each thread count value is only read once when the test plan is first compiled and is not resolved again after this point, so it remains fixed.

This feature is indeed useful, and surprisingly difficult to implement even with commercial tools such as Loadrunner. I would compare it to finding a loudspeakers maximum volume. You would manually turn the volume up until it started to crackle, then turn it back down slightly to maintain that maximum volume. In the same way, to find the peak capacity of an application, you want control to 'turn the volume up' until errors are seen, then back it down slightly to see if it stabilizes. You can then maintain that load to find where the bottleneck is.
Anyway, to answer the question, what I have done in the past is use an external influence, such as a file name or similar. Then combine that with the thread unique reference you can control which threads run and which are held (by pausing or similar).
For example, if you start with 100 threads, then create a file called '5.txt' in a specific location, you can add code such that if the threads sees that it's own reference is equal to or lower than the number then it can run. If not then it drops into a pause. At the start of this example 5 threads would run, and 95 would pause. You can then rename the file to '25.txt', and threads 6 to 25 would start running. It would work the other way too, changing it to '20.txt' would mean threads 21-25 pause again.
The key is to start enough threads to exceed your expected peak.

you can change it based on a variable which you set in a startup thread. See below.
In Jmeter how do I set a variable number of threads using a beanshell sampler variable?
However once the thread group has started you can not modify it. To the guy who said this feature would not be useful I disagree. There are many types of load tests and they do not all have the same number of users running for the duration. Here are just 2 example types of enterprise load tests we conduct at the bank where I work:
duration test - same number of users run the entire time (possibly
with a short ramp-up period)
Break point test - ramp up the number of users incrementally till the
application breaks
Spike test - run with a constant number of users but sporadically
throw in a large number of users
A break point test ramps up the number of users until the application breaks (the point being to see how high your app can scale). You can sort of do this using the thread groups "ramp up period" property. If you set ramp up time to 1000 and the number of threads to 100 it will add 1 thread every 10 seconds.
Spike tests are like duration tests but at some intervals a large number of users log in. This is used to guage the applications response time during peak hours or how it will respond if you all of a sudden get a large number of users (a very real scenario).
I find that Jmeter does not handle all load test scenarios that are needed in enterprise load testing. One work around Im considering is to just start all the threads but find a way to make some of them sleep. So you could set the number of threads to 1000 but somehow make 980 of them sleep or do nothing. Then maybe when the time_in_seconds%5==0 (every 5 minutes) you allow the other threads to run - simulating a spike test. The idea is you can hard code the threads to 1000 and will always have 1000 threads running - but they don't all have to be doing something at all times.
(in other words you can probably find a way but you have to get creative)
Update:
I just found this plugin which allows different types of testing. Have not tried it yet but looks promising:
http://jmeter-plugins.org/wiki/ThroughputShapingTimer/

You can set/change the number of threads at runtime using command line option...
you can use function calls, or variable references to User Parameters (which in turn could be functions), or variable references to variables set up by functions earlier in the test. There's more than one way to do it.
Suppose you want to be able to vary the number of threads in a test plan. Choose a suitable property name, say group1.threads. Replace the thread count in the GUI (or the JMX, if you're feeling brave!) with the following function call:
Please set below property in JMeter thread group as below
${__property(group1.threads)}
Then, when starting JMeter, define the property on the command line:
jmeter -Jgroup1.threads=10

We can't predict the user of our site.
Sure you can. This is what the HTTP logs of your existing site are for. You can also use logs from tools like Omniture or your CDN logs. If you look at the combination of Actual user IP address, request and referer tags in the logs you will be able to build a traversal map of ever single user on your site. You will be able to profile the high hit unique leaf node pages of a given business process to understand how many times a particular business process happens an hour. You will be able to examine abandonment by taking a look at the funnel in tools such as Omniture. If you need tools for this analysis I recommend Splunk. It's easy to install and configure. Time to value is very fast.
The more log data you have which you are using to profile the closer you can come to actual for what users do during a day/week/month/spot sale/end of quarter/end of year/etc....You need to combine actual at a point in time with actual from earlier points in time to project growth over time since you will need to allow for growth in your performance testing model.
If you don't get the values right then the value of your test as a predictor of what will/can happen in production will be quite low. This is not a failure of any given tool, but a failure in process on the planning front for the actual load model used as part of the test requirements. If you cannot build these models then you need to pull someone into your team who can.
This ability to produce a valid load model independent of tool is the difference between tests which reduce risk and throwing load.

By enabling the BeanShell server you can vary properties at runtime.
Just enable it and telnet on port 9001 (warning: not secure!)
Based on a test I did, unfortnately, it appears that the thread count it's not applied at runtime. However you can still manipulate the load of the test by other means, for example apply a costant throughput timer parametrized with a property named "throughput" and vary it at runtime like this:
setprop("throughput","2000");
It's well explained in the guide.

Methods of limiting emulated cpu speed

I'm writing a MOS 6502 processor emulator as part of a larger project I've undertaken in my spare time. The emulator is written in Java, and before you say it, I know its not going to be as efficient and optimized as if it was written in c or assembly, but the goal is to make it run on various platforms and its pulling 2.5MHZ on a 1GHZ processor which is pretty good for an interpreted emulator. My problem is quite to the contrary, I need to limit the number of cycles to 1MHZ. Ive looked around but not seen many strategies for doing this. Ive tried a few things including checking the time after a number of cycles and sleeping for the difference between the expected time and the actual time elapsed, but checking the time slows down the emulation by a factor of 8 so does anyone have any better suggestions or perhaps ways to optimize time polling in java to reduce the slowdown?

The problem with using sleep() is that you generally only get a granularity of 1ms, and the actual sleep that you will get isn't necessarily even accurate to the nearest 1ms as it depends on what the rest of the system is doing. A couple of suggestions to try (off the top of my head-- I've not actually written a CPU emulator in Java):
stick to your idea, but check the time between a large-ish number of emulated instructions (execution is going to be a bit "lumpy" anyway especially on a uniprocessor machine, because the OS can potentially take away the CPU from your thread for several milliseconds at a time);
as you want to execute in the order of 1000 emulated instructions per millisecond, you could also try just hanging on to the CPU between "instructions": have your program periodically work out by trial and error how many runs through a loop it needs to go between instructions to "waste" enough CPU to make the timing work out at 1 million emulated instructions / sec on average (you may want to see if setting your thread to low priority helps system performance in this case).

I would use System.nanoTime() in a busy wait as #pst suggested earlier.
You can speed up the emulation by generating byte code. Most instructions should translate quite well and you can add a busy wait call so each instruction takes the amount of time the original instruction would have done. You have an option to increase the delay so you can watch each instruction being executed.
To make it really cool you could generate 6502 assembly code as text with matching line numbers in the byte code. This would allow you to use the debugger to step through the code, breakpoint it and see what the application is doing. ;)
A simple way to emulate the memory is to use direct ByteBuffer or native memory with the Unsafe class to access it. This will give you a block of memory you can access as any data type in any order.

You might be interested in examining the Java Apple Computer Emulator (JACE), which incorporates 6502 emulation. It uses Thread.sleep() in its TimedDevice class.

Have you looked into creating a Timer object that goes off at the cycle length you need it? You could have the timer itself initiate the next loop.
Here is the documentation for the Java 6 version:
http://download.oracle.com/javase/6/docs/api/java/util/Timer.html