I want to change the number of threads for a JMeter test plan at runtime.
I have Googled my problem and found a proposed solution to use JMeter plugins. But in this solution I would have to schedule the thread group before running the test plan, which I don't want. I also found another potential solution which changes the property, but doesn't affect test plan behavior at run time.
Ultimately, what I am trying to do is change the thread number given in a thread group and have it immediately increase or decrease the number of threads in the current running test plan.
Is this possible?
IMHO that's just a fancy feature that has no real benefit when doing proper performance testing.
In order to generate relevant test output (report), you need repeatability, and clearly defined test methodology and scenarios. In order to compare impact of any application/server/infrastructure changes, you need repeatability.
What do you mean by
We can't predict the user of our site
That's why we do performance testing at the first place. To find out what is our application/infrastructure limit.
I.e. the most significant metric you can produce is how your application response time changes when number of parallel users change. But not change erratically, in run time.
With jMeter plugins' Ultimate thread group you can cover any imaginable scenario.
The short answer is: no, you cannot change the number of threads dynamically during runtime. Each thread count value is only read once when the test plan is first compiled and is not resolved again after this point, so it remains fixed.
This feature is indeed useful, and surprisingly difficult to implement even with commercial tools such as Loadrunner. I would compare it to finding a loudspeakers maximum volume. You would manually turn the volume up until it started to crackle, then turn it back down slightly to maintain that maximum volume. In the same way, to find the peak capacity of an application, you want control to 'turn the volume up' until errors are seen, then back it down slightly to see if it stabilizes. You can then maintain that load to find where the bottleneck is.
Anyway, to answer the question, what I have done in the past is use an external influence, such as a file name or similar. Then combine that with the thread unique reference you can control which threads run and which are held (by pausing or similar).
For example, if you start with 100 threads, then create a file called '5.txt' in a specific location, you can add code such that if the threads sees that it's own reference is equal to or lower than the number then it can run. If not then it drops into a pause. At the start of this example 5 threads would run, and 95 would pause. You can then rename the file to '25.txt', and threads 6 to 25 would start running. It would work the other way too, changing it to '20.txt' would mean threads 21-25 pause again.
The key is to start enough threads to exceed your expected peak.
you can change it based on a variable which you set in a startup thread. See below.
In Jmeter how do I set a variable number of threads using a beanshell sampler variable?
However once the thread group has started you can not modify it. To the guy who said this feature would not be useful I disagree. There are many types of load tests and they do not all have the same number of users running for the duration. Here are just 2 example types of enterprise load tests we conduct at the bank where I work:
duration test - same number of users run the entire time (possibly
with a short ramp-up period)
Break point test - ramp up the number of users incrementally till the
application breaks
Spike test - run with a constant number of users but sporadically
throw in a large number of users
A break point test ramps up the number of users until the application breaks (the point being to see how high your app can scale). You can sort of do this using the thread groups "ramp up period" property. If you set ramp up time to 1000 and the number of threads to 100 it will add 1 thread every 10 seconds.
Spike tests are like duration tests but at some intervals a large number of users log in. This is used to guage the applications response time during peak hours or how it will respond if you all of a sudden get a large number of users (a very real scenario).
I find that Jmeter does not handle all load test scenarios that are needed in enterprise load testing. One work around Im considering is to just start all the threads but find a way to make some of them sleep. So you could set the number of threads to 1000 but somehow make 980 of them sleep or do nothing. Then maybe when the time_in_seconds%5==0 (every 5 minutes) you allow the other threads to run - simulating a spike test. The idea is you can hard code the threads to 1000 and will always have 1000 threads running - but they don't all have to be doing something at all times.
(in other words you can probably find a way but you have to get creative)
Update:
I just found this plugin which allows different types of testing. Have not tried it yet but looks promising:
http://jmeter-plugins.org/wiki/ThroughputShapingTimer/
You can set/change the number of threads at runtime using command line option...
you can use function calls, or variable references to User Parameters (which in turn could be functions), or variable references to variables set up by functions earlier in the test. There's more than one way to do it.
Suppose you want to be able to vary the number of threads in a test plan. Choose a suitable property name, say group1.threads. Replace the thread count in the GUI (or the JMX, if you're feeling brave!) with the following function call:
Please set below property in JMeter thread group as below
${__property(group1.threads)}
Then, when starting JMeter, define the property on the command line:
jmeter -Jgroup1.threads=10
We can't predict the user of our site.
Sure you can. This is what the HTTP logs of your existing site are for. You can also use logs from tools like Omniture or your CDN logs. If you look at the combination of Actual user IP address, request and referer tags in the logs you will be able to build a traversal map of ever single user on your site. You will be able to profile the high hit unique leaf node pages of a given business process to understand how many times a particular business process happens an hour. You will be able to examine abandonment by taking a look at the funnel in tools such as Omniture. If you need tools for this analysis I recommend Splunk. It's easy to install and configure. Time to value is very fast.
The more log data you have which you are using to profile the closer you can come to actual for what users do during a day/week/month/spot sale/end of quarter/end of year/etc....You need to combine actual at a point in time with actual from earlier points in time to project growth over time since you will need to allow for growth in your performance testing model.
If you don't get the values right then the value of your test as a predictor of what will/can happen in production will be quite low. This is not a failure of any given tool, but a failure in process on the planning front for the actual load model used as part of the test requirements. If you cannot build these models then you need to pull someone into your team who can.
This ability to produce a valid load model independent of tool is the difference between tests which reduce risk and throwing load.
By enabling the BeanShell server you can vary properties at runtime.
Just enable it and telnet on port 9001 (warning: not secure!)
Based on a test I did, unfortnately, it appears that the thread count it's not applied at runtime. However you can still manipulate the load of the test by other means, for example apply a costant throughput timer parametrized with a property named "throughput" and vary it at runtime like this:
setprop("throughput","2000");
It's well explained in the guide.
Related
Problem:
I'm looking for a programming language and runtime whose execution can be "timed" in steps of code.
To be more concrete I need a language runtime/interpreter that can execute like 100 steps(not lines). After the execution the called method (of the runtime) returns while keeping its state. Later you can tell the runtime to continue execution for another 100 steps and so on.
It's somewhat like a VM only for execution of a single program.
Question:
Are there any runtimes for given languages that fullfill those criteria?
Preferred languages are Julia(julialang.org) and Java but I'm looking forward to all tips you can have for me. (keywords for search, problems in realisation, partial solutions, other languages that support it etc)
What i need it for:
I'd like to create a mod for minecraft that has codeable blocks but in order to prevent the whole minecraft world to be stuck due to players mistake and to be able to save the game/state at any time i need to be able to execute code of these codeable blocks in a fixed amount of time and save the current state of that codeable block's runtime after any of these runs.
Aaron aka rapus95
A practical solution might be to use Java threads and have a timer thread interrupt the work thread when its time limit is reached. However, there is an interesting abstraction known as "engines" that can be implemented using call/cc in Scheme: http://www.scheme.com/tspl4/examples.html#g208 This lets you pair a piece of work to do (represented as a 0-argument procedure) with an amount of "fuel" that it's allowed to consume. The "thread" stops when it runs out of fuel.
I have a set of 196 test methods. The Execution time of these testcases vary every time I run it. It has been run in a controlled environment,Say,For garbage collection, I included null in teardown().
Every time before executing the tests, I also make sure CPU usage, Memory usage, Disk space, System load are same for every start.
Also,The time variation is not in any particular order. I need to know why don't we get stable execution time while executing the same test cases again?
I made 93 cases stable by including a warm up period in the class. Other cases are related to database connections (reading a data or updating a data in database). Is it possible to have same execution time every time i run these testcases. (Execution time refers to junit testcase execution time)
Two primary things come to mind with Java performance:
You need to warmup the JVM, your tests are being interpreted as bytecode and are at the mercy of the JVM. That means executing the same test upwards of thousands of times during the same run.
JUnit tests are not measured with much accuracy. In fact, it's pretty much impossible to get an exact performance reading, even with libraries build specifically for this. This is why taking an average of multiple samples is generally suggested.
Yet, this and those others suggested by Reto are just what could be causing variance by Java. Where a variance of milliseconds is more than expected. For an example of this, create a unit test that takes a thread and puts it to sleep for 10 ms. Watch as you're given results anywhere from 7 ms to 13 ms to 17 ms or more. It just isn't a reliable way to measure things.
If you're connecting to a network, uploading data to a database, etc. I can't speak on behalf of that, but you need to take into account the variance of those systems as well.
I would suggest breaking your three tests with the greatest variance into smaller blocks. Try and isolate where your biggest bottleneck is, then concentrate on optimizing that operation or set of operations. I would think that connecting to the database takes the greatest amount of time, next to that would likely be executing the query. But you should isolate the measurement of these operations to make sure of that.
I work at a retailer and we consider to introduce CQ5 as a CMS.
However, after doing some research and talking to consultants it turns out, that there may be things that may be "complicated". Perhaps one of you can shed a little light on this.
The first thing is, we were told that when you use the Multi Site Manager to create multi language pages (about 80 languages) the update process can be as slow as half an hour until a change is ultimately published. Did someone of you experience something similar?
The other thing is, that the TarOptimizer has pretty long running times. I was told that runs that take up to 24 hours are not uncommon. Again my question: Did someone of you had such a problem or has an explanation for this?
I am really looking forward to your response.
These are really 2 separate question, but I'll address them based on my experience.
The update process for creating new multi-language pages will vary based on the number of languages, and also the number of publish instances and web-servers (assuming you're using dispatcher to cache) you are running. This is because the replication process is where the bottleneck is (at least in my experience), and as such if you're trying to push out a large amount of content across a large number of publishers with a large number of front-end web-servers whose cache needs to be cleared, there will be some delay in getting this to happen since replication is an asynchronous process. The longest delay I've seen for this has been in the 10-15 minute range, that was with 12 publishers and 12 front end webservers, but this comes with the obvious caveat that your mileage may vary.
For the Tar Optimzation job, I'd encourage you to take a look at this page as it has a lot of good info about the Tar Optizer job and how to tune it. The job can take a long time to run when you have a large repository, especially on an instance with a large number of write operations, but the run times can be configured so that it only runs during a given time period, and it will pick up where it left off the night before if the total run time is longer than the allowed run time. By default, it runs from 2-5 am each night, so if it takes more than that 3 hour period, it will continue where it left off the next night, allowing it to optimize the entire repository over a period of a few days if needed.
I want to filter what classes are being cpu-profiled in Java VisualVm (Version 1.7.0 b110325). For this, I tried under Profiler -> Settings -> CPU-Settings to set "Profile only classes" to my package under test, which had no effect. Then I tried to get rid of all java.* and sun.* classes by setting them in "Do not profile classes", which had no effect either.
Is this simply a bug? Or am I missing something? Is there a workaround? I mean other than:
paying for a better profiler
doing sampling by hand (see One could use a profiler, but why not just halt the program?)
switch to the Call Tree view, which is no good since only the Profiler view gives me the percentages of consumed CPU per method.
I want to do this mainly to get halfway correct percentages of consumed CPU per method. For this, I need to get rid of the annoying measurements, e.g. for sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() (around 70%). Many users seem to have this problem, see e.g.
Java VisualVM giving bizarre results for CPU profiling - Has anyone else run into this?
rmi.transport.tcp.tcptransport Connectionhandler consumes much CPU
Can't see my own application methods in Java VisualVM.
The reason you see sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() in the profile is that you left the option Profile new Runnables selected.
Also, if you took a snapshot of your profiling session you would be able to see the whole callstack for any hotspot method - this way you could navigate from the run() method down to your own application logic methods, filtering out the noise generated by the Profile new Runnables option.
OK, since your goal is to make the code run as fast as possible, let me suggest how to do it.
I'm no expert on VisualVM, but I can tell you what works. (Only a few profilers actually tell you what you need to know, which is - which lines of your code are on the stack a healthy fraction of wall-clock time.)
The only measuring I ever bother with is some stopwatch on the overall time, or alternatively, if the code has something like a framerate, the number of frames per second. I don't need any sort of further precision breakdown, because it's at best a remote clue to what's wasting time (and more often totally irrelevant), when there's a very direct way to locate it.
If you don't want to do random-pausing, that's up to you, but it's proven to work, and here's an example of a 43x speedup.
Basically, the idea is you get a (small, like 10) number of stack samples, taken at random wall-clock times.
Each sample consists (obviously) of a list of call sites, and possibly a non-call site at the end.
(If the sample is during I/O or sleep, it will end in the system call, which is just fine. That's what you want to know.)
If there is a way to speed up your code (and there almost certainly is), you will see it as a line of code that appears on at least one of the stack samples.
The probability it will appear on any one sample is exactly the same as the fraction of time it uses.
So if there's a call site or other line of code using a healthy fraction of time, and you can avoid executing it, the overall time will decrease by that fraction.
I don't know every profiler, but one I know that can tell you that is Zoom.
Others may be able to do it.
They may be more spiffy, but they don't work any quicker or better than the manual method when your purpose is to maximize performance.
I'm working on a system at the moment. It's a complex system but it boils down to a Solver class with a method like this:
public int solve(int problem); // returns the solution, or 0 if no solution found
Now, when the system is up and running, a run time of about 5 seconds for this method is expected and is perfectly fast enough. However, I plan to run some tests that look a bit like this:
List<Integer> problems = getProblems();
List<Integer> solutions = new ArrayList<Integer>(problems.size);
Solver solver = getSolver();
for (int problem: problems) {
solutions.add(solver.solve(problem));
}
// see what percentage of solutions are zero
// get arithmetic mean of non-zero solutions
// etc etc
The problem is I want to run this on a large number of problems, and don't want to wait forever for the results. So say I have a million test problems and I want the tests to complete in the time it takes me to make a cup of tea, I have two questions:
Say I have a million core processor and that instances of Solver are threadsafe but with no locking (they're immutable or something), and that all the computation they do is in memory (i.e. there's no disk or network or other stuff going on). Can I just replace the solutions list with a threadsafe list and kick off threads to solve each problem and expect it to be faster? How much faster? Can it run in 5 seconds?
Is there a decent cloud computing service out there for Java where I can buy 5 million seconds of time and get this code to run in five seconds? What do I need to do to prepare my code for running on such a cloud? How much does 5 million seconds cost anyway?
Thanks.
You have expressed your problem with two major points of serialisation: Problem production and solution consumption (currently expressed as Lists of integers). You want to get the first problems as soon as you can (currently you won't get them until all problems are produced).
I am assuming as well that there is a correlation between the problem list order and the solution list order – that is solutions.get(3) is the solution for problems.get(3) – this would be a huge problem for parallelising it. You'd be better off having a Pair<P, S> of problem/solution so you don't need to maintain the correlation.
Parallelising the solver method will not be difficult, although exactly how you do it will depend a lot on the compute costs of each solve method (generally the more expensive the method the lower the overhead costs of parallelising, so if these are very cheap you need to batch them). If you end up with a distributed solution you'll have much higher costs of course. The Executor framework and the fork/join extensions would be a great starting point.
You're asking extremely big questions. There is overhead for threads, and a key thing to note is that they run in the parent process. If you wanted to run a million of these solvers at the same time, you'd have to fork them into their own processes.
You can use one program per input, and then use a simple batch scheduler like Condor (for Linux) or HPC (for Windows). You can run those on Amazon too, but there's a bit of a learning curve, it's not just "upload Java code & go".
Sure, you could use a standard worker-thread paradigm to run things in parallel. But there will be some synchronization overhead (e.g., updates to the solutions list will cause lock contention when everything tries to finish at the same time), so it won't run in exactly 5 seconds. But it would be faster than 5 million seconds :-)
Amazon EC2 runs between US$0.085 and US$0.68 per hour depending on how much CPU you need (see pricing). So, maybe about $120. Of course, you'll need to set up something separate to distribute your jobs across various CPUs. One option might be just to use Hadoop (see this question about whether Hadoop is right for running simulations.
You could read things like Guy Steele's talk on parallelism for more info on how to think parallel.
Use an appropriate Executor. Have a look at http://download.oracle.com/javase/6/docs/api/java/util/concurrent/Executors.html#newCachedThreadPool()
Check out these article on concurrency:
http://www.vogella.de/articles/JavaConcurrency/article.html
http://www.baptiste-wicht.com/2010/09/java-concurrency-part-7-executors-and-thread-pools/
Basically, Java 7's new Fork/Join model will work really well for this approach. Essentially you can set up your million+ tasks and it will spread them as best it can accross all available processors. You would have to provide your custom "Cloud" task executor, but it can be done.
This assumes, of course, that your "solving" algorithm is rediculously parallel. In short, as long as the Solver is fully self-contained, they should be able to be split among an arbitrary number of processors.