I'm just curious really, there doesn't seem to be any counter arguments. So should a profiler be used for everything? I'd imagine the performance gain would require make it beneficial every time.
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil"
Profile when needed, otherwise you will waste hours of your precious time on saving milliseconds of machine cheap time.
Profilers tend to slow your code down significantly, so they should definitely not be used in production code. I also wouldn't use one for development unless I was specifically looking for performance problems, since it increases the time taken in the code / test cycle.
If you have a very performance-critical application, then profile as much as possible during your development cycle so you can find your potential bottlenecks as early as possible and adjust your designs around them.
Otherwise, only profile when you run into an unexpected performance problem, have questions about the performance of specific algorithms or subsystems, or near the end of the development cycle if you have time leftover to work on improving performance even if it's not strictly necessary.
In most cases getting the application to work as expected on time is much more important than getting it to run fast, and profiling can eat up a lot of development time.
Profile periodically, as needed.
Don't worry about the performance of a profiler.
Its job is to identify your performance problem,
not to pretend you don't have one.
Actually...
I don't use profilers. I do this instead, because it works better.
Profilers only tell you about certain kinds of problems - those that can be localized to a function.
After you fix the problems they tell you about, you are left being limited by the ones they don't tell you about.
There's a PDF slide show here, showing how this works.
Related
Are there any known techniques (and resources related to them, like research papers or blog entries) which describe how do dynamically programatically detect the part of the code that caused a performance regression, and if possible, on the JVM or some other virtual machine environment (where techniques such as instrumentation can be applied relatively easy)?
In particular, when having a large codebase and a bigger number of committers to a project (like, for example, an OS, language or some framework), it is sometimes hard to find out the change that caused a performance regression. A paper such as this one goes a long way in describing how to detect performance regressions (e.g. in a certain snippet of code), but not how to dynamically find the piece of the code in the project that got changed by some commit and caused the performance regression.
I was thinking that this might be done by instrumenting pieces of the program to detect the exact method which causes the regression, or at least narrowing the range of possible causes of the performance regression.
Does anyone know about anything written about this, or any project using such performance regression detection techniques?
EDIT:
I was referring to something along these lines, but doing further analysis into the codebase itself.
Perhaps not entirely what you are asking, but on a project I've worked on with extreme performance requirements, we wrote performance tests using our unit testing framework, and glued them into our continuous integration environment.
This meant that every check-in, our CI server would run tests that validated we hadn't slowed down the functionality beyond our acceptable boundaries.
It wasn't perfect - but it did allow us to keep an eye on our key performance statistics over time, and it caught check-ins that affected the performance.
Defining "acceptable boundaries" for performance is more an art than a science - in our CI-driven tests, we took a fairly simple approach, based on the hardware specification; we would fail the build if the performance tests exceeded a response time of more than 1 second with 100 concurrent users. This caught a bunch of lowhanging fruit performance issues, and gave us a decent level of confidence on "production" hardware.
We explicitly didn't run these tests before check-in, as that would slow down the development cycle - forcing a developer to run through fairly long-running tests before checking in encourages them not to check in too often. We also weren't confident we'd get meaningful results without deploying to known hardware.
With tools like YourKit you can take a snapshot of the performance breakdown of a test or application. If you run the application again, you can compare performance breakdowns to find differences.
Performance profiling is more of an art than a science. I don't believe you will find a tool which tells you exactly what the problem is, you have to use your judgement.
For example, say you have a method which is taking much longer than it used to do. Is it because the method has changed or because it is being called a different way, or much more often. You have to use some judgement of your own.
JProfiler allows you to see list of instrumented methods which you can sort by average execution time, inherent time, number of invocations etc. I think if this information is saved over releases one can get some insight into regression. Offcourse the profiling data will not be accurate if the tests are not exactly same.
Some people are aware of a technique for finding (as opposed to measuring) the cause of excess time being taken.
It's simple, but it's very effective.
Essentially it is this:
If the code is slow it's because it's spending some fraction F (like 20%, 50%, or 90%) of its time doing something X unnecessary, in the sense that if you knew what it was, you'd blow it away, and save that fraction of time.
During the general time it's being slow, at any random nanosecond the probability that it's doing X is F.
So just drop in on it, a few times, and ask it what it's doing.
And ask it why it's doing it.
Typical apps are spending nearly all their time either waiting for some I/O to complete, or some library function to return.
If there is something in your program taking too much time (and there is), it is almost certainly one or a few function calls, that you will find on the call stack, being done for lousy reasons.
Here's more on that subject.
Is it possible that attaching a profiler to a JVM (let's say VisualVM) could make some methods run slower, while not effecting others and thus causing a skew in the results that makes it look like a certain piece of code is a hotspot when in fact it's not. I will ask specifically about reflection calls for an example. I'm running some code that shows a lot of time spent in Spring AOP calls (specifically invokeJoinpointUsingReflection) - which the author says runs fine in testing (using an in code microbenchmark) but when they profiled it showed this method to take longer then other non-reflection methods. (sorry if that' a little unclear) So it got my wondering if the profiler could really have this effect and lead a developer down a false trail. Feel free to answer with any examples, the reflection part is just my example.
Profilers regularly give mis-leading information, but in generally they are usually right. Where they tend to skew the result is in very simple methods which might be further optimised if profiling wasn't enabled.
If in doubt I suggest you use another profiler, such as YourKit (evalation version should be fine) It has more light weight recording, but can have the same issues.
Heisenberg famously observed that collecting information from a system always disturbs it, so you can't get an undisturbed observation. (Thus the software term, "Heisenbug"). Yes, collecting profiling information can cause the actual performance to be changed in ways that will misdirect you.
Whether that is true in a significant way for your particular JVM or profiler, and how much disturbance occurs, is a matter of engineering.
Most profilers are sample based, and thus the more data you collect, the more accurate the results are. As far as I know, there is no bias for or against methods written purely in Java.
Certain profilers require a calibration step, e.g. NetBeans and VisualVM. You might verify the vintage and settings for your chosen profiler.
I'm working on refactoring some code in Java, so I I'm timing things to make sure the code doesn't get any slower. However, the new refactored code seems to take more time than the original code. Remarkably, when I run the code with a profiler, the new code is significantly faster than the old code. The primary difference is that the old code is recursive, while the new code is iterative. Can a profiler affect the recursive code by a factor of several hundred thousand while only affecting the iterative code by a factor of 1.5?
I'm running on Mac OS X 10.6.6, 3 GB RAM, 2.4 GHz CPU, using the default Netbeans 6.9 profiler with Java 1.6.0__22 64-Bit Server.
(Both methods have self-timing code using System.currentTimeMillis() to allow me to compare the times when not using a profiler, but this shouldn't affect things noticeably.)
Yes. Most profiles do instrumentation at the level of method invocations. In recursive form, the profiler must take a lot more measurements than in iterative form. While profilers do try to extract their overhead from the reported numbers, this is very difficult to do reliable. Different profilers will be better/worse at this.
I'm working on refactoring some code in Java, so I I'm timing things to make sure the code doesn't get any slower. However, the new refactored code seems to take more time than the original code.
Yes. Code typically runs slower under a profiler.
Therefore, you should compare times of old / new version of your application, either both run under the profiler, or both run normally.
Also be aware that a profiler can actually distort performance characteristics. And different profilers may disagree about where the code hotspots are. So it is a good idea to run/compare versions of your application without profiling before you adopt an optimization that you are trialing.
(Both methods have self-timing code using System.currentTimeMillis() to allow me to compare the times when not using a profiler, but this shouldn't affect things noticeably.)
There are traps here too:
The best possible granularity of currentTimeMillis() is 1 millisecond, but on some OSes it might be tens of milliseconds.
If you don't take steps to avoid this, your manual timings can include distorting overheads such as the JIT compilation times and the GC times. The effects could be quite subtle.
I would say if you want to measure speed, just measure speed, don't profile. They're not the same thing. Instrumenting profilers put a lot of overhead into each function call, and if all you want is an overall speed difference, it won't be accurate because you're partly measuring the cost of the instrumentation itself.
If you want to find out what is taking the time, that is different from measuring. A wall-clock-time stack-sampling profiler (not instrumentation) that reports line-level percent is your best bet. It doesn't matter if it slows the program down, because it's purpose is not to measure speed; it's purpose is to find out where the time is going and why, on a percentage basis. It would be OK if it or something else slowed the program down by 10%, or 10 times, if it showed you where the time was being taken, independent of speed.
I only say this because lots of people are confused about this point, and the confusion gets solidified into lots of profilers.
More on that subject.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
First of all I should mention that I'm aware of the fact that performance optimizations can be very project specific. I'm mostly not facing these special issues right now. I'm facing a bunch of performance issues with the JVM itself.
I wonder now:
which code-optimization make sense
from a compiler perspective: for
example to support the garbage
collector I declared variables as
final - very much following PMD's
suggestions here from Eclipse.
what best practices there are for: vmargs,
heap and other stuff passed to the
JVM for initialization. How do I get
the right values here? Is there any
formula or is it try and error?
Java automates a lot, does many optimization on byte-code level and stuff. However I think most of that must be planed by a developer in order to work.
So how do you speed up your programs in Java? :)
Which code-optimization make sense from a compiler perspective: for example to support the garbage collector I declared variables as final - very much following PMD's suggestions here from Eclipse.
Assuming you are talking about potential micro-optimizations you can make to your code, the answer is pretty much none. The best way to increase your application performance is to run a profiler to figure out where the performance bottlenecks are, then figure out if there is anything you can do to speed them up.
All of the classic tricks like declaring classes, variables and methods final, reorganizing loops, changing primitive types are pretty much a waste of effort in most cases. The JIT compiler can typically do a much better job than you can. For example, recent JIT compilers will analyse all loaded classes to figure out which method calls are not subject to overloading, without you declaring the classes or methods as final. It will then use a quicker call sequence, or even inline the method body.
Indeed, the Sun experts say that some programmer attempts at optimization fail because they actually make it harder for JIT compiler to apply the optimizations it knows about.
On the other hand, higher level algorithmic optimizations are definitely worthwhile ... provided that your profiler tells you that your application is spending a significant amount of time in that area of the code.
Using arrays instead of collections can be a worthwhile optimization in unusual cases, and in rare cases using object pools might be too. But these optimizations 1) will make your code more complicated and bug prone and 2) can slow your application down if used inappropriately. These kinds of optimizations should only be tried as a last resort. For example, if your profiling says that such and such a HashMap<Integer,Integer> is a CPU bottleneck or a memory hog, then it is a better idea to look for an existing specialized Map or Map-like library class than to try and implement the map yourself using arrays. In other words, optimize at the high level.
If you spend long enough or your application is small enough, careful micro-optimization will probably give you a faster application (on a given JVM version / hardware platform) than just relying on the JIT compiler. If you are implementing a smallish application to do large-scale number crunching in Java, the pay-off of micro-optimization may well be considerable. But this is clearly not a typical case! For typical Java applications, the effort is large enough and the performance difference is small enough that micro-optimization is not worthwhile.
(Incidentally, I don't see how declaring a variable can make any possible difference to GC performance. The GC has to trace a variable every time it is encountered whether or not it is final. Besides, it is an open secret that final variables can actually change under certain circumstances, so it would be unsafe for the GC to assume that they don't. Unsafe as in "creates a dangling pointer resulting in a JVM crash".)
I see this a lot. The sequence generally goes:
Thinking performance is about compiler optimizations, big-O, and so on.
Designing software using the recommended ideas, lots of classes, two-way linked lists, trees with pointers up, down, left, and right, hash sets, dictionaries, properties that invoke other properties, event handlers that invoke other event handlers, XML writing, parsing, zipping and unzipping, etc. etc.
Since all those data structures were like O(1) and the compiler's optimizing its guts out, the app should be "efficient", right? Well, then, what's that little voice telling one that the startup is slow, the shutdown is slow, the loading and unloading could be faster, and why is the UI so sluggish?
Hand it off to the "performance expert". With luck, that person finds out, all this stuff is done in the recommended way, but that's why it's cranking its heart out. It's doing all that stuff because it's the recommended way to do things, not because it's needed.
With luck, one has the chance to re-engineer some of that stuff, to make it simple, and gradually remove the "bottlenecks". I say, "with luck" because often it's just not possible, so development relies on the next generation of faster processors to take away the pain.
This happens in every language, but moreso in Java, C#, C++, where abstraction has been carried to extremes. So by all means, be aware of best practices, but also understand what simple software is. Typically it consists of saving those best practices for the circumstances that really need them.
which code-optimization make sense
from a compiler perspective?
All the ones that a compiler can't reason about, because a compiler is very dumb and Java doesn't have "design by contract" (which, hence, cannot help the dumb compiler reason about your code).
For example if you're crunching data and using use int[] or long[] arrays, you may know something about your data that is IMPOSSIBLE for the compiler to figure out and you may use low-level bit-packing/compacting to improve the locality of reference in that part of your code.
Been there, done that, saw gigantic speedup. So much for the "super smart compiler".
This is just one example. There are a huge number of cases like this.
Remember that a compiler is really stupid: it cannot know that if ( Math.abs(42) > 0 ) will always return true.
This should give some food for thoughts to people that think that those compilers are "smart" (things would be different here if Java had DbC, but it doesn't).
what best practices there are for:
vmargs, heap and other stuff passed to
the JVM for initialization. How do I
get the right values here? Is there
any formula or is it try and error?
The real answer is: there shouldn't be. Sadly the situation is so pathetic that such low-level hackery is needed, due to serious failure on Java's part. Oh, one more "tiny" detail: playing with VM fine-tuning only works for server-side app. It doesn't work for desktop apps.
Anyone who has worked on Java desktop applications installed on hundreds or thousands of machines, on various OSes knows all too well what the issue is: full GC pauses making your app look like it's broken. The Apple VM on OS X 10.4 comes to mind for it's particularly afwul, but ALL the JVMs are subject to that issue.
What is worse: it is impossible to "fine tune" the GC's parameters across different OSes / VMs / memory configuration when your application is going to be run on hundreds/thousands of different configuration.
Anyone disputing that: please tell me how you "fine tune" your app knowing that it is going to be run both on octo-cores Mac loaded with 20 GB of ram (I've got users with such setups) and old OS X 10.4 PowerBook that have 768 MB of ram. Please?
But it is not bad: you should not, in the first place, have to be concerned with super-low-level detail like GC "fine tuning". The very fact that this is hinted to is a testimony to one area where Java has a major issue.
Java fans will keep on saying "the GC is super fast, object creation is cheap" while this is blatantly wrong. There's a reason with Trove' TIntIntHashMap runs around circles an HashMap<Integer,Integer>.
There's also a reason why at every new JVM release you'll get countless release notes explaining why -XXGCHyperSteroidMultiTopNotch offers better performance than the last "big JVM param" that every cool Java programmer had to know: maybe the JVM wasn't that great at GC'ing after all.
So to answer your question: how do you speed up Java programs? Easy, do like what the Trove guys did: stop needlessly creating gigantic amount of objects and stop needlessly auto(un)boxing primitives because they will kill your app's perfs.
A TIntIntHashMap OWNS the default HashMap<Integer,Integer> for a reason: for the same reason my apps are now much faster than before.
I stopped believing in crap like "object creation costs nothing" and "the GC is super-optimized, don't worry about it".
I'm using Java to crunch data (I know, I'm a bit crazy) and the one thing that made my app faster was to stop believing all the propaganda surrounding the "cheap object creation" and "amazingly fast GC".
The truth is: INSTEAD OF TRYING TO FINE-TUNE YOUR GC SETTINGS, STOP CREATING THAT MUCH GARBAGE IN THE FIRST PLACE. This can be stated this way: if changing the GC settings radically changes the way your app run, it may be time to wonder if all the needless junk objects your creating are really needed.
Oh, you know what, I'm betting we'll see more and more release notes explaining why Java version x.y.z's GC is faster than version x.y.z-1's GC ;)
Generally there are two kinds of performance optimizations you need to do with Java:
Algorithmic optimization. Choose an algorithm which behaves like you need to. For instance, a simple algorithm may perform best for small datasets, but the overhead of preparing a smarter algorithm may first pay off for much larger datasets.
Bottleneck identification. Here you need to be familiar with a profiler that can tell you what the problem is (humans always guess wrong) - memory leak?, slow method? etc... A good one to start with is VisualVM which can attach to a running program, and is available in the latest Sun JDK. When you know the problem, you can fix it.
Todays JVM's are surprisingly robust when it comes to performance. Any microoptimizations you can apply will, in practically all cases, have only very minor impact on performance. This is easy to understand if you take a look on how typical language constructs (e.g. FOR vs WHILE) translate to bytecode - they are almost indistinguishable.
Making methods/variables final has absolutely no impact on performance on a decent JIT'd JVM. The JIT will keep track of which methods are really polymorphic and optimize away the dynamic dispatch where possible. Static methods can still be faster, since they don't have a this-reference = one less local variable (which at the same time, limits their application). Most efficient micro optimizations are not so much Java specific, for example code with lots of conditional statements can become very slow due to branch mispredictions by the processor. Sometimes conditionals can be replaced by other, sequential code flow constructs (often at the cost of readability), reducing the number of mispredicted branches (and this applies to all languages that somehow compile to native code).
Note that profilers tend to inflate the time spent in short, frequently called methods. This is due to the fact that profilers need to instrument the code to keep track of invocations - this can interfere with the JIT's ability to inline those methods (and the instrumentation overhead becomes significantly larger than the time spent actually executing the methods body). Manual inlining, while apparently very performance boosting under a profiler has in most cases no effect under "real world" conditions. Don't rely purely on the profilers results, verify that optimizations you make have real impact under real runtime conditions, too.
Notable performance boosts can only be expected from changes that reduce the amount of work done, more cache friendly data layout or superior algorytms. Java partially limits your possibilities for cache friendly data layouts, since you have no control where the parts (arrays/objects) that form your data structure will be located in memory in relation to each other. Still, there are plenty of opportunities where choosing the right data structure for the job can make a huge difference (e.g. ArrayList vs LinkedList).
There is little you can do to aid the garbage collector. However, a point worth noting is, while object allocation in Java is very very fast, there is still the cost of object initialization (which is mostly under your control). Poor performance of applications that creating lots of (short lived) objects is more likely to be attributed to poor cache utilization than to the garbage collectors work.
Different applications types require different optimization strategies - so before asking about specific optimizations, find out where your application really spends its time.
If you are experiencing performance issues with your application, you should seriously consider trying some profiling (eg: hprof) to see whether the problem is algorithmic in nature, and also checking the GC performance logging (eg: -verbose:gc) to see if you could benefit from tuning your JVM GC options.
It is worth noting that the compiler does next to no optimisations, and the JVM doesn't optimise at the byte code level either. Most of the optimisations are performed by the JIT in the JVM and it optmises how the code is converted to native machine code.
The best way to optimise your code is to use a profiler which measures how much time and resources your application is using when you give it a realistic data set. Without this information you are just guessing and you can change alot of code where it really, really doesn't matter and find you have added bugs in the process.
Many come to the conclusion that its never worth optmising you code, even counter productive as it can waste time and introduce bugs and I would say that is true for 95+% of your code. However, with aprofiler you can measure the critical pieces of code and optmise the <5% worth optimising and done carefully, you won't get too many issues from trying to optimise your code.
It's hard to answer this too thoroughly because you haven't even mentioned what sort of project you're talking about. Is it a desktop application? A server-side application?
Desktop applications favor application startup time, so the HotSpot client VM is a good start. Client applications don't necessarily need all of their heap space all the time, so a good balance between starting heap and max heap is useful. (Like, maybe -Xms128m -Xmx512m)
Server applications favor overall throughput, which is something the HotSpot server VM is tuned for. You should always allocate the min and max heap sizes the same on a server application. There is an added cost at the system level to it having to malloc() and free() during garbage collection. Use something like -Xms1024m -Xmx1024m.
There are several different garbage collectors also, which are tuned to different application types.
Take a read through the Java SE 6 Performance White Paper if you want more info on the garbage collector and other performance related items from Java 6.
I'm asked to shorten the start-up period of a long starting app, however I have also to obligate to my managers to the amount of time i will reduce the startup - something like 10-20 seconds.
As I'm new in my company I said I can obligate with timeframe of months (its a big server and I'm new and I plan to do lazy load + performance tuning).
That answer was not accepted I was required to do some kind of a cache to hold important data in another server and then when my server starts up it would reach all its data from that cache - I find it a kind of a workaround and I don't really like it.
do you like it?
What do you think I should do?
PS when I profiled the app I saw many small issues that make the start-up long (like 2 minutes) it would not be a short process to fix all and to make lazy load.
Any kind of suggestions would help.
Language is Java.
Thanks
Rule one of performance optimisation: measure it. Get hard figures. At each stage of optimisation measure the performance gain/loss/lack of change. You (and your managers) are not in a position to say that a particular optimisation will or will not work before you try it and measure it. You can always ask to test & measure a solution before implementing it.
Rule two of performance optimisation (or anything really): choose your battles. Please bear in mind that your managers may be very experienced with the system in question, and may know the correct solution already; there may be other things (politics) involved as well, so don't put your position at risk by butting heads at this point.
I agree with MatthieuF. The most important thing to do is to measure it. Then you need to analyze the measurements to see which parts are most costly and also which resource (memory, CPU, network, etc) is the bottleneck.
If you know these answers you can propose solutions. You might be able to create small tests (proof of concepts) of your solution so you can report back early to your managers.
There can be all kind of solutions for example simply buying more hardware might be the best way to go. It's also possible that buying more hardware will have no results and you need to make modifications. The modifications can be optimizing the software, the database or other software. It might be to choose better algorithms, to introduce caching (in expense of more memory usage) or introduce multi threading to take advantage of multiple CPU cores. You can also make modifications to the "surroundings" of your application such as the configuration/version of your operating system, Java virtual machine, application server, database server and others. All of these components have settings which can affect the performance.
Again, it's very important to measure, identify the problem, think of a solution, build solution (maybe in proof of concept) and measure if solution is working. Don't fall in to the trap of first choosing a solution without knowing the problem.
It sounds to me as if you've come in at a relatively junior position, and your managers don't (yet) trust your abilities and judgment.
I don't understand why they would want you to commit to a particular speed-up without knowing if it was achievable.
Maybe they really understand the code and its problems, and know that a certain level of speed-up is achievable. In this case, they should have a good idea how to do it ... so try and get them to tell you. Even if their ideas are not great, you will get credit for at least giving them a try.
Maybe they are just trying to apply pressure (or pass on pressure applied to them) in order to get you to work harder. In this case, I'd probably give them a worth-while but conservative estimate. Then spend some time investigating the problems more thoroughly. And if after a few days research you find that your "off the cuff" estimates are significantly off the mark, go back to the managers with a more accurate estimate.
On the technical side, a two minute start-up times sounds rather excessive to me. What is the application doing in all that time? Loading data structures from files or a database? Recalculating things? Profiling may help answer some of these questions, but you also need to understand the system's architecture to make sens of the profile stats.
Without knowing what the real issues are here, I'd suggest trying to get the service to become available early while doing some of the less critical initialization in the background, or lazily. (And your managers' idea of caching some important data may turn out to be a good one, if viewed in this light.) Alternatively, I'd see if it was feasible to implement a "hot standby" for the system, or replicate it in such a way that allowed you to reduce startup times.