I have a very large Java app. It runs on Tomcat and is your typical Spring/Hibernate webapp. It is also an extremely large Java program. It's easy for me to test the performance of database queries, since I can run those separately, but I have no idea to look for Java bottlenecks on a stack like this. I tried Eclipse's TPTP profiler, but it really didn't seem to like my program, and I suspect that it is because my program is too large. Does anyone have any advice on profiling a large webapp?
The Visual VM profiler that now comes with the JDK can be attached to running processes and may at least give an initial overview of the performance. It is based on the Netbeans profiler.
Try jProfiler. It's easy to integrate with Tomcat
If you can get Tomcat and your application running in Netbeans.
Then you can use the Netbeans built-in profiler to test performance, memory usage, etc ...
Wikipage on tomcat in Netbeans.
I have used YourKit to profile applications with an 8 GB heap and it worked quite well.
Check JAMon. It's not a profiler, but it's the best tool for profiling that I can recommend. It's very easy to integrate with spring. We use it in test and live environment.
I've never found an easy way to do this because there's typically so much going on that it's hard to get a clear overall picture. With things like Hibernate even more so because the correct behavior may be to grab a big chunk of memory for cached data, even though your app's not really "doing anything", so another memory inefficient process that you run may get swamped in profiling.
Are you profiling for memory, speed, or just in general looking for poor performance? Try to test processes that you suspect are bad in isolation, it's certainly much easier.
JProbe, JProfiler, both good, free demos are available. Testing inside an IDE complicates the memory issues, I've found it easier not to bother.
Try JProfiler. It has a trial license and it is very full featured. To use it, you'll have to:
Add the JProfiler agent as an argument to your java command
Start the program on the server
Start JProfiler and choose the "Connect to an application running remotely"
Give it the port number and whatever host it's running on
All these are in the instructions that come with JProfiler, but the important part is that you'll connect through a host and port to your running app.
As for what to profile, I'm sure you have an idea of things that could be memory/CPU intensive - loading large data sets, sorting, even simple network I/O if it's done incorrectly. Do these things (it's great if you can automate load testing using some scripts that bang on your server) and collect a snapshot with JProfiler.
Then view the graphs at your leisure. Turn on CPU monitoring and watch where the CPU cycles are being spent. You'll be able to narrow down by percentage in each method call, so if you're using more than 1 or 2% of CPU in methods that you have source for, go investigate and see if you can make them less CPU intensive.
Same goes for memory. Disable all the CPU profiling, enable all the memory profiling, run the tests again and get your snapshot.
Rinse, repeat.
You might also take this time to read up on memory management and garbage collection. There's no better time to tune your garbage collection than when you're already profiling: http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html
Pay special attention to the part on the eden/survivor object promotion. In web apps you get a lot of short-lived objects, so it often makes sense to increase the young generations at the expense of the tenured generations.
Related
I have got some basic working knowledge of YourKit Java profiler. I want to perform memory & CPU profiling of my Selenium WebDriver+TestNG framework. It contains a large number of tests in the form of PageObject classes and Test classes. I have checked out for any online resource that can show some direction on how to do this but could not find any.
Has anyone done memory+CPU profiling on webdriver+TestNG tests? Is it possible first of all to do memory profiling of such java applications? Need some directions.
Profiling tests is a bit tricky affair due to number of factors involved like effects of parallel test execution or sequence of test execution . You can get an overall picture for memory or cpu by having the visualvm (or even jconsole ) monitoring on your tests . For getting statistics for individual tests i believe you need to have a professional profiler (i am not aware about any open source tools to do that ) . I don't prefer intrusive profilers for performance tests , hence you can also get an overall picture of the CPU samples during the tests using Hprof as well .
Having said that nowadays most ides have pretty good profilers (or at least profiler plug-in) and if you are required to give a rough estimate of the cpu and memory numbers you can use them as well .
I consider running VisualVM against a production JVM to see what's going on there - it started to consume too much CPU for some reason.
It must not result in a JVM failure so I'm trying to estimate all the risks.
The only issue that I see on their site that could potentially bring JVM down is related to class sharing and -Xshare JVM option, but afaik class sharing is not enabled in server mode and/or on x64 systems.
So is it really safe to run VisualVM against a production JVM, if it's not - what are the risks that one should consider, and how much load (CPU/memory) does running VisualVM against a JVM (and profiling with it) put on it?
Thanks
AFAIK VisualVM can be used in production, but I would only use it on a server which is lightly loaded. What you could do is wait for the service to slow down and later when its not used as much test it to see if some of the collections are surprising large. Or you could trigger a heap dump and analyze it offline.
And you can't get stats on method calls without significant overhead. Java 6 and 7 are better than java 5 but it could still slow your application by 30% even wityh a commercial profiler.
Actually, you can get some information without a lot of overhead by using stack dumps. There is even a script to help you do this at https://gist.github.com/851961
This type of profiling is the least intrusive that you can get.
I have an application which runs a rather long analysis (lots of number crunching) so running the application once takes about 3-4 hours, fully utilizing all of the cores. Now I am pretty sure my code is not water-tight so I want to profile and look for potential weak points.
I have been reading quite a bit on jvisualvm, and played around with it a bit too. However it appears as one chooses either cpu or memory profiling, while this article from Javalobby has an interesting quote where the author says:
I realise that both the CPU and Memory Profiling could have been done simultaneously, but for the purpose of this article I wanted to keep them seperate.
Could anyone deny or confirm this? If this is possible it would very useful, so I don't start over and over to profile in different modes. If it's not possible, would it be possible to queue to different profiling analyses so I can run them overnight?
Thanks,
It is not possible to do CPU and Memory profiling together, but you can switch between CPU and memory very easily especially when using 'Sampler' tab. For your case, I would start with just simple monitoring. Looking at the graphs, you should be able to tell, if you have memory problem or not. If you have memory problem, I would try to fix it first and that turn your attention to the CPU profiling.
I find that profilers tend to underestimate the cost of object allocation, so I usually enable memory profiling with cpu profiling, as I feel this gives a more realistic CPU profiling result. (Even if I don't look at the memory profiling report)
If in doubt I suggest you run the CPU profile, with and without memory profiling and you can get very different results. In my experience it is worth optimising for both results. ;)
BTW: I use YourKit, but I don't imagine VisualVM to be very different in this regard.
I am not a Java dev, but an app landed on my desk. It's a web-service server-side app that runs in a Tomcat container. The users hit it up from a client application.
The users constantly complain about how slow it is and the app has to be restarted about twice a week, cause things get really bad.
The previous developer told me that the app simply runs out of memory (as it loads more data over time) and eventually spends all its time doing garbage collection. Meanwhile, the Heap Size for Tomcat is set at 6GB. The box itself has 32GB of RAM.
Is there any harm in increasing the Heap Size to 16GB?
Seems like an easy way to fix the issue, but I am no Java expert.
You should identify the leak and fix it, not add more heap space. Thats just a stop gap.
You should configure tomcat to dump the heap on error, then analyze the heap in one of any number of tools after a crash. You can compute the retained sizes of all the clases, which should give you a very clear picture of what is wrong.
Im my profile I have a link to a blog post about this, since I had to do it recently.
No, there is no harm in increasing the Heap Size to 16GB.
The previous developer told me that the app simply runs out of memory (as it loads more data over time)
This looks like a memory leak, a serious bug in application. If you increase the amount of memory available from 6 to 16 GiB, you're still gonna have to restart the application, only less frequent. Some experienced developer should take a look at the application heap while running (look at hvgotcodes tips) and fix the application.
To resolve these issues you need to do performance testing. This includes both CPU and memory analysis. The JDK (6) bundles a tool called VisualVM, on my Mac OS X machine this is on the path by default as "jvisualvm". That's free and bundled, so it's a place to start.
Next up is the NetBeans Profiler (netbeans.org). That does more memory and CPU analysis. It's free as well, but a bit more complicated.
If you can spend the money, I highly recommend YourKit (http://www.yourkit.com/). It's not terribly expensive but it has a lot of built-in diagnostics that make it easier to figure out what's going on.
The one thing you can't do is assume that just adding more memory will fix the problem. If it's a leak, adding more memory may just make it run really badly a bit longer between restarts.
I suggest you use a profiling tool like JProfiler, VisualVM, jConsole, YourKit etc. You can take a heap dump of your application and analyze which objects are eating up memory.
I have been developing a small Java utility that uses two frameworks: Encog and Jetty to provide neural network functionality for a website.
The code is 'finished' in that it does everything it needs to do, but I have some problems with memory usage. When running on my development machine the memory usage seems to fluctuate between about 4MB and 13MB when the application is doing things (training neural networks) and at most it uses about 18MB. This is very good usage and I think it is due to the fact that I call System.GC() fairly regularly. I do this because the processing time doesn't matter for me, but the memory usage does.
So it all works fine on my machine, but as soon as I put it online on our server (shared unix hosting with memory limits) it uses about 19MB to start with and rises to hundreds of MB of memory usage when doing things. These are the same things that I have been doing in testing. The only way, I believe, to reduce the memory usage, is to quit the application and restart it.
The only difference that I can tell is the Java Virtual Machine that it is being run on. I do not know about this and I have tried to find the reason why it is acting this way, but a lot of the documentation assumes a great knowledge of Java and Virtual Machines. Could someone please help m with some reasons why this may be happening and perhaps some things to try to stop it.
I have looked at using GCJ to compile the application, but I don't know if this is something I should be putting a lot of time in to and whether it will actually help.
Thanks for the help!
UPDATE: Developing on Mac OS 10.6.3 and server is on a unix OS but I don't know what. (Server is from WebFaction)
I think it is due to the fact that I
call System.GC() fairly regularly
You should not do that, it's almost never useful.
A garbage collector works most efficiently when it has lots of memory to play with, so it will tend to use a large part of what it can get. I think all you need to do is to set the max heap size to something like 32MB with an -Xmx32m command line parameter - the default depends on whether the JVM believes it's running on a "server class" system, in which case it assumes that you want the application to use as much memory as it can in order to give better throughput.
BTW, if you're running on a 64 bit JVM on the server, it will legitimately need more memory (usually about 30%) than on a 32bit JVM due to larger references.
Two points you might consider:
Calls of System.gc can be disabled by a commandline parameter (-XX:-DisableExplicitGC), I think the behaviour also depends on the gc algorithm the vm uses. Normally invoking the gc should be left to the jvm
As long as there is enough memory available for the jvm I don't see anything wrong in using this memory to increase application and gc performance. As Michael Borgwardt said you can restrict the amount of memory the vm uses at the command line.
Also you may want to look at what mode the JVM has been started when you deploy it online. My guess its a server VM.
Take a look at the differences between the two right here on stackoverflow. Also, see what garbage collector is actually running on the actual deployment. See if you can tweek the GC behaviour, or change the GC algorithm.See the -X options if its a Sun JVM.
Basically the JVM takes the amount of memory it is allowed to as needed, in order to make the "new" operation as fast as possible (this is a science in itself).
So if you have a lot of objects being used, and then discarded, you will slowly and surely fill up the available memory. Then you can ask for garbage collection, but it is just a hint, and the JVM may choose not to listen.
So, you need another mechanism to keep memory usage down. The typical approach is to limit the amount of memory with -Xoptions, but be careful since the JVM you use on your pc may be very different from the one you deploy on, and the memory need may therefore be different.
Is there a deliberate requirement for low memory usage? If not, then just let it run and see how the JVM behaves. Use jvisualvm to attach and monitor.
Perhaps the server uses more memory because there is a higher load on your app and so more threads are in use? Jetty will use a number of threads to spread out the load if there are a lot of requests. Its worth a look at the thread count on the server versus on your test machine.