I consider running VisualVM against a production JVM to see what's going on there - it started to consume too much CPU for some reason.
It must not result in a JVM failure so I'm trying to estimate all the risks.
The only issue that I see on their site that could potentially bring JVM down is related to class sharing and -Xshare JVM option, but afaik class sharing is not enabled in server mode and/or on x64 systems.
So is it really safe to run VisualVM against a production JVM, if it's not - what are the risks that one should consider, and how much load (CPU/memory) does running VisualVM against a JVM (and profiling with it) put on it?
Thanks
AFAIK VisualVM can be used in production, but I would only use it on a server which is lightly loaded. What you could do is wait for the service to slow down and later when its not used as much test it to see if some of the collections are surprising large. Or you could trigger a heap dump and analyze it offline.
And you can't get stats on method calls without significant overhead. Java 6 and 7 are better than java 5 but it could still slow your application by 30% even wityh a commercial profiler.
Actually, you can get some information without a lot of overhead by using stack dumps. There is even a script to help you do this at https://gist.github.com/851961
This type of profiling is the least intrusive that you can get.
Related
i am running a couple of small servers that are written in java. There is one that listens on a standard socket, the rest communicate with each other using ActiveMQ. i noticed something strange where if i leave the system idle for a couple of hours, the RSS memory either grows by several to tens of megs or shrinks by several to tens of megs. I used jconsole to see what was going on in the servers, but the memory usage and object creation stayed relatively flat. I tested this with both Oracle Java and OpenJDK. I tried using the recommended solution of setting the MALLOC_ARENA_MAX=4, but that did not have an effect. Is there something else going on in the JVM that i am not aware of, and is there a way to stop it?
Setup:
CentOS 7.4.1708 (2G ram, 4 processors)
Oracle Java 8u162
OpenJDK 1.8.0.212
glibc 2.17
Is there something else going on in the JVM
Yes. Garbage collection, JIT compilation, class loading / unloading, logging, I/O etc. More details here.
RSS of a Java process can easily go up and down by hundreds of megabytes - "several megs" is not typically an issue at all. To find where the native memory is allocated from, turn on Native Memory Tracking feature and/or use async-profiler as described in this answer.
I have been developing a small Java utility that uses two frameworks: Encog and Jetty to provide neural network functionality for a website.
The code is 'finished' in that it does everything it needs to do, but I have some problems with memory usage. When running on my development machine the memory usage seems to fluctuate between about 4MB and 13MB when the application is doing things (training neural networks) and at most it uses about 18MB. This is very good usage and I think it is due to the fact that I call System.GC() fairly regularly. I do this because the processing time doesn't matter for me, but the memory usage does.
So it all works fine on my machine, but as soon as I put it online on our server (shared unix hosting with memory limits) it uses about 19MB to start with and rises to hundreds of MB of memory usage when doing things. These are the same things that I have been doing in testing. The only way, I believe, to reduce the memory usage, is to quit the application and restart it.
The only difference that I can tell is the Java Virtual Machine that it is being run on. I do not know about this and I have tried to find the reason why it is acting this way, but a lot of the documentation assumes a great knowledge of Java and Virtual Machines. Could someone please help m with some reasons why this may be happening and perhaps some things to try to stop it.
I have looked at using GCJ to compile the application, but I don't know if this is something I should be putting a lot of time in to and whether it will actually help.
Thanks for the help!
UPDATE: Developing on Mac OS 10.6.3 and server is on a unix OS but I don't know what. (Server is from WebFaction)
I think it is due to the fact that I
call System.GC() fairly regularly
You should not do that, it's almost never useful.
A garbage collector works most efficiently when it has lots of memory to play with, so it will tend to use a large part of what it can get. I think all you need to do is to set the max heap size to something like 32MB with an -Xmx32m command line parameter - the default depends on whether the JVM believes it's running on a "server class" system, in which case it assumes that you want the application to use as much memory as it can in order to give better throughput.
BTW, if you're running on a 64 bit JVM on the server, it will legitimately need more memory (usually about 30%) than on a 32bit JVM due to larger references.
Two points you might consider:
Calls of System.gc can be disabled by a commandline parameter (-XX:-DisableExplicitGC), I think the behaviour also depends on the gc algorithm the vm uses. Normally invoking the gc should be left to the jvm
As long as there is enough memory available for the jvm I don't see anything wrong in using this memory to increase application and gc performance. As Michael Borgwardt said you can restrict the amount of memory the vm uses at the command line.
Also you may want to look at what mode the JVM has been started when you deploy it online. My guess its a server VM.
Take a look at the differences between the two right here on stackoverflow. Also, see what garbage collector is actually running on the actual deployment. See if you can tweek the GC behaviour, or change the GC algorithm.See the -X options if its a Sun JVM.
Basically the JVM takes the amount of memory it is allowed to as needed, in order to make the "new" operation as fast as possible (this is a science in itself).
So if you have a lot of objects being used, and then discarded, you will slowly and surely fill up the available memory. Then you can ask for garbage collection, but it is just a hint, and the JVM may choose not to listen.
So, you need another mechanism to keep memory usage down. The typical approach is to limit the amount of memory with -Xoptions, but be careful since the JVM you use on your pc may be very different from the one you deploy on, and the memory need may therefore be different.
Is there a deliberate requirement for low memory usage? If not, then just let it run and see how the JVM behaves. Use jvisualvm to attach and monitor.
Perhaps the server uses more memory because there is a higher load on your app and so more threads are in use? Jetty will use a number of threads to spread out the load if there are a lot of requests. Its worth a look at the thread count on the server versus on your test machine.
I am trying to figure out why Jetty 6.1.22 is running out of memory on my laptop. I have 2 web applications running JBoss Seam, Hibernate (with EHCache), and separate Quartz scheduler instances.
With little load, the server dies throwing OutOfMemory.
What can I look for? Would you think that I am not properly closing handles for input streams or files?
I tried profiling my application with Netbeans, but it works off and on. Usually, it ends up locking up even though it doesn't use that much CPU or memory.
Walter
What are you JVM's execution parameters?
Try increasing available heap memory though -Xms (default) -Xmx (max) and -Xmn (min) JVM's flags.
You can also monitor your Application Server execution with JConsole. It's usually helpful for finding out where is you application leaking.
add -XX:+HeapDumpOnOutOfMemoryError when invoking the jvm and when you get the OOM situation you will get a .hprof file dumped. You can open it later with several tools and you'll be able to see where the memory is going...
The tool I use is Eclipse Memory Analyzer, it's pretty good.
I can strongly recommend attaching to the troublesome program with jvisualvm in the JDK.
This allows you to investigate memory and cpu usage over time and inspect what happens in general.
I have a very large Java app. It runs on Tomcat and is your typical Spring/Hibernate webapp. It is also an extremely large Java program. It's easy for me to test the performance of database queries, since I can run those separately, but I have no idea to look for Java bottlenecks on a stack like this. I tried Eclipse's TPTP profiler, but it really didn't seem to like my program, and I suspect that it is because my program is too large. Does anyone have any advice on profiling a large webapp?
The Visual VM profiler that now comes with the JDK can be attached to running processes and may at least give an initial overview of the performance. It is based on the Netbeans profiler.
Try jProfiler. It's easy to integrate with Tomcat
If you can get Tomcat and your application running in Netbeans.
Then you can use the Netbeans built-in profiler to test performance, memory usage, etc ...
Wikipage on tomcat in Netbeans.
I have used YourKit to profile applications with an 8 GB heap and it worked quite well.
Check JAMon. It's not a profiler, but it's the best tool for profiling that I can recommend. It's very easy to integrate with spring. We use it in test and live environment.
I've never found an easy way to do this because there's typically so much going on that it's hard to get a clear overall picture. With things like Hibernate even more so because the correct behavior may be to grab a big chunk of memory for cached data, even though your app's not really "doing anything", so another memory inefficient process that you run may get swamped in profiling.
Are you profiling for memory, speed, or just in general looking for poor performance? Try to test processes that you suspect are bad in isolation, it's certainly much easier.
JProbe, JProfiler, both good, free demos are available. Testing inside an IDE complicates the memory issues, I've found it easier not to bother.
Try JProfiler. It has a trial license and it is very full featured. To use it, you'll have to:
Add the JProfiler agent as an argument to your java command
Start the program on the server
Start JProfiler and choose the "Connect to an application running remotely"
Give it the port number and whatever host it's running on
All these are in the instructions that come with JProfiler, but the important part is that you'll connect through a host and port to your running app.
As for what to profile, I'm sure you have an idea of things that could be memory/CPU intensive - loading large data sets, sorting, even simple network I/O if it's done incorrectly. Do these things (it's great if you can automate load testing using some scripts that bang on your server) and collect a snapshot with JProfiler.
Then view the graphs at your leisure. Turn on CPU monitoring and watch where the CPU cycles are being spent. You'll be able to narrow down by percentage in each method call, so if you're using more than 1 or 2% of CPU in methods that you have source for, go investigate and see if you can make them less CPU intensive.
Same goes for memory. Disable all the CPU profiling, enable all the memory profiling, run the tests again and get your snapshot.
Rinse, repeat.
You might also take this time to read up on memory management and garbage collection. There's no better time to tune your garbage collection than when you're already profiling: http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html
Pay special attention to the part on the eden/survivor object promotion. In web apps you get a lot of short-lived objects, so it often makes sense to increase the young generations at the expense of the tenured generations.
I've just upgraded some old Java source which has been running on a Sun Java 1.4.2 VM to Sun Java (JRE) 6 VM. More or less the only thing I had to change was to add explicit datatypes for some abstract objects (Hashmap's, Vector's and so on). The code itself it quite memory intensive, using up to 1G of heap memory (using -Xmx1024m as a parameter to start the VM).
Since I read alot about better performance on newer Java VM's, this was one of the reasons I did this upgrade.
Can anyone think of a reason why the performance is worse in my case now (just in general, of course, since you can't take a look at the code)?
Has anyone some advice for a non Java guru what to look for if I wanted to optimize (speed wise) existing code? Any hints, recommended docs, tools?
Thanks.
Not much information here. But here are a couple of things you might want to explore:
Start the VM with Xmx and Xms as the same value (in your case 1024M)
Ensure that the server jvm dll is being used to start the virtual machine.
Run a profiler to see what objects are hogging memory or what objects are not being garbage collected
Hook up your VM with the jconsole and trace through the objects
If your application nearly runs out of free space, garbage collection time may dominate computation time.
Enable gc debugging to look for this. Or, even better, simply start jconsole and attach it to your program.
Theoretically it could be that you application consumes more memory, because there were changes to the way Strings share their internal char[]. Less sharing is done after 1.4.
Check my old blog at http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/5100 (new blog is here)
I would compare the Garbage Collector logs to see whether memory usage is really the problem.
If that doesn't help, us a profiler such as Yourkit to find the differences.
Definitely use a profiler on the app (YourKit is great)...it's easy to waste a lot of time guessing at the problem when most of the time you'll be able to narrow it down really quickly in the profiler.