I've been tasked with debugging a Java (J2SE) application which after some period of activity begins to throw OutOfMemory exceptions. I am new to Java, but have programming experience. I'm interested in getting your opinions on what a good approach to diagnosing a problem like this might be?
This far I've employed JConsole to get a picture of what's going on. I have a hunch that there are object which are not being released properly and therefor not being cleaned up during garbage collection.
Are there any tools I might use to get a picture of the object ecosystem? Where would you start?
I'd start with a proper Java profiler. JConsole is free, but it's nowhere near as full featured as the ones that cost money. I used JProfiler, and it was well worth the money. See https://stackoverflow.com/questions/14762/please-recommend-a-java-profiler for more options and opinions.
Try the Eclipse Memory Analyzer, or any other tool that can process a java heap dump, and then run your app with the flap that generates a heap dump when you run out of memory.
Then analyze the heap dump and look for suspiciously high object counts.
See this article for more information on the heap dump.
EDIT: Also, please note that your app may just legitimately require more memory than you initially thought. You might try increasing the java minimum and maximum memory allocation to something significantly larger first and see if your application runs indefinitely or simply gets slightly further.
The latest version of the Sun JDK includes VisualVM which is essentially the Netbeans profiler by itself. It works really well.
http://www.yourkit.com/download/index.jsp is the only tool you'll need.
You can take snapshots at (1) app start time, and (2) after running app for N amount of time, then comparing the snapshots to see where memory gets allocated. It will also take a snapshot on OutOfMemoryError so you can compare this snapshot with (1).
For instance, the latest project I had to troubleshoot threw OutOfMemoryError exceptions, and after firing up YourKit I realised that most memory were in fact being allocated to some ehcache "LFU " class, the point being that we specified loads of a certain POJO to be cached in memory, but us not specifying enough -Xms and -Xmx (starting- and max- JVM memory allocation).
I've also used Linux's vmstat e.g. some Linux platforms just don't have enough swap enabled, or don't allocate contiguous blocks of memory, and then there's jstat (bundled with JDK).
UPDATE see https://stackoverflow.com/questions/14762/please-recommend-a-java-profiler
You can also add an "UnhandledExceptionHandler" to your Application's Thread. This will catch 'uncaught' exception, like an out of memory error, and you will at least have an idea where the exception was thrown. Usually this not were the problem is but the 'new' that couldn't be satisfied. As a rule I always add the UnhandledExceptionHandler to a Thread if nothing else to add logging.
Related
I'm experiencing a very odd problem with a Java application running under Tomcat.
We tried to update the production code from a fresh newly produced in a 1-week sprint, the application has been running over months without hiccups and then this new code makes our Linux servers start swapping after some time.
The very strange thing is that when looking at VisualVM for memory usage it never exceeds the maximum heap size, the JVM does not throw an OutOfMemory, the machine only starts swapping and the JVM keeps running even after that.
So, it seems that's leaking memory from somewhere, it seems like it's from the new code but it's odd that it's not inside the JVM, any ideas in how to debug that?
Thanks!
Swapping is not a conclusive indicator of leakage. It results from low physical memory. Use vmstat on Linux to get swap usage. Try using a different machine, experiment with configurations --swap size, physical memory size, address space.
If you are confident that the problem is in your program try this:
Estimate the median and peak memory that your program should use. You must be able to account for all deviations from these metrics. If you cannot, proceed to step 3.
Assuming you did step 1 correctly and were able to account for all deviations, you can rule out the leak (sorry about such vague suggestions but debugging is only as good as the detective). You should now focus on GC tuning. First, enable GC logging. See if your heap is actually full and where the GC is spending most of its time collecting. This may be a good starting point to start optimizations. Try to see if adjusting GC options helps. Try experimenting with collection algorithms, max/min heap sizes, gen ratios etc. Only experiment when you have ruled out a leak (step 1).
Assuming you did step 1 correctly and were not able to account for all deviations, you can assume that you have a leak somwhere. Use a memory profiler to see what objects contribute to the heap size growth most. Leave a profiler running for an extended period of time --have your program handle some requests it routinely expects to get and then leave it relatively isolated after that. If the memory level keeps on growing you may have a leak somewhere. If not, then it is probably not a memory leak. Can you pin point the part of your program that may be creating them? If yes, try sending several requests that only target that part of your program. Does it replicate the problem deterministically? If no, repeat step 3. If yes, use divide and conquer and reapply step 3 till you can find the class/method that are the culprits. It can be a certain combination of multiple portions as well (meaning that individually they may look innocent but together they may form a brilliant crime syndicate).
Hope this helps, if not then please leave a comment to my post.
All the very best on your exercise!
I would suggest you look into creating heap dumps without using jvisualvm. For Unix-based Oracle JVM's this is normally done by sending a signal 3 to the JVM using kill.
For full details see http://www.startux.de/index.php/java/45-java-heap-dumpyvComment45
You can then see if the patterns changes.
If you do not get an idea from this, then this might be because you are storing a sub-string from a very large original string (which carries the underlying string array around), or because you hold on to operating system resources like open database connections etc.
You have checked your connection pool looks good?
If you aren't using it, I'd recommend using visual VM version 1.3.2 and all the plug-ins. It's a big jump up from earlier versions.
What happens to the perm gen space?
What are the memory settings you're using? Min and max, of course, but what about perm space size?
I am not a Java dev, but an app landed on my desk. It's a web-service server-side app that runs in a Tomcat container. The users hit it up from a client application.
The users constantly complain about how slow it is and the app has to be restarted about twice a week, cause things get really bad.
The previous developer told me that the app simply runs out of memory (as it loads more data over time) and eventually spends all its time doing garbage collection. Meanwhile, the Heap Size for Tomcat is set at 6GB. The box itself has 32GB of RAM.
Is there any harm in increasing the Heap Size to 16GB?
Seems like an easy way to fix the issue, but I am no Java expert.
You should identify the leak and fix it, not add more heap space. Thats just a stop gap.
You should configure tomcat to dump the heap on error, then analyze the heap in one of any number of tools after a crash. You can compute the retained sizes of all the clases, which should give you a very clear picture of what is wrong.
Im my profile I have a link to a blog post about this, since I had to do it recently.
No, there is no harm in increasing the Heap Size to 16GB.
The previous developer told me that the app simply runs out of memory (as it loads more data over time)
This looks like a memory leak, a serious bug in application. If you increase the amount of memory available from 6 to 16 GiB, you're still gonna have to restart the application, only less frequent. Some experienced developer should take a look at the application heap while running (look at hvgotcodes tips) and fix the application.
To resolve these issues you need to do performance testing. This includes both CPU and memory analysis. The JDK (6) bundles a tool called VisualVM, on my Mac OS X machine this is on the path by default as "jvisualvm". That's free and bundled, so it's a place to start.
Next up is the NetBeans Profiler (netbeans.org). That does more memory and CPU analysis. It's free as well, but a bit more complicated.
If you can spend the money, I highly recommend YourKit (http://www.yourkit.com/). It's not terribly expensive but it has a lot of built-in diagnostics that make it easier to figure out what's going on.
The one thing you can't do is assume that just adding more memory will fix the problem. If it's a leak, adding more memory may just make it run really badly a bit longer between restarts.
I suggest you use a profiling tool like JProfiler, VisualVM, jConsole, YourKit etc. You can take a heap dump of your application and analyze which objects are eating up memory.
My Linux server need to be able to handle 30+ eclipse instances for developers. I did a quick test of running 10 eclipse instances. The Java process associated with each eclipse initially around 200MB RSS memory, increased up to around 550MB, when more projects are loaded.
But Java process doesn't seem to release memory, after closing/deleting all projects within eclipse instances. I still see it uses over 550MB RSS.
How can I change Eclipse or Java settings so that memory foot print got reduced when developers closed down projects or being idle for a while?
Thanks
You may want to experiment with these (and other) JVM tuning options to make the JVM less reluctant to return memory to the OS:
-XX:MaxHeapFreeRatio Maximum percentage of heap free after GC to avoid shrinking. Default is 70.
-XX:MinHeapFreeRatio Minimum percentage of heap free after GC to avoid expansion. Default is 40.
However, I suspect that you won't see the eclipse process shrink to anywhere near its initial size, since eclipse is a huge, complex application that probably lazy-loads (but does not unload, once used) a lot of classes and associated data structures.
I've never seen Java release memory.
I don't think you will get any value out of trying to get it to release memory with Eclipse, I've watched that little memory counter for YEARS and never once see the allocated memory drop.
You might try one of these.
After each session, exit the JVM and restart.
Set your -Xmx lower.
Separate your instances into categories with high -Xmx and low -Xmx and let the user determine which one he wants.
As a side-thought, if it really mattered to you, you MIGHT be able to run multiple eclipse instances under one VM. It would probably be WAY too much work (man-weeks to man-years), but if you could get it right you could reduce overhead by like 150-200mb/instance. The disadvantage would be that a VM crash (Pretty rare these days) would kill everyone.
Testing this theory would be a matter of calling eclipse's main from within an existing JVM and trying to get it to display somewhere useful. The rest of the man-year is spent trying to figure out where they used evil static variables or singletons and changing them to something else.
Switch the Java to use the G1 garbage collector with the HeapFreeRatio parameters. Use these options in eclipse.ini:
-XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
-XX:MinHeapFreeRatio=5
-XX:MaxHeapFreeRatio=25
Now when Eclipse eats up more than 1 GB of RAM for a complicated operation and switched back to 300 MB after Garbage Collection the memory will be released back to the operating system.
I would suggest checking on garbage collection, setting right options or even forcing GC periodically might increase time till eclipse memory usage grows high.
Following link might be useful http://www.eclipsezone.com/eclipse/forums/t93757.html
There is a memory leak happens in an application when a short lived object holds a long lived object,
My question is how can we identify
1) which object lives longer and shorter, any tool which measures life of an object?
2nd Question
I am constantly getting the Out of Memory Space Error and I tried increasing the Heap memory to 2 GB, but still i am getting, please suggest me any open source tool with which i can identify the memory leak issue and fix.
At present I am restarting the server every time as a temporary solution, but Suggest me any thing which i can fix permanently.
You can use the VisualVM tool included in the JDK:
http://download.oracle.com/javase/6/docs/technotes/tools/share/jvisualvm.html
Documentation available here:
https://visualvm.dev.java.net/docindex.html
There are 2 options:
It just may be your application doesn't have enough heap allocated. Measure size of your input and give application corresponding heap;
There's memory-leak: take profiler, examine your heap, find objects which shouldn't be there or there too much of them ('short-living objects', in your terms), identify which 'long-living' object holds them, fix this. You should know your code to understand which objects must be 'short-living' and which must be 'long-living'.
I've found the Heap Walker in Netbeans very usefull
As said, jvisualvm have good tools to analyze the heap live.
But you can also use jvisualvm or -XX:+HeapDumpOnOutOfMemoryError to take a heap dump in a file. And then take the file to your destkop, to open it in Eclipse Memory Analyzer. Eclipse MAT is even better to analyze the memory.
Out of Memory occurs on a server because it literally uses up all memory it's allowed to have. Not sure about what application you're using for hosting the server, but for Apache, you need to add the line -Xmx512m where 512 is the maximum amount of megabytes it's allowed to have.
If you leave the application to run long enough, it's going to happen. This isn't because of memory leaks in Java but the server itself which has a tendency to do so. You can't change this behavior, but you can at least increase the default memory of 256 mb. With the heavy loading site that I work on everyday, 256 mb lasts about 30 minutes for me unfortunately. I've found that 1024 mb is reasonable and rarely crashes due to out of memory exceptions.
I'd strike me as very unusual for Java to be incapable of garbage collecting correctly unless the programmer took a hand at overriding typical functionality.
I think you can track memory leaks with jsconsole (which comes shipped with JDK6 if i'm not mistaken).
A short-lived object holding a reference to a long-lived object will not cause problems. (a good overview , including generational garbage collection).
2GB is an awful lot of objects/references. If you're running out of heap space at 2Gb you're likely holding onto massive amounts of data and/or keeping open resources when you're done with them. You should post at the very least a description of what your application does and how long it takes to die.
You can get some sense of what's happening quickly by watching the garbage collector (e.g. run with "-verbose:gc" which will tell you when the garbage collector is running and how much it collects).
I have a standalone Java problem running in a linux server. I started the jvm with -Xmx256m. I attached a JMX monitor and can see that the heap never really passes 256Mb. However, on my linux system when I run the top command I can see that:
1) First of all, the RES memory usage of this process is around 350Mb. Why? I suppose this is because of memory outside of the heap?
2) Secondly, the VIRT memory usage of this process just keeps growing and growing. It never stops! It now shows at 2500Mb! So do I have a leak? But heap doesn't increase, it just cycles!
Ultimately this poses a problem because the swap of the system keeps growing and eventually the system dies.
Any ideas what is going on?
The important question I want to ask, what are some scenarios that this could be a result of my code and not the JVM, kernal, etc. For example, if the number of threads keeps growing, would that fit the description of my observations? Anything similar that you can suggest me to look out for?
A couple of potential problems:
Direct allocated buffers and memory mapped files are allocated outside of the Java heap, and can't conveniently be disposed.
An area of stack is reserved for each new thread.
Permanent generation (code and interned strings) is outside of the usual stack. It can be a problem is class loaders leak (usually when reloading webapps).
It's possible that the C heap is leaking.
pmap -x should show how your memory has disappeared.
Swap Sun vs IBM JVM to test
RES will include code + non-head data. Also, some things that you think would be stored in the heap aren't, such as the thread stack and "class data". (It's a matter of definition but code and class data are controlled by -XX:MaxPermSize=.)
This one sounds like a memory leak in either the JVM implementation, the linux kernel, or in library JNI code.
If using the Sun JVM, try IBM, or vice versa.
I'm not sure exactly how dlopen works, but code accessing system libraries might be remapping the same thing repeatedly, if that's possible.
Finally, you should use ulimit to make the system fail earlier, so you can repeat tests easily.
WRT #1, it's normal for your RSS to be larger than your heap. This is because system libraries and non-Java code are included in the RSS but not the heap size.
WRT #2, Yes, it sounds like you have a leak of some sort. If the system itself is crashing, you are likely consuming too much of a system resources, like sockets, threads, or files.
Try using lsof to see what files the JVM has open. Run this a few times as your memory increases. If the JVM is crashing, be sure to set the -XX:+HeapDumpOnOutOfMemoryError option.
In my experience, the most common cause of non-heap memory leak in Java is thread leak.
A tool you may find useful is jvmtop, which lets you monitor heap size, thread number and other metrics in real time.
Sounds like you have a leak. Can't you do profiling to see which function is driving the memory up? I am not sure though.
If I had to take a stab in the dark, I would say that the JVM you are using has a memory leak.