I have encountered the infamous OutOfMemoryException in my application and instead of simply increasing the amount of Heap Space available I tried to look into what the problem was, just in case, there was some sort of leak from my application.
I added the JVM parameter -XX:+HeapDumpOnOutOfMemoryError which creates a Heap Dump when the OutOfMemory Error is encountered. I then analysed the dump file produced with different profiling tools. I then started playing around with the -Xmx parameter and observing patterns.
What puzzled me is the following. Why is it that on analyzing the dump I found that the total size of all objects was much less than the total that I set using the -Xmx parameter?
For example, say I set the -Xmx to '2048m'. When I analyzed the dump file I found a total of 400Mb of objects on the Heap. I was expecting to find 2GB. Am I missing something?
My guess is that since modern GCs partition the heap into separate memory areas (young / tenured / permanent generations), it is enough for the permanent generation space to fill up completely for an out of memory error to occur. You can configure the ratio of different generation spaces using various JVM command line options.
Here is a good article about Tuning Garbage Collection with the 5.0 Java[tm] Virtual Machine (I couldn't find a more recent one but I think the basics still apply in newer VMs).
Re-read your error message - it may tell you which type of memory you ran out of. I'd guess that it was PermGen space. Permgen is used for class definitions (amongst other things). You can tune the space for PermGen via -XX:MaxPermSize PermGen isn't part of the heap, so isn't included in heap dump.
See this answer for more information on looking at PermGen.
If this isn't the issue, then try setting your initial heap size (-Xms) to the same as the maximum. Doing this means that the heap won't grow which should make understanding what's going on easier.
I recommend using jvisualvm (part of the JDK) to look at your memory utilisation as your program runs.
Related
I found that one of my spring boot project's memory (RAM consumption) is increasing day by day. When I uploaded the jar file to the AWS server, it was taking 582 MB of RAM (Max Allocated RAM is 1500 MB), but each day, the RAM is increasing by 50MB to 100 MB and today after 5 days, it's taking 835 MB. Right now the project is having 100-150 users and with normal usage of Rest APIs.
Because of this increase in the RAM, couple of times the application went down with the following error (error found from the logs):
Exception in thread "http-nio-3384-ClientPoller" java.lang.OutOfMemoryError: Java heap space
So to resolve this, I found that by using JAVA Heap Dump, I can find the objects/classes that are taking the memory. So by using Jmap in the command line, I've created a heap dump and uploaded it to Heap Hero and Eclipse Memory Analyzer Tool. In both of them I found the following:
1. Total Waste memory is: 64.69MB (73%) (check below screenshot)
2. Out of these, 34.06MB is taken by Byte [] array and LinkedHashmap[] (check below screenshot), which I have never used in my whole project. I searched for it in my project but didn't found.
3. Following 2 large objects taking 32 MB and 20 MB respectively.
1. Java Static io.netty.buffer.ByteBufUtil.DEFAULT_ALLOCATOR
2. Java Static com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.connectionFinalizerPhantomRefs`
So I tried to find this netty.buffer. in my project, but I don't find anything which matched with netty or buffer.
Now my question is how can I reduce this memory leak or how can I find the exact memory consumption objects/class/variable so that I can reduce the heap size.
I know few of the experts will ask for the source code or anything similar to that but I believe that from the heap dump we can find the memory leak or live objects that are available in the memory. I am looking for that option or anything that reduces this heap dump!
I am working on this issue for the past 3 weeks. Any help would be appreciated.
Thank you!
Start with enabling the JVM native memory tracker to get an idea which part of the memory is increasing by adding the flag -XX:NativeMemoryTracking=summary. There is some performance overhead according to the documentation (5-10%), but if this isn't a issue I would recommend running the JVM with this flag enabled even in production.
Then you can check the values using jcmd <PID> VM.native_memory (there's a good writeup in this answer: Java native memory usage)
If there is indeed a big chunk of native memory allocated, it's likely this is allocated by Netty.
How do you run your application in AWS? If it's running in a Docker image, you might have stumbled upon this issue: What would cause a java process to greatly exceed the Xmx or Xss limit?
If this is the case, you may need to set the environment variable MALLOC_ARENA_MAX if your application is using native memory (which Netty does) and running on a server with a large number of cores. It's perfectly possible that the JVM allocates this memory for Netty but doesn't see any reason to release it, so it will appear to only continue to grow.
If you want to control how much native memory can be allocated by Netty, you can use the JVM flag -XX:MaxDirectMemorySize for this (I believe the default is the same value as Xmx) and lower it in case you application doesn't require that much memory.
JVM memory tuning is a complex process and it becomes even more complex when native memory is involved - as the linked answer shows it's not as easy as simply setting the Xms and Xmx flag and expecting that no more memory will be used.
Heap dump is not enough to detect memory leaks.
You need to look at the difference of two consecutive heaps snapshots both taken after calling the GC.
Or you need a profiling tool that can give the generations count for each class.
Then you should only look at your domain objects (not generic objects like bytes or strings ...etc) that survived the GC and passed from the old snapshot to the new one.
Or, if using the profiling tool, look for old domain objects that still alive and growing for many generations.
Having objects lived for many generations and keeps growing means those objects are still refernced and the GC is not able to reclaim them. However, living for many generations alone is not enough to cause a leak because cached or static Objects may stay for many generations. The other important factor is that they keep growing.
After you detected what object is being leaked, you may use heap dumb to analyse those objects and get the references.
My Java program reads data from a stream and creates an in-memory cache of parts of it. At some point it throws an OutOfMemoryError, and I've caused it to create a heap dump at that time so that I can see what causes the issue.
But when I load the heap dump I see that about half of the memory is unused: I've started the VM with -Xmx8000m, and the heap dump, when loaded into Eclipse Memory Analyzer or VirtualVM only shows about 4GB in use. The dump file itself however is around 8GB in file size.
What is also odd is that both tools report a lot of int arrays of size int[262136] as "unreferenced objects", i.e. garbage. There is about 4GB of those - so that really points to them not being garbage but being the reason for the OOM..
My code does not create in arrays of this size, at all, btw.
Why do I get this OOM, and what is the matter with those int[] arrays?
I am running on a Java 11 JDK, but the same issue also occurs on Java 14.
This is a problem with Java's garbage collector, and it was very hard to find.
The default garbage collector for these versions is the G1 collector. This garbage collector divides the available memory into fixed size memory regions. These are always a power of 2 big, starting at 1MB, and depend on the -Xmx max memory parameter.
Those int[262136] arrays are a trick that the gc uses to somehow mark these regions as Java objects. This int array takes exactly 1mb of space, so it has the size of the region. It marks them as being unreferenced so most tools do not see them, or mark them as garbage. This is highly misleading as it seems to be the cause of the OOM issue.
The real reason for the OOM is that the caching code allocates (and releases) objects that are considered "Humongous objects" by the G1 garbage collector. It has huge problems with reclaiming or moving these objects, and this apparently causes memory fragmentation - which in turn causes the OOM even though enough memory appears to be available. For some reason the gc logging does not give any indication that this could be an issue 8-(.
A good test to see whether this is the cause of your issue is to run the same program with either the old "mark and sweep GC" (by adding the parameter -XX:+UseConcMarkSweepGC to the Java command line; this works best, but this gc has been removed starting from Java 15), or by trying with the parallel GC (by adding -XX:+UseParallelGC).
To solve this either use one of the above GC's, or play with the -XX:G1HeapRegionSize parameter. Set it to a larger power-of-2 size (like 2m, 4m, 16m) to see if that fixes the issue.
Some more information on this can be found on the site of jxray.com, a heap dump analysis tool: https://jxray.com/documentation#humongous_objs, and in an Oracle article about the G1 collector at https://www.oracle.com/technical-resources/articles/java/g1gc.html.
Problem
We are trying to find the culprit of a big memory leak in our web application. We have pretty limited experience with finding a memory leak, but we found out how to make a java heap dump using jmap and analyze it in Eclipse MAT.
However, with our application using 56/60GB memory, the heap dump is only 16GB in size and is even less in Eclipse MAT.
Context
Our server uses Wildfly 8.2.0 on Ubuntu 14.04 for our java application, whose process uses 95% of the available memory. When making the dump, our buffers/cache used space was at 56GB.
We used the following command to create the dump: sudo -u {application user} jmap -dump:file=/mnt/heapdump/dump_prd.bin {pid}
The heap dump file size is 16,4GB and when analyzing it with Eclipse MAT, it says there are around 1GB live objects and ~14,8GB unreachable/shallow heap.
EDIT: Here is some more info about the problem we see happening. We monitor our memory usage, and we see it grow and grow, until there is ~300mb free memory left. Then it stays around that amount of memory, until the process crashes, unfortunately without error in the application log.
This makes us assume it is a hard OOM error because this only happens when the memory is near-depleted. We use the settings -Xms25000m -Xmx40000m for our JVM.
Question
Basically, we are wondering why the majority of our memory isn't captured in this dump. The top retained size classes don't look too suspicious, so we are wondering if there is something heap dump-related what we are doing wrong.
When dumping its heap, the JVM will first run a garbage collection cycle to free any unreachable objects.
How can I take a heap dump on Java 5 without garbage collecting first?
In my experience, in a true OutOfMemoryError where your application is simply demanding more heap space than is available, this GC is a fool's errand and the final heap dump will be the size of the max. heap size.
When the heap dump is much smaller, that means the system was not truly out of memory, but perhaps had memory pressure. For example, there is the java.lang.OutOfMemoryError: GC overhead limit exceeded error, which means that the JVM may have been able to free enough memory to service some new allocation request, but it had to spend too much time collecting garbage.
It's also possible that you don't have a memory problem. What makes you think you do? You didn't mention anything about heap usage or an OutOfMemoryError. You've only mentioned the JVM's memory footprint on the operating system.
In my experience, having a heap dump much smaller than the real memory used can be due to a leak in the JNI.
Despite you don't use directly any native code, there are certain libraries that use it to speed up.
In our case, it was a Deflater and Inflater not properly ended.
We use Eclipse memory analyzer for Java (Tomcat Web application) dump. The total heap in the resulting piechart is shown as 86 Mb. At the same time the heap limit for that JVM is set at 1.5GB and we saw the total JVM usage going up to 2.8 GB.
Why the deviation is so big?
Please invoke jmap -heap TOMCAT_PID and you will see current heap usage (eden, survivor spaces, old and perm)
Also please notice that real usage of memory for Java will be XMX + MaxPerm + XSS * threads number. I'll recommend you reading great post about memory consuming in Java:
http://plumbr.eu/blog/why-does-my-java-process-consume-more-memory-than-xmx
Your heap memory is only a part of the memory used by the JVM. Additionally you have native memory and permgen.
You can limit the permgen memory via command line parameters. See What does PermGen actually stand for?. In my experience, the permgen limit was defaulted to something like 1G, which was way more than we ever needed. I think we overrode it to 128m.
Native memory is a lot trickier. This is memory allocated by native libraries used directly or transitively by your code.
In jrockit, you can get a print out of a memory summary via jrcmd print_memusage. Not sure how to do that in other JVMs.
Also: http://www.ibm.com/developerworks/linux/library/j-nativememory-linux/index.html
See this reference - MAT Does Not Show the Complete Heap :
Symptom: When monitoring the memory usage interactively, the used heap size is much bigger than what MAT reports.
During the index creation, the Memory Analyzer removes unreachable objects because the various garbage collector algorithms tend to leave some garbage behind (if the object is too small, moving and re-assigning addresses is to expensive). This should, however, be no more than 3 to 4 percent. If you want to know what objects are removed, enable debug output as explained here: MemoryAnalyzer/FAQ#Enable_Debug_Output
Also, there should be some more information in this Q/A: eclipse memory analyzer sees small part (363,2MB) of entire heap dump (8GB)
Try the Keep Unreachable Objects option in Preferences -> Memory Analyzer -> Keep Unreachable Objects.
I'm trying to understand why out ColdFusion 9 (JRun) server is throwing the following error:
java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
The JVM arguments are as follows:
-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -
I had jconsole running when the dump happened and I am trying to reconcile some numbers with the -XX:MaxPermSize=192m setting above. When JRun died it had the following memory usage:
Heap
PSYoungGen total 136960K, used 60012K [0x5f180000, 0x67e30000, 0x68d00000)
eden space 130624K, 45% used [0x5f180000,0x62c1b178,0x67110000)
from space 6336K, 0% used [0x67800000,0x67800000,0x67e30000)
to space 6720K, 0% used [0x67110000,0x67110000,0x677a0000)
PSOldGen total 405696K, used 241824K [0x11500000, 0x2a130000, 0x5f180000)
object space 405696K, 59% used [0x11500000,0x20128360,0x2a130000)
PSPermGen total 77440K, used 77070K [0x05500000, 0x0a0a0000, 0x11500000)
object space 77440K, 99% used [0x05500000,0x0a043af0,0x0a0a0000)
My first question is that the dump shows the PSPermGen being the problem - it says the total is 77440K, but it should be 196608K (based on my 192m JVM argument), right? What am I missing here? Is this something to do with the other non-heap pool - the Code Cache?
I'm running on a 32bit machine, Windows Server 2008 Standard. I was thinking of increasing the PSPermGen JVM argument, but I want to understand why it doesn't seem to be using its current allocation.
Thanks in advance!
An "out of swap space" OOME happens when the JVM has asked the operating system for more memory, and the operating system has been unable to fulfill the request because all swap (disc) space has already been allocated. Basically, you've hit a system-wide hard limit on the amount of virtual memory that is available.
This can happen through no fault of your application, or the JVM. Or it might be a consequence of increasing -Xmx etc beyond your system's capacity to support it.
There are three approaches to addressing this:
Add more physical memory to the system.
Increase the amount of swap space available on the system; e.g. on Linux look at the manual entry for swapon and friends. (But be careful that the ratio of active virtual memory to physical memory doesn't get too large ... or your system is liable to "thrash", and performance will drop through the floor.)
Cut down the number and size of processes that are running on the system.
If you got into this situation because you've been increasing -Xmx to combat other OOMEs, then now would be good time to track down the (probable) memory leaks that are the root cause of your problems.
"ChunkPool::allocate. Out of swap space" usually means the JVM process has failed to allocate memory for its internal processing.
This is usually not directly related to your heap usage as it is the JVM process itself that has run out of memory. Check the size of the JVM process within windows. You may have hit an upper limit there.
This bug report also gives an explanation.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5004956
This is usually caused by native, non java objects not being released by your application rather than java objects on the heap.
Some example causes are:
Large thread stack size, or many threads being spawned and not cleaned up correctly. The thread stacks live in native "C" memory rather than the java heap. I've seen this one myself.
Swing/AWT windows being programatically created and not dispoed when no longer used. The native widgets behind AWT don't live on the heap as well.
Direct buffers from nio not being released. The data for the direct buffer is allocated to the native process memory, not the java heap.
Memory leaks in jni invocations.
Many files opened an not closed.
I found this blog helpfull when diagnosing a similar problem. http://www.codingthearchitecture.com/2008/01/14/jvm_lies_the_outofmemory_myth.html
Check your setDomainEnv.cmd (.sh)file. there will be three different conditions on PermSize
-XX:MaxPermSize=xxxm -XX:PermSize=xxxm. Change everywhere