We're running against an issue related with memory.
Our monitor is saying that out service is consuming 800MiB of memory.
We've created a heap dump. However, this heap dump is telling us that total heap size is 41MiB.
Out service is running inside a container. We've read a lot some issue related with java and docker containers. However, we're trying to get some more information.
How could we otain more memory related information about java vm?
Any ideas?
The big discrepancy between the memory usage reported by the heap dump and the usage reported by external monitoring could be due to a few things. Some things to look at include:
A JVM's memory utilization does not just consist of the Java heap. It includes the JVM executable and shared libraries, metaspace, the native (malloc) heap and memory segments that hold thread stacks. It potentially also includes memory mapped files, and memory segments shared with other processes.
The memory used by the Java heap includes objects that are unreachable, and space that isn't currently allocated to objects; i.e. free heap space.
How can you get more information about Java memory utilization?
A primary source is "Java Platform, Standard Edition Troubleshooting Guide":
The Native Memory Tracking section explains how to use jcmd to get an overall picture.
The JFR Memory Tab provides live monitoring of Java Heap usage stats and the GC.
The Jmap Utility can be used to obtain heap and GC configuration information, heap histograms and classloader stats.
GC logging can give useful insights into GC behavior.
Related
I'm struggling to understand why my Java application is slowly consuming all memory available to the pod causing Kubernetes to mark the pod as out of memory. The JVM (OpenJDK 8) is started with the following arguments:
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2
I'm monitoring the memory used by the pod and also the JVM memory and was expecting to see some correlation e.g. after major garbage collection the pod memory used would also fall. However I don't see this. I've attached some graphs below:
Pod memory:
Total JVM memory
Detailed Breakdown of JVM (sorry for all the colours looking the same...thanks Kibana)
What I'm struggling with is why when there is a significant reduction in heap memory just before 16:00 does the pods memory not also fall?
It looks like you are creating a pod with a resource limit of 1GB Memory.
You are setting -XX:MaxRAMFraction=2 which means you are allocating 50% of available memory to the JVM which seem to match what you are graphing as Memory Limit.
JVM then reserves around 80% of that which is what you are graphing in Memory Consumed.
When you look at Memory Consumed you will not see internal garbage collection (as in your second graph), because that GC memory is released back to JVM but is still reserved by it.
Is it possible that there is a memory leak in your java application? it is possibly causing more memory to get reserved over time, until the JVM limit (512MB) is met and your pod gets OOM killed.
Problem
We are trying to find the culprit of a big memory leak in our web application. We have pretty limited experience with finding a memory leak, but we found out how to make a java heap dump using jmap and analyze it in Eclipse MAT.
However, with our application using 56/60GB memory, the heap dump is only 16GB in size and is even less in Eclipse MAT.
Context
Our server uses Wildfly 8.2.0 on Ubuntu 14.04 for our java application, whose process uses 95% of the available memory. When making the dump, our buffers/cache used space was at 56GB.
We used the following command to create the dump: sudo -u {application user} jmap -dump:file=/mnt/heapdump/dump_prd.bin {pid}
The heap dump file size is 16,4GB and when analyzing it with Eclipse MAT, it says there are around 1GB live objects and ~14,8GB unreachable/shallow heap.
EDIT: Here is some more info about the problem we see happening. We monitor our memory usage, and we see it grow and grow, until there is ~300mb free memory left. Then it stays around that amount of memory, until the process crashes, unfortunately without error in the application log.
This makes us assume it is a hard OOM error because this only happens when the memory is near-depleted. We use the settings -Xms25000m -Xmx40000m for our JVM.
Question
Basically, we are wondering why the majority of our memory isn't captured in this dump. The top retained size classes don't look too suspicious, so we are wondering if there is something heap dump-related what we are doing wrong.
When dumping its heap, the JVM will first run a garbage collection cycle to free any unreachable objects.
How can I take a heap dump on Java 5 without garbage collecting first?
In my experience, in a true OutOfMemoryError where your application is simply demanding more heap space than is available, this GC is a fool's errand and the final heap dump will be the size of the max. heap size.
When the heap dump is much smaller, that means the system was not truly out of memory, but perhaps had memory pressure. For example, there is the java.lang.OutOfMemoryError: GC overhead limit exceeded error, which means that the JVM may have been able to free enough memory to service some new allocation request, but it had to spend too much time collecting garbage.
It's also possible that you don't have a memory problem. What makes you think you do? You didn't mention anything about heap usage or an OutOfMemoryError. You've only mentioned the JVM's memory footprint on the operating system.
In my experience, having a heap dump much smaller than the real memory used can be due to a leak in the JNI.
Despite you don't use directly any native code, there are certain libraries that use it to speed up.
In our case, it was a Deflater and Inflater not properly ended.
We use Eclipse memory analyzer for Java (Tomcat Web application) dump. The total heap in the resulting piechart is shown as 86 Mb. At the same time the heap limit for that JVM is set at 1.5GB and we saw the total JVM usage going up to 2.8 GB.
Why the deviation is so big?
Please invoke jmap -heap TOMCAT_PID and you will see current heap usage (eden, survivor spaces, old and perm)
Also please notice that real usage of memory for Java will be XMX + MaxPerm + XSS * threads number. I'll recommend you reading great post about memory consuming in Java:
http://plumbr.eu/blog/why-does-my-java-process-consume-more-memory-than-xmx
Your heap memory is only a part of the memory used by the JVM. Additionally you have native memory and permgen.
You can limit the permgen memory via command line parameters. See What does PermGen actually stand for?. In my experience, the permgen limit was defaulted to something like 1G, which was way more than we ever needed. I think we overrode it to 128m.
Native memory is a lot trickier. This is memory allocated by native libraries used directly or transitively by your code.
In jrockit, you can get a print out of a memory summary via jrcmd print_memusage. Not sure how to do that in other JVMs.
Also: http://www.ibm.com/developerworks/linux/library/j-nativememory-linux/index.html
See this reference - MAT Does Not Show the Complete Heap :
Symptom: When monitoring the memory usage interactively, the used heap size is much bigger than what MAT reports.
During the index creation, the Memory Analyzer removes unreachable objects because the various garbage collector algorithms tend to leave some garbage behind (if the object is too small, moving and re-assigning addresses is to expensive). This should, however, be no more than 3 to 4 percent. If you want to know what objects are removed, enable debug output as explained here: MemoryAnalyzer/FAQ#Enable_Debug_Output
Also, there should be some more information in this Q/A: eclipse memory analyzer sees small part (363,2MB) of entire heap dump (8GB)
Try the Keep Unreachable Objects option in Preferences -> Memory Analyzer -> Keep Unreachable Objects.
Is there anywhere that a hostspot JVM process stores memory besides these places:
perm gen
tenured generation
eden space
from space
to space
code cache
That is to say: What possible ways are there that the hotspot process can reserve & commit memory on a machine such that said memory would not show up in the statistics for one of those places?
Some answers I can think of (let me know if these are true):
virtual memory used for thread stacks are not represented in those numbers
any loaded dlls or files.
EDIT:
some other answers given:
java.exe itself
JNI methods could allocate memory itself
any native code (eg. from dlls) could allocate memory.
general JVM metadata for running itself.
You're correct so far (DLLs include all JNI libraries and whatever memory they've allocated). The VM also has its own code (e.g., the contents of java), bookkeeping information about the memory that's allocated to Java programs, and potentially memory used by VM agents. Basically, what you described in your first list are the items that make up the "running memory" of the virtual machine; the rest of the JVM's memory are all of the items that represent the virtual machine's "hardware", such as the libraries that connect it to the OS's networking, graphics, and so on.
I'm trying to understand why out ColdFusion 9 (JRun) server is throwing the following error:
java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
The JVM arguments are as follows:
-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -
I had jconsole running when the dump happened and I am trying to reconcile some numbers with the -XX:MaxPermSize=192m setting above. When JRun died it had the following memory usage:
Heap
PSYoungGen total 136960K, used 60012K [0x5f180000, 0x67e30000, 0x68d00000)
eden space 130624K, 45% used [0x5f180000,0x62c1b178,0x67110000)
from space 6336K, 0% used [0x67800000,0x67800000,0x67e30000)
to space 6720K, 0% used [0x67110000,0x67110000,0x677a0000)
PSOldGen total 405696K, used 241824K [0x11500000, 0x2a130000, 0x5f180000)
object space 405696K, 59% used [0x11500000,0x20128360,0x2a130000)
PSPermGen total 77440K, used 77070K [0x05500000, 0x0a0a0000, 0x11500000)
object space 77440K, 99% used [0x05500000,0x0a043af0,0x0a0a0000)
My first question is that the dump shows the PSPermGen being the problem - it says the total is 77440K, but it should be 196608K (based on my 192m JVM argument), right? What am I missing here? Is this something to do with the other non-heap pool - the Code Cache?
I'm running on a 32bit machine, Windows Server 2008 Standard. I was thinking of increasing the PSPermGen JVM argument, but I want to understand why it doesn't seem to be using its current allocation.
Thanks in advance!
An "out of swap space" OOME happens when the JVM has asked the operating system for more memory, and the operating system has been unable to fulfill the request because all swap (disc) space has already been allocated. Basically, you've hit a system-wide hard limit on the amount of virtual memory that is available.
This can happen through no fault of your application, or the JVM. Or it might be a consequence of increasing -Xmx etc beyond your system's capacity to support it.
There are three approaches to addressing this:
Add more physical memory to the system.
Increase the amount of swap space available on the system; e.g. on Linux look at the manual entry for swapon and friends. (But be careful that the ratio of active virtual memory to physical memory doesn't get too large ... or your system is liable to "thrash", and performance will drop through the floor.)
Cut down the number and size of processes that are running on the system.
If you got into this situation because you've been increasing -Xmx to combat other OOMEs, then now would be good time to track down the (probable) memory leaks that are the root cause of your problems.
"ChunkPool::allocate. Out of swap space" usually means the JVM process has failed to allocate memory for its internal processing.
This is usually not directly related to your heap usage as it is the JVM process itself that has run out of memory. Check the size of the JVM process within windows. You may have hit an upper limit there.
This bug report also gives an explanation.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5004956
This is usually caused by native, non java objects not being released by your application rather than java objects on the heap.
Some example causes are:
Large thread stack size, or many threads being spawned and not cleaned up correctly. The thread stacks live in native "C" memory rather than the java heap. I've seen this one myself.
Swing/AWT windows being programatically created and not dispoed when no longer used. The native widgets behind AWT don't live on the heap as well.
Direct buffers from nio not being released. The data for the direct buffer is allocated to the native process memory, not the java heap.
Memory leaks in jni invocations.
Many files opened an not closed.
I found this blog helpfull when diagnosing a similar problem. http://www.codingthearchitecture.com/2008/01/14/jvm_lies_the_outofmemory_myth.html
Check your setDomainEnv.cmd (.sh)file. there will be three different conditions on PermSize
-XX:MaxPermSize=xxxm -XX:PermSize=xxxm. Change everywhere