So every couple of days my java process on Ubuntu is killed automatically, and I can't figure out why.
My box has 35.84 GB of RAM, when I launch my Java process I pass it the -Xmx28g parameter, so it should be using way less than the maximum RAM available.
I ran jstat as follows:
# jstat -gccause -t `pgrep java` 60000
The last few lines of output from jstat immediately before the process was killed were:
Time S0 S1 E O P YGC YGCT FGC FGCT GCT LGCC GCC
14236.1 99.98 0.00 69.80 99.40 49.88 1011 232.305 11 171.041 403.347 unknown GCCause No GC
14296.2 93.02 0.00 65.79 99.43 49.88 1015 233.000 11 171.041 404.041 unknown GCCause No GC
14356.1 79.20 0.00 80.50 99.55 49.88 1019 233.945 11 171.041 404.986 unknown GCCause No GC
14416.2 0.00 99.98 24.32 99.64 49.88 1024 234.945 11 171.041 405.987 unknown GCCause No GC
This seems to be what went down in the /var/log/syslog around this time: https://gist.github.com/1369135
There is really nothing running on this server other than my java app. What's going on?
edit: I'm running java version 1.6.0_20, the only notable parameters I'm passing to java on startup are "-server -Xmx28g". I'm not using an application server but my app embeds the "Simple web framework".
Assuming the problem is the OOM killer, then it has killed your process in a desperate attempt to keep the OS functioning in a severe memory shortage crisis.
I would conclude that:
your JVM is actually using significantly more than 28Gb; i.e. you've got significant non-heap memory usage, and
the OS is not configured with an adequate amount of swap space.
I'd try adding more swap space, so that the OS can swap out parts of your application in an emergency.
Alternatively, reduce the JVM's heap size.
Note that "-Xmx ..." sets the maximum heap size, not the maximum amount of memory that your JVM can use. The JVM puts some stuff outside the heap, including such things as the memory for thread stacks and memory-mapped files that your application is using.
The syslog confirms that it is the OOM killer at work.
In what way does the linked syslog say so?
It says this:
Nov 15 13:53:49 ip-10-71-94-36 kernel: [3707038.606133] Out of memory: kill process 6368 (run.sh) score 4747288 or a child
Nov 15 13:53:49 ip-10-71-94-36 kernel: [3707038.606146] Killed process 9359 (java)
The console says that java was killed, not that it quit.
Correct. It was killed by the operating system's OOM killer.
If it had run out of memory it would typically throw an OutOfMemory exception, which it didn't.
That is what would have happened if you had filled up the Java heap.
That is not what is going on here. The actual problem is that there is not enough physical RAM to hold the Java heap. The OOM killer deals with it ...
I'm running with such a huge heap because I need to store millions of objects each of which require several kilobytes of RAM.
Unfortunately, you are trying to use way more RAM than is available on the system. This is causing virtual memory to thrash, affecting the entire operating system.
When the system starts to thrash badly, the OOM killer (not the JVM) identifies your Java process as the cause of the problem. It then kills it (with a SIGKILL) to protect the rest of the system. If it didn't, there is a risk that the entire system would lock up completely and need to be hard rebooted.
Finally, you said:
My box has 35.84 GB of RAM ...
That is rather a strange value. 32 GiB is 34,359,738,368 bytes or 34.35 GB.
But based on that and the observed behavior, I suspect that that is the available virtual memory rather than physical RAM. Alternatively, your "box" could be a virtual machine with RAM overcommit enabled at the hypervisor level.
Welcome to the OOM-killer, a linux 'feature' that is the bane of large-memory applications everywhere. There's no simple recipe to deal, just google for it and start reading and weaping.
While I can't put my mental fingers on a concise explanation of the shenigans of the OOM killer, I recall that the critical tuning parameter is called 'swappiness'. On one of our big servers, we have:
/etc/sysctl.conf:vm.swappiness=20
Read http://www.gentooexperimental.org/~patrick/weblog/archives/2009-11.html.
What JVM are you using? and what application server? It's possible that you're allocating way too much memory, and that can be problematic - the garbage collector might have trouble doing its job.
I'm not sure if this is your case, but I found quite interesting this article explaining the way Linux overcommits memory.
wow, can you actually have 28 GB of heap?! May be you should try reducing it, keep it at no more than 50% of the RAM I think (so ~18 GB, or may be even 15 GB). Plus 171 Full GCs are a lot! How long was this app running? 171 in 2-3 days sounds huge. btw the gist indicates an OOM before termination - I think reducing the heap will fix it ( you may be limiting the JVM from expanding native space). Try adjusting various parameters, try stack size for example (-Xss) if needed. Check max perm size and other sections too. Its a memory problem and it may not necessarily be the heap.
Ubuntu has a "watchdog" process which kills other processes when memory runs low.
See the manpage:
http://manpages.ubuntu.com/manpages/natty/man8/watchdog.8.html
Related
What is the possible reason of constant OOM errors?
On our developement environment we have 3 nodes, VM on master node is initialized with: Xms=3GB, Xmx=3GB. Also there are about 30 proxies and 30 endpoints defined. Developers loading their changes (car files) constantly during the day without restarting carbon. Few times a day it freezes. Maybe constant configartion changes kill carbon? On preproduction environment carbon works flawlessly :/
I did heap dump and the 'leak suspect report' result is:
One instance of "org.apache.axis2.context.ConfigurationContext" loaded
by "axis2" occupies 661 590 744 (79,50%) bytes. The memory is
accumulated in one instance of
"java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "".
Report result:
Histogram:
According to the Heap Dump most of the retained heap is occupied by the ConfigurationContext instances. So this OOM issue seems to occur due to the heavy configuration load in your development system. May be due to large capps deployed frequently by lot of developers. Since that incident does not happen in your production, that ESB never goes OOM.
Thanks & Regards,
Ravindra.
Freezing happens mostly due to full gc cycles happening in the system, You can change your gc settings in order to do a proper gc configuration
For a typical deployment wso2 suggest to have below GC configuration. -Xms512m -Xmx2048m -XX:MaxPermSize=1024m Since you are using 3gb of RAM you have to adjust that based on your scenario. You can refer to https://docs.wso2.com/display/CLUSTER44x/Production+Deployment+Guidelines for more information regarding the deployments
Since you are using Xms=3GB, Xmx=3GB there is a possibility it will wait until the whole 3gb get filled and then do full gc cycles. Therefore better to adjust the Xms value to about 1gb so it motivate to have minor gc cycles and cleanup the unnecessary stuffs rather than waiting for full gc
I have a VPS with 20GB RAM, Ubuntu OS. I am trying to allocate 10GB RAM as the maximum heap to java using JAVA_TOOL_OPTIONS but I couldn't. Please see the attached screenshot. It shows available memory as 17GB. Its working when I try to set to 7GB. But heap error occurs only when it is > 7GB. I have already installed glassfish and allocated 3Gb to its cluster. Its working fine. But why I am not able to allocate greater than 7GB when I have 17GB RAM free.
TOP
ULIMITS
Java -version
OverCommit memory
My Hardware is Virtual Hosted. Below is the configuration
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
Vendor ID: GenuineIntel
CPU family: 6
Model: 26
Stepping: 5
CPU MHz: 2266.802
BogoMIPS: 4533.60
Virtualization: VT-x
If I had to guess, you don't have a contiguous block of RAM that's 7GB, which does seem weird, but without knowing more about your VM's allocation it's hard to say.
Here's what Oracle has to say on the matter (http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#gc_oom):
The VM prints "OutOfMemoryError" and exits. Increasing max heap size
doesn't help. What's going on?
The Java HotSpot VM cannot expand its heap size if memory is
completely allocated and no swap space is available. This can occur,
for example, when several applications are running simultaneously.
When this happens, the VM will exit after printing a message similar
to the following.
Exception java.lang.OutOfMemoryError: requested bytes
-Xmx-Xms-Xmx
For more information, see the evaluation section of bug 4697804.
I think you may be out of swap space. When I add up the memory in the "virt" column, it comes to 40+ Gb.
Why it's taking that much swap space ? What needs to be done in order to fix this ?
Well, according to top you are running:
Glassfish - 9.1G
MySQL daemon - 5.4G
Hudson - 8.9G
Nexus - 6G
Glassfish - 6.9G (2nd instance)
and sundry other stuff. The "virt" is their total virtual memory footprint, and some of that will be code segments which may be shared.
They mostly seem to have a small "res" (resident memory) at the moment which is why there is so much free RAM. However, if a few of them sprang into life at the same time the system the demand for RAM would skyrocket, and the system might start to thrash.
My recommendation would be to move the Hudson and Nexus services to a separate VM. Or if that is not possible, increase the size of your swap space ... and hope that you don't thrash.
This is true. But is this a normal behaviour?
Yes.
is this how memory allocation works?
Yes. This is indeed how virtual memory works.
I am confused with Resident memory, virtual memory and physical memory now.
Rather than explain it in detail, I suggest you start by reading the Wikipedia page on virtual memory.
The reason why I wasn't able to allocate more than 5G is because of the fact that privvmpages is set to 5G.
We can get that information in linux by this command "cat /proc/user_beancounters"
Also, in VPS, hosting provider will not allow us to customize this value. We have to either go for large virtual or dedicated server to increase this limit.
This was the root cause. However, Stephen and Robin's explanations on the Virtual Memory and RES memory were spot on. Thanks Guys
My tomcat was auto-shutdown suddenly.I checked in log file and found that It was killed with message:
kernel: Killed process 17420, UID 0, (java) total-vm:8695172kB, anon-rss:4389088kB, file-rss:20kB
My setting for running tomcat is -Xms2048m -Xmx4096m -XX:NewSize=256m -XX:MaxNewSize=512m -XX:PermSize=256m -XX:MaxPermSize=1024m
My system when run command "free -m" is:
total used free shared buffers cached
Mem: 7859 7713 146 0 97 1600
-/+ buffers/cache: 6015 1844 Swap: 0 0 0
I monitor program with "top -p", the result as below
Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 8048440k total, 7900616k used, 147824k free, 100208k buffers Swap: 0k total, 0k used, 0k free, 1640888k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4473 root 20 0 8670m 2.5g 6568 S 0.0 32.6 71:07.84 java
My question is:
1.Why VIRT = 8670m (in "top -p" result) is greater than Mem: 8048440k total but my application is still running?
Why my tomcat was kill by kernel? I don't see any strange with memory (It's similar with when it's running)
To avoid this error happen, what will I do and why?
The only thing I know that causes the kernel to kill tasks in Linux is the out of memory killer. This article from Oracle might be a little more recently and relevant.
The solution depends on what else is running on the system. From what you showed, you have less than 2GB of usable memory, but your Java heap max is topping out around 4GB. What we don't know is how big the Java heap is at the time you took that snapshot. If it's at its initial 2GB, then you could be running close to the limit. Also based on your formatting, you have no swap space to use as a fallback.
If you have any other significant processes on the system, you need to account for their maximum memory usage. The short answer is try to reduce the Xmx and MaxPermSize if at all possible, you'll have to analyze your load to see if this is possible or will cause unreasonable GC CPU usage.
Some notes:
Java uses more memory than the heap, it has memory for the native code running the VM itself.
Java 8 stores permgen outside of the heap, so I believe it adds memory on top of the Xmx parameter, you may want to note that if running Java 8.
As you reduce the memory limit, you'll hit 3 ranges:
Far above real requirements: no noticeable difference
Very close to real requirements: server freezes/stops responding and uses 100% CPU (GC overhead)
Below real requirements: OutOfMemoryErrors
It's possible for a process's VM size to exceed RAM+swap size per your first question. I remember running Java on a swapless embedded system with 256MB RAM and seeing 500MB of memory usage and being surprised. Some reasons:
In Linux you can allocate memory, but it's not actually used until you write to it
Memory-mapped files (and probably things like shared memory segments) count towards this limit. I believe Java opens all of the jar files as memory mapped files so included in that virt size are all of the jars on your classpath, including the 80MB or so rt.jar.
Shared objects probably count towards VIRT but only occupy space once (i.e. one copy of so loaded for many processes)
I've heard, but I can't find a reference right now, that Linux can actually use binaries/.so files as read-only "swap" space, meaning essentially loading a 2MB binary/so will increase your VM size by 2MB but not actually use all of that RAM, because it pages in from disk only the parts actually accessed.
Linux OS has a OOM Mechanism, when OS's memory is insufficient. The OOM will kill the cost max memory program(In most situations, Linux Out Of Memory Management). Obviously Your tomcat own the max memory.
How to solve? In my experience, you must observe the memory usage of OS, you can use the top command to observe, and find the proper process. and at the same time, you can use the jvisualvm to observe the usage memory of tomcat.
I have a Jetty server that I use for websocket connections for an app I am working on. The only issue is that Jetty is consuming way too much virtual memory (!2.5GB of virtual memory) and around 650RES.
My issue is that as mentioned above, most of the memory (around 12gb) is not the heap size so analyzing it and understanding what is happening is harder.
Do you have any tips on how to understand where the 12gb consumption is coming from and how to figure out memory leaks or any other issues with the server?
I wanted to clerify what I mean by virtual memory (because my understanding could be wrong). Virtual memory is "VIRT" when I run top. Here is what I get:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
-------------------------------------------------------------
9442 root 20 0 12.6g 603m 10m S 0 1.3 1:50.06 java
Thanks!
Please paste the JVM Options you use on startup. You can adjust the maximum memory used by the JVM with the -Xmx option as already mentioned.
Your application has been using only 603MB reserved memory. So doesn't look like it should concern you. You can get some detailed information about memory usage by either using "jmap", enable jmx and connect via jconsole or use a profiler. If you want to stay in *nix land you can also try "free" if your OS supports it.
In your case Jetty is NOT occupying 12,5 gig of memory. It's occupying 603MB. Google for "virtual memory linux" for example and you should get plenty of information about the difference between virtual and reserved memory.
Virtual memory has next to no cost in a 64-bit environment so I am not sure what the concern is. The resident memory is 650 MB or a mere 1.3% of MEM. It's not clear it is using much memory.
The default maximum heap size is 1/4 of the main memory for 64-bit JVMs. If you have 48 GB of memory you might find the default heap size is 12 GB and with some shared libraries, threads etc this can result in a virtual memory size of 12.5 GB. This doesn't mean you have a memory leak, or that you even have a problem but if you would prefer you can reduce the maximum heap size.
BTW: You can buy 32 GB for less than $200. If you are running low on memory, I would buy some more.
I am just testing some JNI calls with a simple stand alone java program on a 16-core CPU. I am using 64-bit JVM & OS is Linux AS5. But, as soon as I start my test program with 64-bit c++ libraries, I see that the 'SIZE' column shows 16G. The top command output is something like this:
PID PSID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
3505 31483 xxxxxxx 23 16 0 16G 215M sleep 0:02 0.00% java
I understand that my heap is ok, but JNI memory can increase the process size, but I am confused as to why it starts with 16G - SIZE, which I believe is the Virtual Memory size? Is it really taking that much memory? Should I be concerned with it?
Odd, but it could be that your JNI calls are either somehow rapidly leaking memory or allocating a number of massive chunks of memory. What command line are you using to start the JVM, and what is your JNI code doing?
The comment by Greg Hewgill basically says it all: You're using only 215 MB of real memory. On my machine a nothing-doing Java process without any VM arguments gets 8 GB. You've got four times as many cores, so starting with twice as much memory makes sense to me.
As this number uses nothing but the virtual address space, which is something like 2*48 B or maybe 2*64 B, it's actually free. You could use -mx1G in order to limit the memory to 1 GB, but note that it is a hard limit and you'll get an OOME when the process needs more.