I am running a java application with -Xms120g -Xmx120g on Mac laptop with 16GB of memory. What I saw was that the application was running and it consumed uptil now 36 GB of memory. I am guessing the Mac OS is using virtual memory to meet the demands of the application. However, I was wondering if I set a value of -Xms to something that the machine can't possibly provide will the application crash immediately or will it run and then only crash when it actually uses all the memory that the system can provide.
Found an explaination here:
Why am I able to set -Xmx to a value greater than physical and virtual memory on the machine on both Windows and Solaris?
So it works to excede the available virtual memory but eventually the metadata needed to handle the heap will need to much memory and it crashes.
Related
I have a VPS with 20GB RAM, Ubuntu OS. I am trying to allocate 10GB RAM as the maximum heap to java using JAVA_TOOL_OPTIONS but I couldn't. Please see the attached screenshot. It shows available memory as 17GB. Its working when I try to set to 7GB. But heap error occurs only when it is > 7GB. I have already installed glassfish and allocated 3Gb to its cluster. Its working fine. But why I am not able to allocate greater than 7GB when I have 17GB RAM free.
TOP
ULIMITS
Java -version
OverCommit memory
My Hardware is Virtual Hosted. Below is the configuration
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
Vendor ID: GenuineIntel
CPU family: 6
Model: 26
Stepping: 5
CPU MHz: 2266.802
BogoMIPS: 4533.60
Virtualization: VT-x
If I had to guess, you don't have a contiguous block of RAM that's 7GB, which does seem weird, but without knowing more about your VM's allocation it's hard to say.
Here's what Oracle has to say on the matter (http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#gc_oom):
The VM prints "OutOfMemoryError" and exits. Increasing max heap size
doesn't help. What's going on?
The Java HotSpot VM cannot expand its heap size if memory is
completely allocated and no swap space is available. This can occur,
for example, when several applications are running simultaneously.
When this happens, the VM will exit after printing a message similar
to the following.
Exception java.lang.OutOfMemoryError: requested bytes
-Xmx-Xms-Xmx
For more information, see the evaluation section of bug 4697804.
I think you may be out of swap space. When I add up the memory in the "virt" column, it comes to 40+ Gb.
Why it's taking that much swap space ? What needs to be done in order to fix this ?
Well, according to top you are running:
Glassfish - 9.1G
MySQL daemon - 5.4G
Hudson - 8.9G
Nexus - 6G
Glassfish - 6.9G (2nd instance)
and sundry other stuff. The "virt" is their total virtual memory footprint, and some of that will be code segments which may be shared.
They mostly seem to have a small "res" (resident memory) at the moment which is why there is so much free RAM. However, if a few of them sprang into life at the same time the system the demand for RAM would skyrocket, and the system might start to thrash.
My recommendation would be to move the Hudson and Nexus services to a separate VM. Or if that is not possible, increase the size of your swap space ... and hope that you don't thrash.
This is true. But is this a normal behaviour?
Yes.
is this how memory allocation works?
Yes. This is indeed how virtual memory works.
I am confused with Resident memory, virtual memory and physical memory now.
Rather than explain it in detail, I suggest you start by reading the Wikipedia page on virtual memory.
The reason why I wasn't able to allocate more than 5G is because of the fact that privvmpages is set to 5G.
We can get that information in linux by this command "cat /proc/user_beancounters"
Also, in VPS, hosting provider will not allow us to customize this value. We have to either go for large virtual or dedicated server to increase this limit.
This was the root cause. However, Stephen and Robin's explanations on the Virtual Memory and RES memory were spot on. Thanks Guys
I have a Java SE desktop application which uses a lot of memory (1,1 GB would be desired). All target machines (Win 7, Win Vista) have plenty of physical memory (at least 4GB, most of them have more). There is also enough free memory.
Now, when the machines have some uptime and a lot of programs were started and terminated, the memory becomes fragmented (this is what I assume). This leads to the following error when the JVM is started:
JVM creation failed
Error occurred during initialization of VM
Could not reserve enough space for object heap
Even closing all running programs doesn't help in such a situation (despite Task Manager and other tools report enough free memory). The only thing thas helps is to reboot the machine and fire up the Java applicaton as one of the first programs launched.
As far as I've investigated, the Oracle VM requires one contiguous chunk of memory.
Is there any other way to assign about 1,1 GB of heap to my java application when this amount is available but may be fragmented?
I start my JVM with the following arguments:
-J-client -J-Xss2m -J-Xms512m -J-Xmx1100m -J-XX:PermSize=64m -J-Dsun.zip.disableMemoryMapping=true
Is there any other way to assign about 1,1 GB of heap to my java application when this amount is available but may be fragmented?
Use an OS which doesn't get fragmented virtual memory. e.g. 64-bit windows or any version of UNIX.
BTW It is hard for me to imagine how this is possible in the first place but I know it to be the case. Each process has its own virtual memory so its arrangement of virtual memory shouldn't depend on anything which is already running or has run before.
I believe it might be a hang over from the MS-DOS TSR days. Shared libraries loaded are given absolute addresses (added to the end of the signed address space, 2 GB, the high half is reserved for the OS and the last 512 MB for the BIOS) in memory meaning they must use the same address range in every program they are used in. Over time the maximum address is determined by the lowest shared library loaded or used (I don't know which one by I suspect the lowest loaded)
Using Oracle Java 1.7.0_05 on Ubuntu Linux 3.2.0-25-virtual, on an amazon EC2 instance having 7.5 GB of memory, we start three instances of java, each using the switch -Xmx 2000m.
We use the default Ubuntu EC2 AMI configuration of no swap space.
After running these instances for some weeks, one of them freezes -- possibly out of memory. But my question isn't about finding our memory leak.
When we try to restart the app, java gives us a message that it cannot allocate the 2000 mb of memory. We solved the problem by rebooting the server.
In other words, 2000 + 2000 + 2000 > 7500?
We have seen this issue twice, and I'm sorry to report we don't have good diagnostics. How could we run out of space with only two remaining java processes each using a max of 2000 mb? How should we proceed to diagnose this problem the next time it occurs? I wish I had a "free -h" output, taken while we cannot start the program, to show here.
TIA.
-Xmx sets the maximum size of the JVM heap, not the maximum size of the java process, which allocates more memory besides the heap available to the application: its own memory, the Permanent generation, what's allocated inside JNI libraries, etc.
There may be other processes using memory therefore the JVM cannot be started with 2G. If you really need that much memory for 3 Java processes each and you only have 7.5 total you might want to change your EC2 configuration to have more memory. Your just leaving 1.5 for everything else include the kernal, Oracle etc.
I am working on java application and I have set the following configurations in the VM option
-Xms and -Xmx options are set to 1024m
-XX:MaxPermSize=128m Hardware :32 bit Windows 7 system with 2GB ram.
I often encounter with java out of swap space error. What could be the reason? Please help me.
The reason is that your operating system is not configured with enough swap space for the job mix that you are running. Swap space is an area on disc where the operating system puts copies memory pages when there are more virtual memory pages than physical memory pages.
So what has happened is that your JVM has asked for more virtual memory than the operating system can give it.
(Updated to include Peter's comments)
Some possible fixes:
Add more physical memory assuming that the hardware and OS allow this. (In this case, the answer the OS should allow it ...)
Configure the system with more swap space.
Kill some of the other non-essential applications and services running on the machine.
Change the Java application's JVM options to reduce the heap size.
The Java release notes under "Hotspot VM" say about this exact error;
If you see this symptom, consider increasing the available swap space
by allocating more of your disk for virtual memory and/or by limiting
the number of applications you run simultaneously. You can also
decrease your usage of memory by reducing the value of the -Xmx flag,
which limits the size of the Java object heap.
In other words, your machine is doing too many other things using up memory to be able to supply the 1GB you're telling the Java VM that it should be able to use.
I have defined -Xmsx 1.3GB in the java VM parameters and my Eclipse does not allow more than this, when running the application I got the below exception:
Exception in thread "Thread-3" java.lang.OutOfMemoryError: Java heap space
What can I do?
You can set the maximum memory eclipse uses with -mx1300m or the like. This limitation will be because you are running 32-bit java on Windows. On a 64-bit OS, you won't have this problem.
However, its the maximum memory size you set for each application in eclipse which matters. What have you set in your run options in eclipse?
Your question is very unclear:
Are you running the application in a new JVM?
Did you set the -Xmx / -Xms parameters in the launcher for the child JVM?
If the answer to either of those questions is "no", then try doing ... both. (In particular, if you don't set at least -Xmx for the child JVM, you'll get the default heap size which is relatively small.)
If the answer to both of those questions is "yes", then the problem is that you are running into the limits of your hardware and/or operating system configuration:
On a typical 32bit Windows, a user process can only address a total 2**31 bytes of virtual memory, and some of that will be used by the JVM binaries, native libraries and various non-heap memory allocations. (On a 32 bit Linux, I believe you can have up to 2**31 + 2**30). The "fix" for this is to use a 64bit OS and a 64bit JVM.
In addition, a JVM is limited on the amount of memory that it can request by the resources of the OS'es virtual memory subsystem. This is typically bounded by the sum of the available RAM and the size of the disc files / partitions used for paging. The "fix" for this is to increase the size of the paging file / partition. Adding more RAM would probably be a good idea too.
You may want to look at the aggressive Heap option http://java.sun.com/docs/hotspot/gc1.4.2/#4.2.2.%20AggressiveHeap|outline
It solved a similiar issue for me.