I had memory problems with one server. It's an amazon micro instance, so its memory is very limited (free -m says 603 MB). That's why I started tomcat with
-server -Xmx290m -Xms290m -XX:MaxPermSize=65m
However, the "java" process takes around 86% of the total memory, which is 518M. 518-355 = 163 MB overhead. That looks like a lot, and is suspicious, especially given than:
a similar application ran on another jvm version on another micro instance doesn't have overhead this big
the same application run locally gives just 40 MB overhead. Locally it runs in Windows 7, 64 bit.
The java version on the problematic server is:
java version "1.7.0_09-icedtea"
OpenJDK Runtime Environment (amzn-2.3.3.13.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
The big discrepancy between the local runtime and the one on the server makes me exclude the option that there are some expensive off-heap objects (e.g. byte buffers) in the application (and I'm not using any of that anyway). I know that the JVM overhead varies, but having more than 1/2 of the heap as overhead sounds too big. So what could be the reason for that? Or is it a normal way of things?
The choice of GC may impact heap size overhead, since each GC scheme must set aside some memory to manage your heap. Also, on such a small VM, you may not benefit much from going 64bit. A 32bit jvm will take up less heap, even when using CompressOOPS, which should be on by default.
So play with your favourite garbage collectors, pick the one that gives the best mix of overhead and latency for you.
Related
I've been using Java for a while now, and my typical ritual of setting up a new dev machine requires the norm of downloading and installing the latest JDK from Oracle's site.
This prompted an unusual question today, does it matter if I use the 32bit or 64bit JRE bundle?
From thinking back on it, I've installed both versions before and my normal toolchain plugs happily in (Eclipse). In my day-to-day programming, I do not recall ever having to change something or think about something in a different way just because I was using the 64bit JRE (or targetting the 64bit JRE for that respect).
From my understanding of 64bit vs. 32bit - it really boils down to how the numbers are stored underneath the covers... and I do know that int is a 32 bits and long is 64 bits... same with float being 32 bits and double is 64 bits -- so is it just that Java has abstracted even this subtlety away, and perhaps has been "64 bit compatible" all along?
I'm sure I'm missing something here besides not being able to install a 64 bit JRE onto a 32 bit system.
64-bit vs. 32-bit really boils down to the size of object references, not the size of numbers.
In 32-bit mode, references are four bytes, allowing the JVM to uniquely address 2^32 bytes of memory. This is the reason 32-bit JVMs are limited to a maximum heap size of 4GB (in reality, the limit is smaller due to other JVM and OS overhead, and differs depending on the OS).
In 64-bit mode, references are (surprise) eight bytes, allowing the JVM to uniquely address 2^64 bytes of memory, which should be enough for anybody. JVM heap sizes (specified with -Xmx) in 64-bit mode can be huge.
But 64-bit mode comes with a cost: references are double the size, increasing memory consumption. This is why Oracle introduced "Compressed oops". With compressed oops enabled (which I believe is now the default), object references are shrunk to four bytes, with the caveat that the heap is limited to four billion objects (and 32GB Xmx). Compressed oops are not free: there is a small computational cost to achieve this big reduction in memory consumption.
As a personal preference, I always run the 64-bit JVM at home. The CPU is x64 capable, the OS is too, so I like the JVM to run in 64-bit mode as well.
As you note, primitive numeric types in Java are well-defined.
However, the choice between 32-bit and 64-bit JVMs can matter if your Java application is using native-code libraries, which may be built for use in a 32-bit application, a 64-bit application, or both.
If you have native libraries that support only 32-bit applications, you either need to use a 32-bit JVM, or build 64-bit versions of the libraries.
Depending on context, for local development I will always use a 64-bit JDK. Primarily because I would likely need the whole memory space for builds and the IDE.
That being said for integration to production, I would recommend 32-bit if it is possible. Why?
For some Java EE servers that are licensed for production use, it would depend on some factors like which machine how many cores etc. For WebSphere Liberty Profile specifically, you are also limited to 2GB.
64-bit JREs would take up slightly more memory and if you're trying to constrain it to something like 2GB or better yet 2x 1GB cluster you would have more flex space to work around in without paying a cent.
From https://plumbr.eu/blog/java/should-i-use-32-or-64-bit-jvm
Problem 1: 30-50% of more heap is required on 64-bit. Why so? Mainly
because of the memory layout in 64-bit architecture. First of all –
object headers are 12 bytes on 64-bit JVM. Secondly, object references
can be either 4 bytes or 8 bytes, depending on JVM flags and the size
of the heap. This definitely adds some overhead compared to the 8
bytes on headers on 32-bit and 4 bytes on references. You can also dig
into one of our earlier posts for more information about calculating
the memory consumption of an object.
Problem 2: Longer garbage collection pauses. Building up more heap
means there is more work to be done by GC while cleaning it up from
unused objects. What it means in real life is that you have to be
extra cautious when building heaps larger than 12-16GB. Without fine
tuning and measuring you can easily introduce full GC pauses spanning
several minutes. In applications where latency is not crucial and you
can optimize for throughput only this might be OK, but on most cases
this might become a showstopper.
To limit your impact for your Java EE environment, offload parts of it to other microservices such as ElasticSearch for search, Hazelcast for caching, your database for data storage and keep your Java EE server to host your application core itself rather than running the services inside it.
I think there are two main differences to consider. One has been mentioned here but not the other.
On the one hand, as other mentioned, the memory and data types. 32-bits and 64-bits JVMs use different native data type sizes and memory-address spaces.
64-bits JVMs can allocate (can use) more memory than the 32-bits ones.
64-bits use native datatypes with more capacity but occupy more space. Because that, the same Object may occupy more space too.
For JVMs which the Garbage Collector (GC) freezes the machine, the 64-bits versions may be slower because the GC must check bigger heaps/objects and it takes more time.
There is an IBM presentation explaining these differences.
And on the other hand, the supported native libraries. Java programs that use JNI to access native libraries require different versions depending on the type of JVM.
32-bits JVMs use 32-bits native libraries and 64-bits JVMs use 64bits libraries.
That means that, if your program uses libraries that rely on native code such as SWT, you will need different versions of them. Note in the SWT download page, there are different versions for Linux/Windows 32- and 64-bits. Note that there are different versions of Eclipse (each one with a different version of SWT) for 32- and 64-bits.
Some applications, such as Alloy, are packaged with 32-bits native libraries. They fail with 64-bit JVMs. You can solve these problems just downloading the corresponding 64-bits native libraries and configuring the JNI appropriately.
Do I need to understand the difference between 32-bit JVM and 64-bit JVM?
If you aren’t building a performance critical application, you don’t have to understand the difference. The subtle difference between 32-bit JVM and 64-bit JVM wouldn’t make much difference to your application. You can skip reading further
Does 64-bit JVM perform better than 32-bit JVM?
Most of us think 64-bit is bigger than 32-bit, thus 64-bit JVM performance will be better than 32-bit JVM performance. Unfortunately, it’s not the case. 64-bit JVM can have a small performance degradation than 32-bit JVM. Below is the excerpt from Oracle JDK documentation regarding 64-bit JVM performance:
“Generally, the benefits of being able to address larger amounts of memory come with a small performance loss in 64-bit VMs versus running the same application on a 32-bit VM.
The performance difference comparing an application running on a 64-bit platform versus a 32-bit platform on SPARC is on the order of 10-20% degradation when you move to a 64-bit VM. On AMD64 and EM64T platforms this difference ranges from 0-15% depending on the amount of pointer accessing your application performs.”
Does 32 bit JVM or 64 bit JVM matter anymore.
What are the things to consider when migrating from 32-bit JVM to 64-bit JVM?
a. GC Pause times
The primary reason to migrate from 32-bit JVM to 64-bit JVM is to attain large heap size (i.e. -Xmx). When you increase heap size, automatically your GC pause times will start to go high, because now there is more garbage in the memory to clear up. You need to do proper GC tuning before doing the migration, otherwise your application can experience several seconds to few minutes pause time.
b. Native Library
If your application is using Java Native Interface (JNI) to access native libraries, then you need to upgrade the native libraries as well. Because 32-bit JVM can use only 32-bit native library. Similarly, 64-bit JVM can use only 64-bit native library.
I am running at the same computer a java game-server and a game-client
the game-client with
java -Xms512m -Xmx1024m -cp etc etc
and the game-server
java -Xmx1024M -Xms1024M -jar etc etc
Computer Properties:
Windows 7 64 bit
8GB RAM
CPU i5-2500 # 3.3GHz
Intel HD Graphics
Problem: The game-client experience serious lags. At the game-server is also connected via LAN another player with no lag issues!
Has the problem of the lag to do anything with java virtual machine? Am I using one instance of machine or two?
Can I setup something different in order to optimize the performance?
I am thinking that the problem has to do with the fact that one machine is running and its max memory is not enough for both instances, but I do not really know how to solve that.
Edit: No app run out of memory.
Solution:
1:
Updated Java version from:
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
to
java version "1.7.0_15"
Java(TM) SE Runtime Environment (build 1.7.0_15-b03)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
2:
Changed the server properties in order to minimize requirements, this seems to be the main reason.
3:
Increased memory:
game-client with java -Xms1024m -Xmx1024m -cp etc etc
and the game-server java -Xmx2048M -Xms2048M -jar etc etc
Server runs at about 700MB now.
Has the problem of the lag to do anything with java virtual machine?
Possibly. You haven't presented enough evidence to be sure one way or the other.
The puzzling thing is that the your client running on a different machine is not laggy.
Am I using one instance of machine or two?
You are running two copies of java then you will have two JVMs.
Can I setup something different in order to optimize the performance?
The answer is probably yes. But you haven't provided enough information to allow us to make solid suggestions.
Lagginess can be caused by a number of things, including:
A network with high latency.
A JVM that has a heap that is too small.
An application that is generating lots of garbage and triggering an excessive number of garbage collection.
A mix of applications that is competing for resources; e.g. physical memory, CPU time or disc or network I/O time.
If you are going to get to the root cause of your problem, you will need to do some monitoring to figure out which of the above is the likely problem. Use the task manager or whatever to check whether the system is CPU bound, short of memory, doing lots of disk or network stuff, etc. Use VisualVM to see what is going on inside the JVMs.
Alternatively, you could try to fix with some totally unscientific "knob twiddling":
try making the -Xms and -Xmx parameters the same (that may reduce lagginess at the start ...)
try increasing the sizes of the JVMs' heaps; e.g. make them 2gb instead of 1gb
try using a more recent version of Java
try using a 64 bit JVM so that you can increase the heap size further
try enabling the CMS or G1 collectors (depending on what version of JVM you are using).
If I knew more about what you were currently using, I could possibly give more concrete suggestions ...
You are using two java apps in same computer, resulting in 2 JVMs running.
For a 64 bit system with 8GB RAM, its recommended to use max 2GB(25% of Physical memory OR 75% of free physical memory up to 2 GB) for JVM for better performance.
You may have to look on JVM size adjustment. For better performance, Xms and Xmx size can be kept same with a max size bracket.
Assigning memory size to Heap is not the only area to think of. JVM uses more memory than just Heap. Others memory areas like Thread Stack, Method areas, Class Loader Subsystem, Native Method Stack etc.
While both of the apps(game-server, game-client) are running, there is a chance of issue in memory management by OS between both apps, resulting in slowness.
In that case, client app can be deployed in another core, if available.
we have a 32 bit JVM running under 64 bit RHEL5 on a box which has plenty of memory (32G). For different reasons, this process requires a pretty large managed heap and permgen space -- currently, it runs with the following VM arguments:
-Xmx2200M -XX:MaxPermSize=128M -XX:+CMSClassUnloadingEnabled
I have started seeing JVM crashes recently because it - seemingly - ran out of native memory (it could not create native threads, or failed to allocate native memory, etc.). These crashes were not (directly) related to the state of the managed heap, as when those crashes happened the managed heap was ~50-70% full.
I know that the memory reserved for the managed process is close to 2.5 G which leaves not more than 0.5G for the JVM itself, BUT
- I don't understand why 0.5 isn't enough for the JVM, even if there is constant GCing going on
- the real question is this: when I connect to the process using jconsole, then it says that (currently)
Committed virtual memory:
3,211,180 kbytes
Which is more than 3G. I can imagine that for some reason JVM thinks that it has
3,211,180 kbytes (3.06G) of memory is but when it tries to go over 3G the memory allocation fails.
Any ideas on
a) why does this happen
b) how is it possible to avoid this
Thanks.
Mate
There is a lot of overhead in a typical VM that is not counted in the VM accounting because it is essentially stolen by the native elements of the process - e.g. mapping in .so files that are used for performing native level code for system libraries are not counted in the base VM accounting. your typical shared library is mapped in at the top GB of memory, so if you try to allocate memory into this region you will be denied, because it would overrun with the shared libraries' memory region - memory allocation on most OS's is performed by a simple bar that is raised when you ask for more memory. When you ask for memory and the bar conflicts with other uses, then it simply fails. Most of the details that follow are about this.
You need to avoid needing so much memory in a 32bit process. This is the fundamental challenge. It is trivial to get a 64bit VM that will allow you to make use of so much more memory than would be otherwise accessible - it is just simply usable in this situation.
If you are using a 32bit process, there is a high probability that you are encountering the effective address space limit of the 32bit process. For windows, this is a maximum of about 3GB - anything above this is reserved for I/O space and the kernel. You can move this, but it has a tendency to break applications/drivers that are designed for the 32bit OS.
For Linux, you end up with ~3GB of usable addressable RAM per process, the rest is used up by things like the kernel and mapped in shared libraries. The limit is referred to as the 'address space limit', and I presume it can be tuned.
How to avoid it? Well, for the most part, you can't, it's a physical limitation of the 32bit address space and the needs of having the kernel/IO in the same address space as the process for a 32bit OS.
With 64 bit OS's you have (most of) all of the 64 bit address space to play around with, which is extensively more than you need to use.
When you start a JVM it allocates it maximum size immediately. How much of that memory is used doesn't really matter. Your application can address about 3 GB of which about 2.3 GB you have allocated to heap and perm gen. The rest is available for shared libraries (typically around 200 MB) and thread stacks.
Worrying about why you can't use the full 3 GB of address isn't very useful when the solution is relatively trivial (use a 64-bit JVM) I am assuming you don't have any shared libraries which are only available in 32-bit. However if you do have additional shared libraries they can easily be using 100s of MB.
I'd like to run a very simple bot written in java on my VPS.
I want to limit jvm memory to let's say 10MB (I doubt it would need any more).
I'm running the bot with the following command:
java -Xms5M -Xmx10M -server -jar
IrcBot.jar "/home/jbot"
But top shows that actual memory reserved for java is 144m (or am I interpreting things wrong here?).
13614 jbot 17 0 144m 16m 6740
S 0.0 3.2 0:00.20 java
Any ideas what can be wrong here?
Java version "1.6.0_20" Java(TM) SE Runtime Environment (build 1.6.0_20-b02) Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode)
BTW. I'm running CentOS - if it matters.
EDIT:
Thank you for your answers.
I can't really accept any of them, since it turns out the problem lies within the language i choose to write the program, not the JVM itself.
-Xmx specifies the max Java heap allocation (-Xms specifies the min heap allocation). The Java process has its own overhead (the actual JVM etc), plus the loaded classes and the perm gen space (set via -XX:MaxPermSize=128m) sits outside of that value too.
Think of your heap allocation as simply Java's "internal working space", not the process as a whole.
Try experimenting and you'll see what I mean:
java -Xms512m -Xmx1024m ...
Also, try using a tool such as JConsole or JVisualVM (both are shipped with the Sun / Oracle JDK) and you'll be able to see graphical representations of the actual heap usage (and the settings you used to constrain the size).
Finally, as #Peter Lawrey very rightly states, the resident memory is the crucial figure here - in your case the JVM is only using 16 MiB RSS (according to 'top'). The shared / virtual allocation won't cause any issues as long as the JVM's heap isn't pushed into swap (non-RAM). Again, as I've stated in some of the comments, there are other JVM's available - "Java" is quite capable of running on low resource or embedded platforms.
Xmx is the max heap size, but besides that there's a few other things that the JVM needs to keep in memory: the stack, the classes, etc. For a brief introduction see this post about the JVM memory structure.
The JVM maps in shared libraries which are about 150m. The amount of virtual memory used is unlikely to be important to use if you are trying to minimise physical main memory.
The number you want to look at is the resident memory which is amount of physical main memory actually used (which is 16 MB)
We run the 32-bit Sun Java 5 JVM on 64-bit Linux 2.6 servers, but apparently this limits our maximum memory per process to 2GB. So it's been proposed that we upgrade to the 64-bit JVM's to remove the restriction. We currently run multiple JVM's (Tomcat instances) on a server in order to stay under the 2GB limit, but we'd like to consolidate them in the interest of simplifying deployment.
If you've done this, can you share your experiences, please? Are you running 64-bit JVM's in production? Would you recommend staying at Java 5, or would it be ok to move to both Java 6 and 64 bits simultaneously? Should we expect performance issues, either better or worse? Are there any particular areas we should focus our regression testing?
Thanks for any tips!
At the Kepler Science Operations Center we have about 50 machines with 32-64G each. The JVMs heaps are typically 7-20G. We are using Java 6. The OS has Linux 2.6 kernel.
When we migrated to 64bit I expected there would be some issues with running the 64-bit JVM But really there have not been. Out of memory conditions are more difficult to debug since the heap dumps are so much larger. The Java Service Wrapper needed some modifications to support larger heap sizes.
There are some sites on the web claiming GC does not scale well past 2G, but I've not seen any problems. Finally, we are doing throughput intensive rather interactive intensive computing. I've never looked at latency differences; my guess is worst case GC latency will be longer with the larger heap sizes.
We use a 64-bit JVM with heaps of around 40 Gb. In our application, a lot of data is cached, resulting in a large "old" generation. The default garbage collection settings did not work well and needed some painful tuning in production. Lesson: make sure that you have adequate load-testing infrastructure before you scale up like this. That said, once we got the kinks worked out, GC performance has been great.
I can confirm Sean's experience. We are running pure-Java, computationally intensive web services (home-cooked Jetty integration, with nowadays more than 1k servlet threads, and >6Gb of loaded data in memory), and all our applications scaled very well to a 64 bit JVM when we migrated 2 years ago. I would advise to use the latest Sun JVM, as substantial improvement in the GC overhead have been done in the last few releases. I did not have any issue with Tanukisoftware's Wrapper either.
Any JNI code you have written that assumes it's running in 32 bits will need to be retested. For problems you may run into porting c code from 32 to 64 bits see this link. It's not JNI specific but still applys. http://www.ibm.com/developerworks/library/l-port64.html
After migrating to JDK6 64bits from JDK5 32bits (Windows server), we got leak in "perm gen space" memory block. After playing with JDK parameters it was resolved. Hope you will be more lucky then we are.
If you use numactl --show you can see the size of the memory banks in your server.
I have found the GC doesn't scale well when it uses more than one memory bank. This is more a hardware than a software issue IMHO but it can effect your GC times all the same.