Develop Reference

Memory running out side of Xmx and Xms

I have run into an issue with a java application I wrote causing hardware performance issues. The problem (I'm fairly certain), is that a few of the machines that I'm running the application on only have 1GB of memory. When I start my java application, I"m setting the heap size to -Xms 512m -Xmx 1024m.
My first question, is my assumption correct that this will obviously cause performance problems because I'm allocating all of the machines memory to the java heap?
This leads to another question. I'm running jconsole on the app and monitoring the apps memory usage. What I'm seeing is that the app consumes about 30mb at startup, gets to about 150mb and the garbage collector runs and it goes back down to 30mb. What I'm also seeing using top on the pid is that the application starts by using about 6% memory then slowly climbs up to about 20%. I do not understand this. Why would it only get up to 20% memory usage when I'm allocating 1GB to it. Shouldn't it go to 100%. Also, why is it using that much memory (20%) when it doesn't appear that the app ever uses more than 150mb?
I think its pretty obvious I need to adjust my Xms and Xmx and that should resolve the issue, but I'm trying to understand better what exactly is happening.

Two possibilities for the memory use:
Your app just does not use that much memory
Or
Your app does not use that much memory fast enough.
What happens:
The garbage collector has several points where it will execute:
Just scheduled: It will clean up easy to remove objects
Full collection: This runs when you hit the set memory limits.
If options 1, the general much lower impact quick collection, can keep your memory use under control, it will not hit the full collection unless it the JVM GC options are set to run a full on a schedule.
With your application I would start setting lower xmx/xms values so that more guaranteed resources are left for the OS, and maybe some paging is prevented.

JAVAEE - Used heap memory just continue increasing until everything breaks

I am running a medium sized jersey web app under tomcat. I find out that the app keeps crashing after certain amount of time of execution (couple of days) due to out of memory problem, I already increased the heap size but that is not the problem as I am facing a memory leak somewhere.
I looked for ways to debug this to no avail. I m using a tool called YouKit Java to help me into this and I realised that the used heap memory keeps growing indefinitly until it breaks. The garbage collection doesnt seem to run at any moment.
heap memory usage afer 16h
I have run a debug app for the entire night and even with minimal to no use this happens: used heap grows from a couple of MB to XXXMB (> 1GB in prod) with no load on it at all. After forcing garbage collection, memory usage goes back to normal.
On the left the sudden decrease after I force GC. On the right, the memory couple of minutes after GC
The next picture shows the used memory growing again after my forced GC with basic usage of the app: page reload, some get queries that returns data from db (sqlite) and some post query that write into db and open some sockets. Note that for everything that I tested, I also run the opposite command that should cancel my changes. but the memory just keeps increasing.
I did a memory snapshot to browse through what is instanciated. Biggest object- dominators shows huge arborescence of object java.lang.ref.Finalizer, knowing that I dont ever call any finalize method (not that I know of at least)
So I am very lost in this, Java is not my biggest strength and I am having lot of hard times debugging this. I am wondering if there is something possibily preventing GC from running and causing this ?
(As I see that after force execution of GC things fall back to a more normal state) Can this be caused by TOMCAT or jersey itself ?
Side notes about the app : It is an API that lets you create tcp tunnel in background (server and client sockets). Every tunnel is spawned in a thread. It also does some data fetching and writing to an sqlite db. I tried to be sure that everything is closed properly (db connections, queries, sockets and is unreference...) when the work is done. For the tunnels, I am relying on a lightly edited version of a library called javatunnel (it may be also bew the culprit but couldn't find anything proving it)

The socket are perhaps not correctly closed so that the garbage collector cannot freed them and the related objects ? I think you should try to investigate deeper in this way.
You can activate garbage collector logs to detect when it runs. You have to add flags -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps at the JVM start-up.

I realised that the used heap memory keeps growing indefinitly until
it breaks. The garbage collection doesnt seem to run at any moment.
In this case you should take a look at your GC parameters in JVM startup arguments.
What is the ratio of your max old size and max new size. If max old size is too high as compared to max new size, your objects will keep on moving from survivor to old generation and wont be garbage collect until old gen size reaches a threshold which is i guess > 75% by default.
So, check you GC tuning. It might help.

Java memory increaseases in Task Manager but not in profiler

We developed an highly CPU intensive Java server application that is having a serious memory leak (or so it seems). As time passes, the application seems to eat up increasingly more memory (as seen with Windows Task Manager) but if I analyse it a specialized Java profiler the memory seems to be staying the same. For example, in task manager I see the application taking over 8gb of memory and growing, but in the Java profiler I see that heap memory is at most 2gb. I tried all possible combinations of JAVA_OPTS (-Xmx, -Xms, all types of GC) and nothing worked, Is the Java process not releasing memory back to OS? Is there any way to force it to do so?

1)
I suggest you to set -Xmx2100m and observe heap usage under load.
JVM may take as much OS memory as it decide to be performant, until it reaches Xmx limit. In modern JVMs default Xmx is calculated upon total memory available in OS, so it may be large value.
I think your app does not have memory leak, your JVM simply allocate a lot of memory, because it can.
Observe your JVM thru jvisualvm.
2)
Second suggestion - do you use any JNI code? Does your app call any native library (ie. dll under windows)?

java memory performance - strange behaviour (or is this normal?)

Hi there
I would like to start by saying that i'm a beginner, but i'm working on a really small and simple Java app, that really shouldn't cause some major problems.
I was monitoring memory usage from windows task manager, and noticed that with my application started, java.exe was using about 70MB of available memory. So I thought to myself, ok, this probably is a little large, but still, nothing that my PC couldn't handle. But really strange thing started happening when i tried to resize my window, memory usage suddenly jumped to like 80-90 MB, and if i would continue dragging my window, randomly resizing, it kept increasing memory usage. I thought it has something to do with calling repainting methods on GUI components during window resize, so i took a few suspicious components that could cause some kind of memory leak, and deleted those from my mainwindow form, leaving my program almost completely stripped down, but this issue persisted. What i noticed later was that if i keep resizing my window, memory usage grows up to 200-220 MB, and then stops this uncontrolled growth there.
So can somebody tell me, could this be a normal behavior having in mind memory management in java?

Java objects created are not necessarily cleaned up once they're finished with. Instead, something called the "garbage collector" periodically runs in the background looking for orphaned objects and deletes them, freeing up memory.
Your application is likely creating lots of temporary objects as it resizes your window. Although no longer being referenced by anything (ie orphans), these objects are hanging around until the garbage collector runs.
You'll probably find that your max memory is 256M (the default) - the garbage collector is probably being called more often as you approach your max memory and the creation of new objects requires memory to be freed up immediately - hence the memory hovering just under 256M as the creation/deletion rate is balanced by demand.
This is completely normal behaviour.

No, this behaviour is perfectly normal. Java memory management is based on automatic garbage collection, which means that unused memory accumulates for a while before being garbage collected (because that is a significant amount of work, you want to do it as rarely as possible.
So the JVM will tend to use a large part of the memory it's allowed to use (the maximum heap size) - and on a modern PC with multiple GBs of memory available, the default maximum heap size will be pretty big. However, if you have a small app that you know won't need much memory, you can adjust the maximum heap size via the command line option -Xmx, for example
java -Xmx64M main.class.name
will restrict the heap to 64MB

Which heap size do you prefer?

I know there is no "right" heap size, but which heap size do you use in your applications (application type, jdk, os)?
The JVM Options -Xms (initial/minimum) and -Xmx (maximum) allow for controlling the heap size. What settings make sense under which circumstances? When are the defaults appropriate?

You have to try your application and see how it performs. for example, I used to always run IDEA out of the box until I've got this new job where I work on this huge monolithic project. IDEA was running very slow and regularly throwing out of memory errors when compiling the full project.
first thing I did is to ramp up the heap to 1 gig. this got rid of the out of memory issues but it was still slow. I also noticed IDEA was regularly freezing for 10 seconds or so after which the used memory was cut in half only to ramp up again and , and that triggered the garbage collection idea. I now use it with -Xms512m, -Xmx768m but, I also added -Xincgc, to activate incremental garbage collection
As a result, I've got my old IDEA back: it runs smooth, doesn't freeze anymore and never uses more than 600m of heap.
For your application you have to use a similar approach. try to determine the typical memory usage and tune your heap for the application to run well in those conditions. But also let advanced users tune the setting, to address out of the ordinary data loads.

It depends on the application type. A desktop application is much different than a web application. An application server is much different than a standalone application.
It also depends on the JVM that you are using. JDK5 and later 6 include enhancements that help understand how to tune your application.
Heap size is important, but its also important to know how it plays with the garbage collector.
JDK1.4 Garbage Collector Tuning
JDK5 Garbage Collector Tuning
JDK6 Garbage Collector Tuning

Actually I always considered it very strange that Java limits the heap size. A native application can usually use as much heap as it wants, until it runs out of virtual address space. The only reason to limit the heap in Java seems the garbage collector, which has a certain kind of "laziness" and may not garbage collect objects, unless there is a necessity to do so. That means if you choose the heap too big, your app constantly uses more memory than is really necessary.
However, Sun has improved the GC a lot over the years and to emulate the behavior of a native C app, I would set the initial heap size to 32 MB (for small programs) or 64 MB (for bigger ones) and the maximum to something between 1-2 GB. If your app really needs over a 1 GB of memory, it is most likely broken (unless you deal with data objects that large), but I see no reason why your app should be killed, just because it goes over a certain heap size.
Of course, this is referring to normal PCs. If you create Java code for mobile phones or other limited devices, you should probably adopt the initial and maximum heap size to the limitations of that device.

Typically i try not to use heaps which are larger than 1GB.
It will cost you on major garbage collections.
Sometime it is better to split your application to a few JVM on the same machine and not you large heap sizes.
Major collection with a large heap size can take >10 mintues (on unoptimized GC applications).

This is entirely dependent on your application and any hardware limitations you may have. There is no one size fits all.
jmap can be used to have a look at what heap you are actually using and is a good starting point for right-sizing the heap.

You need to spend quite some time in JConsole or visualvm to get a clear picture on what the plateau memory usage is. Wait until everything is stable and you see the characteristic sawtooth curve of heap memory usage. The peaks should be your 70-80% heap, depending on what garbage collector you use.
Most garbage collectors trigger full GCs when heap usage reaches a certain percentage. This percentage is from 60% to 80% of max heap, depending on what strategy is involved.

1.3Gb for a heavy GUI application.
Unfortunately on Linux the JVM seems to pre-request 1.3G of virtual memory in that situation, which looks bad even if it's not needed (and causes a lot of confused grumbling from users)

On my most memory intensive app:
-Xms250M -Xmx1500M -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC