Develop Reference

Reducing Java heap size

I have an application that uses a lot of memory diff'ing the contents of two potentially huge (100k+) directories. It makes sense to me that such an operation would use a lot of memory, but once my diff'ing operation is done, the heap remains the same size.
I basically have code that instantiates a class to store the filename, file size, path, and modification date for each file on the source and target. I save the additions, deletions, and updates in other arrays. I then clear() my source and target arrays (which could be 100k+ each by now), leaving relatively small additions, deletions, and updates arrays left.
After I clear() my target and source arrays though, the memory usage (as visible via VirtualVM and Windows Task Manager) doesn't drop. I'm not experienced enough with VirtualVM (or any profiler for that matter) to figure out what is taking up all this memory. VirtualVM's heap dump lists the top few objects with a retained size of a few megabytes.
Anything to help point me in the right direction?

If the used heap goes down after a Garbage Collection, than it likely works as expected. Java increases its heap when it needs more memory, but does not free it -- it prefers to keep it in case the application uses more memory again. See Is there a way to lower Java heap when not in use? for this topic on why the heap is not reduced after the used heap amount lowers.

The VM grows or shrinks the heap based on the command-line parameters -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio. It will shrink the heap when the free percentage hits -XX:MaxHeapFreeRatio, whose default is 70.
There is a short discussion of this in Oracle's bug #6498735.

Depending on your code you might be generating memory leaks and the Garbage collector just can't free them up.
I would suggest to instrument your code in order to find potential memory leaks. Once this is ruled out or fixed, I would start to look at the code itself for possible improvement.
Note that for instance if you use the try/catch/finally block. The finally block might not be called at all (or at least not immediately). If you do some resource freeing in a finally block this might be the answer.
Nevertheless read up on the subject, for instance here: http://www.toptal.com/java/hunting-memory-leaks-in-java

Setting a smaller JVM heap size within a JNI client application

I'm attempting to debug a problem with pl/java, a procedural language for PostgreSQL. I'm running this stack on a Linux server.
Essentially, each Postgres backend (connection process) must start its own JVM, and does so using the JNI. This is generally a major limitation of pl/java, but it has one particularly nasty manifestation.
If native memory runs out (I realise that this may not actually be due to malloc() returning NULL, but the effect is about the same), this failure is handled rather poorly. It results in an OutOfMemoryError due to "native memory exhaustion". This results in a segfault of the Postgres backend, originating from within libjvm.so, and a javacore file that says something like:
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received
1TIDATETIME Date: 2012/09/13 at 16:36:01
1TIFILENAME Javacore filename: /var/lib/PostgreSQL/9.1/data/javacore.20120913.104611.24742.0002.txt
***SNIP***
Now, there are reasonably well-defined ways of ameliorating these types of problems with Java, described here:
http://www.ibm.com/developerworks/java/library/j-nativememory-linux/
I think that it would be particularly effective if I could set the maximum heap size to a value that is far lower than the default. Ordinarily, it is possible to do something along these lines:
The heap's size is controlled from the Java command line using the -Xmx and -Xms options (mx is the maximum size of the heap, ms is the initial size). Although the logical heap (the area of memory that is actively used) can grow and shrink according to the number of objects on the heap and the amount of time spent in GC, the amount of native memory used remains constant and is dictated by the -Xmx value: the maximum heap size. Most GC algorithms rely on the heap being allocated as a contiguous slab of memory, so it's impossible to allocate more native memory when the heap needs to expand. All heap memory must be reserved up front.
However, it is not apparent how I can follow these steps such that pl/java's JNI initialisation initialises a JVM with a smaller heap; I can't very well pass these command line arguments to Postgres. So, my question is, how can I set the maximum heap size or otherwise control these problems in this context specifically? This appears to be a general problem with pl/java, so I expect to be able to share whatever solution I eventually arrive at with the Postgres community.
Please note that I am not experienced with JVM internals, and am not generally familiar with Java.
Thanks

According to slide 19 in this presentation postgresql.conf can have the parameter pljava.vmoptions where you can pass arguments to the JVM.

Java String objects not getting garbage collected on time

I have an interesting problem with Java memory consumption. I have a native C++ application which invokes my Java application.
The Application basically does some language translations\parses a few XML's and responds to network requests. Most of the state of Application doesn't have to be retained so it is full of Methods which take in String arguments and returns string results.
This application continues to take more and more memory with time and there comes a time where it starts to take close to 2 GB memory, which made us suspect that there is a leak somewhere in some Hashtable or static variables. On closer inspection we did not find any leaks. Comparing heap dumps over a period of time, shows the char[] and String objects take huge memory.
However when we inspect these char[], Strings we find that they do not have GC roots which means that they shouldn't be the cause of leak. Since they are a part of heap, it means they are waiting to get garbage collected. After using verious tools MAT\VisualVM\JHat and scrolling through a lot of such objects I used the trial version of yourkit. Yourkit gives the data straightaway saying that 96% of the char[] and String are unreachable. Which means that at the time of taking dump 96% of the Strings in the heap were waiting to get garbage collected.
I understand that the GC runs sparingly but when you check via VisualVM you can actually see it running :-( than how come there are so many unused objects on the heap all time.
IMO this Application should never take more than 400-500 MB memory, which is where it stays for the first 24 hours but than it continues to increase the heap :-(
I am running Java 1.6.0-25.
thanks for any help.

Java doesn't GC when you think it does/should :-) GC is too complex a topic to understand what is going on without spending a couple of weeks really digging into the details. So if you see behavior that you can't explain, that doesn't mean its broken.
What you see can have several reasons:
You are loading a huge String into memory and keep a reference to a substring. That can keep the whole string in memory (Java doesn't always allocate a new char array for substrings - since Strings are immutable, it simply reuses the original char array and remembers the offset and length).
Nothing triggered the GC so far. Some C++ developers believe GC is "evil" (anything that you don't understand must be evil, right?) so they configure Java not to run it unless absolutely necessary. This means the VM will eat memory until it hits the maximum and then, it will do one huge GC run.
build 25 is already pretty old. Try to update to the latest Java build (33, I think). The GC is one of the best tested parts of the VM but it does have bugs. Maybe you hit one.
Unless you see OutOfMemoryException, you don't have a leak. We have an application which eats all the heap you give it. If it gets 16GB of RAM ("just to be safe"), it will use the whole 16GB because we cache what we can. You never see out of memory, because the cache will shrink as needed but system admins routinely freak out "oh god! oh god! It's running out of memory" PANIK No, it's not. Unless Java tells you so, it's not running out of memory. It's just using it efficiently.
Tuning the GC with command line options is one of the best ways to break it. Hundreds of people which know a lot more about the topic than you ever will spent years making the GC efficient. You think you can do better? Good luck. -> Get rid of any "magic" command line options and calls to System.gc() and your problem might go away.

Try decreasing the heap size to 500 Megabytes and see if the software will start garbage collecting or die. Java isnt too fussy about using memory given to it. you might also research GC tuning options which will make the GC more prudent about cleaning stuff up.

String reallyLongString = "this is a really long String";
String tinyString = reallyLongString.substring(2, 3);
reallyLongString = null
The JVM can't collect the memory allocated for the long string in the above case, since there's a reference to part of it.
If you're doing stuff with Strings and you're suffering from memory issues, this might be the cause of your grief.
use tinyString = new String(reallyLongString.substring(2, 3); instead.

There might not be a leak at all - a leak would be if the Strings were reachable. If you've allocated as much as 2GB to the application, there is no reason for the garbage collector to start freeing up memory until you are approaching that limit. If you don't want it taking any more than 500MB, then pass -Xmx 512m when starting the JVM.
You could also try tuning the garbage collector to start cleaning up much earlier.

First of all, stop worrying about those Strings and char[]. In almost every java application I have profiled, they are on the top of memory consumer list. And in almost no of those java application they were the real problem.
If you have not received OutOfMemoryError yet, but do worry that 2GB is too much for your java process, then try to decrease Xmx value you pass to it. If it runs well and good with 512m or 1g, then problem solved, isn't it?
If you get OOM, then one more option you can try is to use Plumbr with your java process. It is memory leak discovery tool, to it can help you if there really is a memory leak.

Simple Class - Is it a Memory Leak?

I've a very simple class which has one integer variable. I just print the value of variable 'i' to the screen and increment it, and make the thread sleep for 1 second. When I run a profiler against this method, the memory usage increases slowly even though I'm not creating any new variables. After executing this code for around 16 hours, I see that the memory usage had increased to 4 MB (initially 1 MB when I started the program). I'm a novice in Java. Could any one please help explain where am I going wrong, or why the memory usage is gradually increasing even when there are no new variables created? Thanks in advance.
I'm using netbeans 7.1 and its profiler to view the memory usage.
public static void main(String[] args)
{
try
{
int i = 1;
while(true)
{
System.out.println(i);
i++;
Thread.sleep(1000);
}
}
catch(InterruptedException ex)
{
System.out.print(ex.toString());
}
}
Initial memory usage when the program started : 1569852 Bytes.
Memory usage after executing the loop for 16 hours : 4095829 Bytes

It is not necessarily a memory leak. When the GC runs, the objects that are allocated (I presume) in the System.out.println(i); statement will be collected. A memory leak in Java is when memory fills up with useless objects that can't be reclaimed by the GC.
The println(i) is using Integer.toString(int) to convert the int to a String, and that is allocating a new String each time. That is not a leak, because the String will become unreachable and a candidate for GC'ing once it has been copied to the output buffer.
Other possible sources of memory allocation:
Thread.sleep could be allocating objects under the covers.
Some private JVM thread could be causing this.
The "java agent" code that the profiler is using to monitor the JVM state could be causing this. It has to assemble and send data over a socket to the profiler application, and that could well involve allocating Java objects. It may also be accumulating stuff in the JVM's heap or non-heap memory.
But it doesn't really matter so long as the space can be reclaimed if / when the GC runs. If it can't, then you may have found a JVM bug or a bug in the profiler that you are using. (Try replacing the loop with one very long sleep and see if the "leak" is still there.) And it probably doesn't matter if this is a slow leak caused by profiling ... because you don't normally run production code with profiling enabled for that long.
Note: calling System.gc() is not guaranteed to cause the GC to run. Read the javadoc.

I don't see any memory leak in this code. You should see how Garbage collector in Java works and at its strategies. Very basically speaking GC won't clean up until it is needed - as indicated in particular strategy.
You can also try to call System.gc().
The objects are created probably in the two Java Core functions.

It's due to the text displayed in the console, and the size of the integer (a little bit).
Java print functions use 8-bit ASCII, therefor 56000 prints of a number, at 8 bytes each char will soon rack up memory.

Follow this tutorial to find your memory leak: Analyzing Memory Leak in Java Applications using VisualVM. You have to make a snapshot of your application at the start and another one after some time. With VisualVM you can do this and compare these to snapshots.

Try setting the JVM upper memory limit so low that the possible leak will cause it to run out of memory.
If the used memory hits that limit and continues to work away happily then garbage collection is doing its job.
If instead it bombs, then you have a real problem...

This does not seem to be leak as the graphs of the profiler also tell. The graph drops sharply after certain intervals i.e. when GC is performed. It would have been a leak had the graph kept climbing steadily. The heap space remaining after that must be used by the thread.sleep() and also (as mentioned in one of answers above) from the some code of the profiler.
You can try running VisualVM located at %JAVA_HOME%/bin and analyzing your application therein. It also gives you the option of performing GC at will and many more options.
I noted that the more features of VisualVM I used more memory was being consumed (upto 10MB). So this increase, it has to be from your profiler as well but it still is not a leak as space is reclaimed on GC.

Does this occur without the printlns? In other words, perhaps keeping the printlns displayed on the console is what is consuming the memory.

java.lang.OutOfMemoryError: Java heap space

I am getting the following error on execution of a multi-threading program
java.lang.OutOfMemoryError: Java heap space
The above error occured in one of the threads.
Upto my knowledge, Heap space is occupied by instance variables only. If this is correct, then why this error occurred after running fine for sometime as space for instance variables are alloted at the time of object creation.
Is there any way to increase the heap space?
What changes should I made to my program so that It will grab less heap space?

If you want to increase your heap space, you can use java -Xms<initial heap size> -Xmx<maximum heap size> on the command line. By default, the values are based on the JRE version and system configuration. You can find out more about the VM options on the Java website.
However, I would recommend profiling your application to find out why your heap size is being eaten. NetBeans has a very good profiler included with it. I believe it uses the jvisualvm under the hood. With a profiler, you can try to find where many objects are being created, when objects get garbage collected, and more.

1.- Yes, but it pretty much refers to the whole memory used by your program.
2.- Yes see Java VM options
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
Ie
java -Xmx2g assign 2 gigabytes of ram as maximum to your app
But you should see if you don't have a memory leak first.
3.- It depends on the program. Try spot memory leaks. This question would be to hard to answer. Lately you can profile using JConsole to try to find out where your memory is going to

You may want to look at this site to learn more about memory in the JVM:
http://developer.streamezzo.com/content/learn/articles/optimization-heap-memory-usage
I have found it useful to use visualgc to watch how the different parts of the memory model is filling up, to determine what to change.
It is difficult to determine which part of memory was filled up, hence visualgc, as you may want to just change the part that is having a problem, rather than just say,
Fine! I will give 1G of RAM to the JVM.
Try to be more precise about what you are doing, in the long run you will probably find the program better for it.
To determine where the memory leak may be you can use unit tests for that, by testing what was the memory before the test, and after, and if there is too big a change then you may want to examine it, but, you need to do the check while your test is still running.

You can get your heap memory size through below programe.
public class GetHeapSize {
public static void main(String[] args) {
long heapsize = Runtime.getRuntime().totalMemory();
System.out.println("heapsize is :: " + heapsize);
}
}
then accordingly you can increase heap size also by using:
java -Xmx2g
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

To increase the heap size you can use the -Xmx argument when starting Java; e.g.
-Xmx256M

Upto my knowledge, Heap space is occupied by instance variables only. If this is correct, then why this error occurred after running fine for sometime as space for instance variables are alloted at the time of object creation.
That means you are creating more objects in your application over a period of time continuously. New objects will be stored in heap memory and that's the reason for growth in heap memory.
Heap not only contains instance variables. It will store all non-primitive data types ( Objects). These objects life time may be short (method block) or long (till the object is referenced in your application)
Is there any way to increase the heap space?
Yes. Have a look at this oracle article for more details.
There are two parameters for setting the heap size:
-Xms:, which sets the initial and minimum heap size
-Xmx:, which sets the maximum heap size
What changes should I made to my program so that It will grab less heap space?
It depends on your application.
Set the maximum heap memory as per your application requirement
Don't cause memory leaks in your application
If you find memory leaks in your application, find the root cause with help of profiling tools like MAT, Visual VM , jconsole etc. Once you find the root cause, fix the leaks.
Important notes from oracle article
Cause: The detail message Java heap space indicates object could not be allocated in the Java heap. This error does not necessarily imply a memory leak.
Possible reasons:
Improper configuration ( not allocating sufficiant memory)
Application is unintentionally holding references to objects and this prevents the objects from being garbage collected
Applications that make excessive use of finalizers. If a class has a finalize method, then objects of that type do not have their space reclaimed at garbage collection time. If the finalizer thread cannot keep up, with the finalization queue, then the Java heap could fill up and this type of OutOfMemoryError exception would be thrown.
On a different note, use better Garbage collection algorithms ( CMS or G1GC)
Have a look at this question for understanding G1GC

In most of the cases, the code is not optimized. Release those objects which you think shall not be needed further. Avoid creation of objects in your loop each time. Try to use caches. I don't know how your application is doing. But In programming, one rule of normal life applies as well
Prevention is better than cure. "Don't create unnecessary objects"

Local variables are located on the stack. Heap space is occupied by objects.
You can use the -Xmx option.
Basically heap space is used up everytime you allocate a new object with new and freed some time after the object is no longer referenced. So make sure that you don't keep references to objects that you no longer need.

No, I think you are thinking of stack space. Heap space is occupied by objects. The way to increase it is -Xmx256m, replacing the 256 with the amount you need on the command line.

To avoid that exception, if you are using JUnit and Spring try adding this in every test class:
#DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_CLASS)

I have tried all Solutions but nothing worked from above solutions
Solution: In My case I was using 4GB RAM and due to that RAM usage comes out 98% so the required amount if Memory wasn't available. Please do look for this also.If such issue comes upgrade RAM and it will work fine.
Hope this will save someone Time

In netbeans, Go to 'Run' toolbar, --> 'Set Project Configuration' --> 'Customise' --> 'run' of its popped up windo --> 'VM Option' --> fill in '-Xms2048m -Xmx2048m'. It could solve heap size problem.