When I get OOM error, how do I decide weather I should increase heap size or there is memory leak problem with my code?
Also, how do I decide with inital heap size of my application? In my current application, we had started with 512MB but now have increased to 4GB. But this was done by trial and error method. Is there any systematic way to decide required heap size?
If my question is sounding too basic, can anyone please share some references which can help to increase understanding of this?
I don't think Java or the JVM actually define "leak". But your programs are clearly affected by them. I'm sure they define "out of memory".
You have a memory leak, if there is an object that will never be examined again by the application, held by a reference in a scope that is never exited, or held by another object that is examined occasionally. Such a leak may not seriously impact the long term stability of your program.
You have a bad leak if you generate a new object with these properties when some bit of code in the application is run, and that bit of code may run an indefinite number of times. Eventually a program with a bad leak runs out of memory.
It seems to me that 'leak' is a poor term in the GC context. Leaks are exactly what will be garbage-collected.
What won't be GC'd, and what causes OutOfMemoryErrors, is memory that isn't leaked when it should be, i.e. references to objects held beyond their true useful lifetime, typically a reference that is a member variable that should be method-local.
Related
My question is very very clear .. if it didn't collect garbage what will happen
And why that thing does not happen in c++
If stale objects are not collected, you will run into an OutOfMemoryError. And this will also happen in C++, if you don't clean up old data (the error is probably not called OutOfMemoryError, but the consequences are the same i guess).
It does happen in C/C++. In a different way. In C/C++ the responsibility lies more with the programmer to keep track of memory allocated and to free it. Programmers have direct access to memory in the form of pointers. In java it is all hidden from programmer. You don't need to keep track of memory, JVM does it for you in the form of garbage collection. That does not mean you can allocate memory to objects wily-nily. Global objects are retained for long time and if GC cannot reach them in time, the memory goes beyond limit and OutOfMemoryException happens.
I am working on an application & my code is having out of memory error. I am not able to see memory utilisation of the code.so I am very confused were to see.
Also after my little analysis I came to kow that there is private static object getting creating & in the constructor of that class. some more objects are getting created. & that class is multithreaded..
so I want to know since the static objects does not get garbage collected.. will all the objects related to the constructor will not be garbage collected??
A static reference is only collected when the class is unloaded, and this only happened when the class loader is not used any more. If you haven't got multiple class loaders it is likely this will never be unloaded (until your program stops)
However, just because an object was once referenced statically doesn't change how it is collected. If you had a static reference to an object and no longer have a reference to that object, it will be collected as normal.
Having multiple threads can make finding bugs harder, but it doesn't change how objects are collected either.
You need to take a memory dump of your application and see why memory is building up. It is possible the objects you retaining are all needed. In this case you need to
reduce your memory requirement
increase your maximum memory.
You might not have a memory leak - you might simply surpassed the amount of avaialble RAM your system can provide.
you can add several JVM arguments to limit the size of RAM allocated to your runtime enviorment, and control the garbage collector - the tradeoff is it usually consumes more CPU.
You say you are not capable of seeing the memory utilisation?
Have you tried using JVisualVM (in $JAVA_HOME/bin/jvisualvm)
It should be capable of attaching to local processes and take heap dumps.
Also, Eclipse Memory Analyzer has some good reports for subsequent analysis
I have a memory leak in Java in which I have 9600 ImapClients in my heap dump and only 7800 MonitoringTasks. This is a problem since every ImapClient should be owned by a MonitoringTask, so those extra 1800 ImapClients are leaked.
One problem is I can't isolate them in the heap dump and see what's keeping them alive. So far I've only been able to pinpoint them by using external evidence to guess at which ImapClients are dangling. I'm learning OQL which I believe can solve this but it's coming slowly, and it'll take a while before I can understand how to perform something recursive like this in a new query language.
Determining a leak exists is difficult, so here is my full situation:
this process was spewing OOMEs a week ago. I thought I fixed it and I'm trying to verify whether my fixed worked without waiting another full week to see if it spews OOMEs again.
This task creates 7000-9000 ImapClients on start then under normal operation connects and disconnects very few of them.
I checked another process running older pre-OOME code, and it showed numbers of 9000/9100 instead of 7800/9600. I do not know why old code will be different from new code but this is evidence of a leak.
The point of this question is so I can determine if there is a leak. There is a business rule that every ImapClient should be a referee of a MonitoringTask. If this query I am asking about comes up empty, there is not a leak. If it comes up with objects, together with this business rule, it is not only evidence of a leak but conclusive proof of one.
Your expectations are incorrect, there is no actual evidence of any leaks occuring
The Garbage Collector's goal is to free space when it is needed and
only then, anything else is a waste of resources. There is absolutely
no benefit in attempting to keep as much free space as possible
available all the time and only down sides.
Just because something is a candidate for garbage collection doesn't
mean it will ever actually be collected, and there is no way to
force garbage collection either.
I don't see any mention of OutOfMemoryError anywhere.
What you are concerned about you can't control, not directly anyway
What you should focus on is what in in your control, which is making sure you don't hold on to references longer than you need to, and that you are not duplicating things unnecessarily. The garbage collection routines in Java are highly optimized, and if you learn how their algorithms work, you can make sure your program behaves in the optimal way for those algorithms to work.
Java Heap Memory isn't like manually managed memory in other languages, those rules don't apply
What are considered memory leaks in other languages aren't the same thing/root cause as in Java with its garbage collection system.
Most likely in Java memory isn't consumed by one single uber-object that is leaking ( dangling reference in other environments ).
Intermediate objects may be held around longer than expected by the garbage collector because of the scope they are in and lots of other things that can vary at run time.
EXAMPLE: the garbage collector may decide that there are candidates, but because it considers that there is plenty of memory still to be had that it might be too expensive time wise to flush them out at that point in time, and it will wait until memory pressure gets higher.
The garbage collector is really good now, but it isn't magic, if you are doing degenerate things, it will cause it to not work optimally. There is lots of documentation on the internet about the garbage collector settings for all the versions of the JVMs.
These un-referenced objects may just have not reached the time that the garbage collector thinks it needs them to for them to be expunged from memory, or there could be references to them held by some other object ( List ) for example that you don't realize still points to that object. This is what is most commonly referred to as a leak in Java, which is a reference leak more specifically.
I don't see any mention of OutOfMemoryError
You probably don't have a problem in your code, the garbage collection system just might not be getting put under enough pressure to kick in and deallocate objects that you think it should be cleaning up. What you think is a problem probably isn't, not unless your program is crashing with OutOfMemoryError. This isn't C, C++, Objective-C, or any other manual memory management language / runtime. You don't get to decide what is in memory or not at the detail level you are expecting you should be able to.
Check your code for finalizers, especially anything relating to IMapclient.
It could be that your MonitoringTasks are being easily collected whereas your IMapclient's are finalized, and therefore stay on the heap (though dead) until the finalizer thread runs.
The obvious answer is to add a WeakHashMap<X, Object> (and Y) to your code -- one tracking all instances of X and another tracking all instances of Y (make them static members of the class and insert every object into the map in the constructor with a null 'value'). Then you can at any time iterate over these maps to find all live instances of X and Y and see which Xs are not referenced by Ys. You might want to trigger a full GC first, to ignore objects that are dead and not yet collected.
I have an interesting problem with Java memory consumption. I have a native C++ application which invokes my Java application.
The Application basically does some language translations\parses a few XML's and responds to network requests. Most of the state of Application doesn't have to be retained so it is full of Methods which take in String arguments and returns string results.
This application continues to take more and more memory with time and there comes a time where it starts to take close to 2 GB memory, which made us suspect that there is a leak somewhere in some Hashtable or static variables. On closer inspection we did not find any leaks. Comparing heap dumps over a period of time, shows the char[] and String objects take huge memory.
However when we inspect these char[], Strings we find that they do not have GC roots which means that they shouldn't be the cause of leak. Since they are a part of heap, it means they are waiting to get garbage collected. After using verious tools MAT\VisualVM\JHat and scrolling through a lot of such objects I used the trial version of yourkit. Yourkit gives the data straightaway saying that 96% of the char[] and String are unreachable. Which means that at the time of taking dump 96% of the Strings in the heap were waiting to get garbage collected.
I understand that the GC runs sparingly but when you check via VisualVM you can actually see it running :-( than how come there are so many unused objects on the heap all time.
IMO this Application should never take more than 400-500 MB memory, which is where it stays for the first 24 hours but than it continues to increase the heap :-(
I am running Java 1.6.0-25.
thanks for any help.
Java doesn't GC when you think it does/should :-) GC is too complex a topic to understand what is going on without spending a couple of weeks really digging into the details. So if you see behavior that you can't explain, that doesn't mean its broken.
What you see can have several reasons:
You are loading a huge String into memory and keep a reference to a substring. That can keep the whole string in memory (Java doesn't always allocate a new char array for substrings - since Strings are immutable, it simply reuses the original char array and remembers the offset and length).
Nothing triggered the GC so far. Some C++ developers believe GC is "evil" (anything that you don't understand must be evil, right?) so they configure Java not to run it unless absolutely necessary. This means the VM will eat memory until it hits the maximum and then, it will do one huge GC run.
build 25 is already pretty old. Try to update to the latest Java build (33, I think). The GC is one of the best tested parts of the VM but it does have bugs. Maybe you hit one.
Unless you see OutOfMemoryException, you don't have a leak. We have an application which eats all the heap you give it. If it gets 16GB of RAM ("just to be safe"), it will use the whole 16GB because we cache what we can. You never see out of memory, because the cache will shrink as needed but system admins routinely freak out "oh god! oh god! It's running out of memory" PANIK No, it's not. Unless Java tells you so, it's not running out of memory. It's just using it efficiently.
Tuning the GC with command line options is one of the best ways to break it. Hundreds of people which know a lot more about the topic than you ever will spent years making the GC efficient. You think you can do better? Good luck. -> Get rid of any "magic" command line options and calls to System.gc() and your problem might go away.
Try decreasing the heap size to 500 Megabytes and see if the software will start garbage collecting or die. Java isnt too fussy about using memory given to it. you might also research GC tuning options which will make the GC more prudent about cleaning stuff up.
String reallyLongString = "this is a really long String";
String tinyString = reallyLongString.substring(2, 3);
reallyLongString = null
The JVM can't collect the memory allocated for the long string in the above case, since there's a reference to part of it.
If you're doing stuff with Strings and you're suffering from memory issues, this might be the cause of your grief.
use tinyString = new String(reallyLongString.substring(2, 3); instead.
There might not be a leak at all - a leak would be if the Strings were reachable. If you've allocated as much as 2GB to the application, there is no reason for the garbage collector to start freeing up memory until you are approaching that limit. If you don't want it taking any more than 500MB, then pass -Xmx 512m when starting the JVM.
You could also try tuning the garbage collector to start cleaning up much earlier.
First of all, stop worrying about those Strings and char[]. In almost every java application I have profiled, they are on the top of memory consumer list. And in almost no of those java application they were the real problem.
If you have not received OutOfMemoryError yet, but do worry that 2GB is too much for your java process, then try to decrease Xmx value you pass to it. If it runs well and good with 512m or 1g, then problem solved, isn't it?
If you get OOM, then one more option you can try is to use Plumbr with your java process. It is memory leak discovery tool, to it can help you if there really is a memory leak.
For a project for school I have to program different kind of algorithms. The problem is, I got a working algorithm. But I have to run it several times and after some time it gives me the following errors:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
I know what the error means, but is it possible to let Java search for empty space during the run? I know it uses a lot of space which isn't used at some point. It sets a lot of object to null during the application run and create a lot of new ones, because of this it runs out of memory.
So concrete: is it possible to let the JVM free some space that is set to null? Or free some space in the time the program is running? I know I can set the JVM to more space, but sooner or later I will run to the same problem.
If you need my IDE (in case it is IDE specific) it is Eclipse.
Please google 'garbage collection'. Java is always looking to reuse space from objects that you aren't using. If you run out of memory, you either need to use -Xmx to configure for more memory, or you have to fix your code to retain fewer objects. You may find that a profiler like jvisualvm would help you find wasteful memory usage.
If you're using an Oracle/Sun JVM, I'd recommend that you download Visual VM 1.3.3, install all the plugins, and start it up. It'll show you what's happening in every heap generation, threads, CPU, objects, etc. It can tell you which class is taking up the most heap space.
You'll figure it out quickly if you have data.
I would use a memory profiler to determine where the memory is being used. Setting to null rarely helps. The GC will always run and free as much space as possible before you get an OOME.
Q: "is it possible to let the JVM free some space that is set to null? Or free some space in the time the program is running?"
A: Yes, use a call to System.gc() will do this, but this will not likely solve your problem as the system does this automatically from time to time. You need to find the object that is using all the memory and fix it in your code. Likely a list that is never cleared and only ever added to.
I actually encountered this issue while implementing a particularly complicated algorithm that required a massive data structure. I had to come and post a question on this website. It turned out I had to use a completely different type of object altogether in order to avoid the memory error.
Here is that question.
GC will reclaim 'unused' memory automatically, so yes, it is possible to free some space at runtime, but it's crucial to understand what's classified as possible to be reclaimed.
Basically an object's space can be reclaimed (garbage collected) if the object itself is unreachable - there are no references to it. When you say 'setting space to null' you're most likely removing just one link (reference) to the object by setting it to null. This will allow to reclaim the object only if that was the only link (reference)
Object First= new Object(); //first object
Object Second= new Object(); //second object
Object SecondPrim=Second; //second reference to second object
First=null;
// First memory will be reclaimed (sooner or later)
Second=null;
// there is still a reference to second object via SecondPrim
// second object will not be reclaimed
Hope this helps. As for checking what's exactly going on I would second advice to profile your program.