Develop Reference

memory leak issue in code

I am working on an application & my code is having out of memory error. I am not able to see memory utilisation of the code.so I am very confused were to see.
Also after my little analysis I came to kow that there is private static object getting creating & in the constructor of that class. some more objects are getting created. & that class is multithreaded..
so I want to know since the static objects does not get garbage collected.. will all the objects related to the constructor will not be garbage collected??

A static reference is only collected when the class is unloaded, and this only happened when the class loader is not used any more. If you haven't got multiple class loaders it is likely this will never be unloaded (until your program stops)
However, just because an object was once referenced statically doesn't change how it is collected. If you had a static reference to an object and no longer have a reference to that object, it will be collected as normal.
Having multiple threads can make finding bugs harder, but it doesn't change how objects are collected either.
You need to take a memory dump of your application and see why memory is building up. It is possible the objects you retaining are all needed. In this case you need to
reduce your memory requirement
increase your maximum memory.

You might not have a memory leak - you might simply surpassed the amount of avaialble RAM your system can provide.
you can add several JVM arguments to limit the size of RAM allocated to your runtime enviorment, and control the garbage collector - the tradeoff is it usually consumes more CPU.

You say you are not capable of seeing the memory utilisation?
Have you tried using JVisualVM (in $JAVA_HOME/bin/jvisualvm)
It should be capable of attaching to local processes and take heap dumps.
Also, Eclipse Memory Analyzer has some good reports for subsequent analysis

Java Memory Leak - find all objects of class X not referenced by class Y

I have a memory leak in Java in which I have 9600 ImapClients in my heap dump and only 7800 MonitoringTasks. This is a problem since every ImapClient should be owned by a MonitoringTask, so those extra 1800 ImapClients are leaked.
One problem is I can't isolate them in the heap dump and see what's keeping them alive. So far I've only been able to pinpoint them by using external evidence to guess at which ImapClients are dangling. I'm learning OQL which I believe can solve this but it's coming slowly, and it'll take a while before I can understand how to perform something recursive like this in a new query language.
Determining a leak exists is difficult, so here is my full situation:
this process was spewing OOMEs a week ago. I thought I fixed it and I'm trying to verify whether my fixed worked without waiting another full week to see if it spews OOMEs again.
This task creates 7000-9000 ImapClients on start then under normal operation connects and disconnects very few of them.
I checked another process running older pre-OOME code, and it showed numbers of 9000/9100 instead of 7800/9600. I do not know why old code will be different from new code but this is evidence of a leak.
The point of this question is so I can determine if there is a leak. There is a business rule that every ImapClient should be a referee of a MonitoringTask. If this query I am asking about comes up empty, there is not a leak. If it comes up with objects, together with this business rule, it is not only evidence of a leak but conclusive proof of one.

Your expectations are incorrect, there is no actual evidence of any leaks occuring
The Garbage Collector's goal is to free space when it is needed and
only then, anything else is a waste of resources. There is absolutely
no benefit in attempting to keep as much free space as possible
available all the time and only down sides.
Just because something is a candidate for garbage collection doesn't
mean it will ever actually be collected, and there is no way to
force garbage collection either.
I don't see any mention of OutOfMemoryError anywhere.
What you are concerned about you can't control, not directly anyway
What you should focus on is what in in your control, which is making sure you don't hold on to references longer than you need to, and that you are not duplicating things unnecessarily. The garbage collection routines in Java are highly optimized, and if you learn how their algorithms work, you can make sure your program behaves in the optimal way for those algorithms to work.
Java Heap Memory isn't like manually managed memory in other languages, those rules don't apply
What are considered memory leaks in other languages aren't the same thing/root cause as in Java with its garbage collection system.
Most likely in Java memory isn't consumed by one single uber-object that is leaking ( dangling reference in other environments ).
Intermediate objects may be held around longer than expected by the garbage collector because of the scope they are in and lots of other things that can vary at run time.
EXAMPLE: the garbage collector may decide that there are candidates, but because it considers that there is plenty of memory still to be had that it might be too expensive time wise to flush them out at that point in time, and it will wait until memory pressure gets higher.
The garbage collector is really good now, but it isn't magic, if you are doing degenerate things, it will cause it to not work optimally. There is lots of documentation on the internet about the garbage collector settings for all the versions of the JVMs.
These un-referenced objects may just have not reached the time that the garbage collector thinks it needs them to for them to be expunged from memory, or there could be references to them held by some other object ( List ) for example that you don't realize still points to that object. This is what is most commonly referred to as a leak in Java, which is a reference leak more specifically.
I don't see any mention of OutOfMemoryError
You probably don't have a problem in your code, the garbage collection system just might not be getting put under enough pressure to kick in and deallocate objects that you think it should be cleaning up. What you think is a problem probably isn't, not unless your program is crashing with OutOfMemoryError. This isn't C, C++, Objective-C, or any other manual memory management language / runtime. You don't get to decide what is in memory or not at the detail level you are expecting you should be able to.

Check your code for finalizers, especially anything relating to IMapclient.
It could be that your MonitoringTasks are being easily collected whereas your IMapclient's are finalized, and therefore stay on the heap (though dead) until the finalizer thread runs.

The obvious answer is to add a WeakHashMap<X, Object> (and Y) to your code -- one tracking all instances of X and another tracking all instances of Y (make them static members of the class and insert every object into the map in the constructor with a null 'value'). Then you can at any time iterate over these maps to find all live instances of X and Y and see which Xs are not referenced by Ys. You might want to trigger a full GC first, to ignore objects that are dead and not yet collected.

Classloader held by weak references?

I've been battling some memory leaks, and I'm currently baffled by this issue. There's a web application classloader that was supposed to be garbage collected, but it isn't (even after I fixed several leaks). I dumped the heap with jmap and browsed it with jhat, found the classloader and checked the rootset references.
If I exclude weak refs, the list is empty! How is that possible, since an object held only by weak references should get garbage collected? (I performed GC many times in jconsole)
If I include weak refs, I get a list of references, all of which come from one of the following fields:
java.lang.reflect.Proxy.loaderToCache
java.lang.reflect.Proxy.proxyClasses
java.io.ObjectStreamClass$Caches.localDescs
java.io.ObjectStreamClass$Caches.reflectors
java.lang.ref.Finalizer.unfinalized
I couldn't find any reason why any of those references should prevent garbage collecting the classloader. Is it a gc bug? Special undocumented case? jmap/jhat bug? Or what?
And the weirdest thing... after sitting idle and gc-ing from time to time for about 40 min, without changing anything, it finally decided to unload classes and collect the classloader.
Note:
If you make a claim about delayed collection of classloaders or weak references, then please specify the circumstances in which it happens, and ideally:
provide a link to an authoritative article that supports your claim
provide a sample program that demonstrates the behavior
If you think the behavior is implementation-dependent, then please focus on what happens in the oracle or icedtea jvm, version 6 or 7 (pick any one of them and be specific).
I'd really like to get to the bottom of this. I actually put some effort into reproducing the issue in a test program, and I failed - the classloader was instantly collected on System.gc() every time unless there was a strong reference to it.

It looks like there's a soft reference involved somewhere. That's the only explanation I could find for the delayed collection (about 40 min). I initially thought soft references were kept until the memory runs out, but I found that that's not the case.
From this page: "softly reachable objects will remain alive for some amount of time after the last time they were referenced. The default value is one second of lifetime per free megabyte in the heap. This value can be adjusted using the -XX:SoftRefLRUPolicyMSPerMB flag"
So I adjusted that flag to 1, and the classloader was collected within seconds!!
I think the soft reference comes from ObjectStreamClass. The question is why jhat doesn't show it in the rootset references. Is it because it's neither strong nor weak? Or because it already found weak references from the same static fields? Or some other reason? Either way, I think this needs to be improved in jhat.

Classes reside in special memory space - permanent generation. To unload classloader. GC should choose to include perm space into scope of collection. Different GC algorithms have a little different behavior, but generally GC will try to avoid perm space collection.
In my experience, even if classloader is not reachable JVM may end up with OutOfMemoryError before it would try to collect PERM space.

using static dictionary as cache may lead to leak problem?

I have memory leak in the web application (servlet) that I am working on. I am suspicious of 1 cause and wanted to hear your ideas about it.
I use hashmaps,hashsets etc. as DB (about 20MB data loaded). These maps,sets get reloaded once in every 10 min. There are huge amount of simultaneous requests. I read that, GC passes objects, that are not collected for a time period/cycle, to a generation (old and permanent generations) which is less checked or garbage collected. I think that my usage for static maps,sets is causing me leak problem. What do you think ?

As Romain noted, the static map is a suspect. If for some reason you can't regularly clean it up explicitly, you may consider using a WeakHashMap instead, which is
A hashtable-based Map implementation with weak keys. An entry in a WeakHashMap will automatically be removed when its key is no longer in ordinary use. More precisely, the presence of a mapping for a given key will not prevent the key from being discarded by the garbage collector, that is, made finalizable, finalized, and then reclaimed. When a key has been discarded its entry is effectively removed from the map, so this class behaves somewhat differently than other Map implementations.
Unfortunately, as of Java6 there seems to be no WeakHashSet in the standard library, but several implementations can be found on the net.

It is not a leak if you have removed all references to it. If you're clearing out your map entirely, then it's not the source of a leak. You should consider the fact that the JVM chooses not to GC tenured generation very often as irrelevant to you - all that matters is that you don't have a reference to it, so the JVM could GC it if it wants to.
There are different strategies that JVMs can use to manage GC, so I'm speaking in generalities instead of specifics here, but GCing tenured spaces tends to be very expensive and has a high impact on the application, so the JVM chooses not to do it often in general.
If you're looking at the amount of heap space used, you'll see a sawtooth pattern as items are added and eventually collected. Don't be concerned about where the top of the sawtooth is, be concerned about where the bottom is (and how close to the maximum amount of heap space available that is).
One way to test if it really is a leak is to load test your app for a long period of time. If you have a leak, the base amount of memory that your app is using will go up over time (the bottom of the sawtooth). If you don't, it will remain constant. If you do have a leak, you can use a profiler to help you find it.

Static Maps are a known source of leaks. The reason being that people put stuff in and do not remove them. If every ten minutes you simply clear the cache and then reload then you should be fine.
I would bet that you are not clearing it properly. The GC part is working properly, I would not worry that it is the issue.

You may also want to consider using WeakReference if you have some way of falling back to the real data if part of your cache is GC-ed but then subsequently required.

I suggest that you check the heap contents using a heap dump and a heap analyzer (such as JVisualVM). This will help you find leakage suspects. The fact that the old generation is collected less frequently does not mean that more memory is leaking; remember that while it may seem full, only a portion of it represents live objects, and the other portion is clearned by the next major GC. Like others said, the problem may be because of incomplete cleaning of the static collections.
The permanent generation never receives promoted objects. It is an out-of-heap area reserved for other purposes, such as reflective information of loaded classes, and interned strings.

can any unused object escape from Garbage Collector?

Is there any possibility that a object which is not referenced anywhere and still existing on heap. I mean is there a possibility that a unused object getting escaped from garbage collector and be there on the heap until the end of the application.
Wanted to know because if it is there, then while coding i can be more cautious.

If an object is no longer referenced, it does still exist on the heap, but it is also free to be garbage-collected (unless we are talking Class objects, which live in PermGen space and never get garbage-collected - but this is generally not something you need to worry about).
There is no guarantee on how soon that will be, but your application will not run out of memory before memory from those objects is reclaimed.
However, garbage collection does involve overhead, so if you are creating more objects than you need to and can easily create less, then by all means do so.
Edit: in response to your comment, if an object is truly not referenced by anything, it will be reclaimed during garbage collection (assuming you are using the latest JVM from Sun; I can't speak toward other implementations). The reason why is as follows: all objects are allocated contiguously on the heap. When GC is to happen, the JVM follows all references to "mark" objects that it knows are reachable - these objects are then moved into another, clean area. The old area is then considered to be free memory. Anything that cannot be found via a reference cannot be moved. The point is that the GC does not need to "find" the unreferenced objects. If anything, I would be more worried about objects that are still referenced when they are not intended to be, which will cause memory leaks.

You should know that, before a JVM throws an out-of-memory exception, it will have garbage collected everything possible.

If an instance is no longer referenced, it is a possible candidate for garbage collection. This means, that sooner or later it can be removed but there are no guaranties. If you do not run out of of memory, the garbage collector might not even run, thus the instance my be there until the program ends.
The CG system is very good at finding not referenced objects. There is a tiny, tiny chance that you end up keeping a weird mix of references where the garbage collector can not decide for sure if the object is no longer referenced or not. But this would be a bug in the CG system and nothing you should worry about while coding.

It depends on when and how often the object is used. If you allocate something then deallocate (i.e., remove all references to it) it immediately after, it will stay in "new" part of the heap and will probably be knocked out on the next garbage collection run.
If you allocate an object at the beginning of your program and keep it around for a while (if it survives through several garbage collections), it will get promoted to "old" status. Objects in that part of the heap are less likely to be collected later.
If you want to know all the nitty-gitty details, check out some of Sun's gc documentation.

Yes; imagine something like this:
Foo foo = new Foo();
// do some work here
while(1) {};
foo.someOp(); // if this is the only reference to foo,
// it's theoreticaly impossible to reach here, so it
// should be GC-ed, but all GC systems I know of will
// not Gc it
I am using definition of: garbage = object that can never be reached in any execution of the code.

Garbage collection intentionally makes few guarantees about WHEN the objects are collected. If memory never gets too tight, it's entirely possible that an unreferenced object won't be collected by the time the program ends.

The garbage collector will eventually reclaim all unreachable objects. Note the "eventually": this may take some time. You can somewhat force the issue with System.gc() but this is rarely a good idea (if used without discretion, then performance may decrease).
What can happen is that an object is "unused" (as in: the application will not use it anymore) while still being "reachable" (the GC can find a path of references from one of its roots -- static fields, local variables -- to the object). If you are not too messy with your objects and structures then you will not encounter such situations. A rule of thumb would be: if the application seems to take too much RAM, run a profiler on it; if thousands of instances of the same class have accumulated without any apparent reason, then there may be some fishy code somewhere. Correction often involves explicitly setting a field to null to avoid referencing an object for too long.

This is theoretically possible (there is no guarantee the GC will always find all objects), but should not worry you for any real application - it usually does not happen and certainly does not affect a significant chunk of memory.

In theory, the garbage collector will find all unused objects. There could, of course, be bugs in the garbage collector…
That said, "In theory there is no difference between theory and practice, in practice, there is." Under some, mostly older, garbage collectors, if an object definition manages to reach the permanent generation, then it will no longer be garbage collected under any circumstances. This only applied to Class definitions that were loaded, not to regular objects that were granted tenured status.
Correspondingly, if you have a static reference to an object, that takes up space in the "regular" object heap, this could conceivably cause problems, since you only need to hold a reference to the class definition from your class definition, and that static data cannot be garbage collected, even if you don't actually refer to any instances of the class itself.
In practice though, this is a very unlikely event, and you shouldn't need to worry about it. If you are super concerned about performance, then creating lots of "long-lived" objects, that is, those that escape "escape-analysis", will create extra work for the garbage collector. For 99.99% of coders this is a total non-issue though.

My advice - Don't worry about it.
Reason - It is possible for a non-referenced object to stay on the heap for some time, but it is very unlikely to adversely affect you because it is guaranteed to be reclaimed before you get an out of memory error.

In general, all objects to which there are no live hard references, will be garbage-collected. This is what you should assume and code for. However, the exact moment this happens is not predictable.
Just for completeness, two tricky situations [which you are unlikely to run into] come into my mind:
Bugs in JVM or garbage collector code
So called invisible references - they rarely matter but I did have to take them into account one or two times during the last 5 years in a performance-sensitive application I work on