Develop Reference

Java Memory Leak - find all objects of class X not referenced by class Y

I have a memory leak in Java in which I have 9600 ImapClients in my heap dump and only 7800 MonitoringTasks. This is a problem since every ImapClient should be owned by a MonitoringTask, so those extra 1800 ImapClients are leaked.
One problem is I can't isolate them in the heap dump and see what's keeping them alive. So far I've only been able to pinpoint them by using external evidence to guess at which ImapClients are dangling. I'm learning OQL which I believe can solve this but it's coming slowly, and it'll take a while before I can understand how to perform something recursive like this in a new query language.
Determining a leak exists is difficult, so here is my full situation:
this process was spewing OOMEs a week ago. I thought I fixed it and I'm trying to verify whether my fixed worked without waiting another full week to see if it spews OOMEs again.
This task creates 7000-9000 ImapClients on start then under normal operation connects and disconnects very few of them.
I checked another process running older pre-OOME code, and it showed numbers of 9000/9100 instead of 7800/9600. I do not know why old code will be different from new code but this is evidence of a leak.
The point of this question is so I can determine if there is a leak. There is a business rule that every ImapClient should be a referee of a MonitoringTask. If this query I am asking about comes up empty, there is not a leak. If it comes up with objects, together with this business rule, it is not only evidence of a leak but conclusive proof of one.

Your expectations are incorrect, there is no actual evidence of any leaks occuring
The Garbage Collector's goal is to free space when it is needed and
only then, anything else is a waste of resources. There is absolutely
no benefit in attempting to keep as much free space as possible
available all the time and only down sides.
Just because something is a candidate for garbage collection doesn't
mean it will ever actually be collected, and there is no way to
force garbage collection either.
I don't see any mention of OutOfMemoryError anywhere.
What you are concerned about you can't control, not directly anyway
What you should focus on is what in in your control, which is making sure you don't hold on to references longer than you need to, and that you are not duplicating things unnecessarily. The garbage collection routines in Java are highly optimized, and if you learn how their algorithms work, you can make sure your program behaves in the optimal way for those algorithms to work.
Java Heap Memory isn't like manually managed memory in other languages, those rules don't apply
What are considered memory leaks in other languages aren't the same thing/root cause as in Java with its garbage collection system.
Most likely in Java memory isn't consumed by one single uber-object that is leaking ( dangling reference in other environments ).
Intermediate objects may be held around longer than expected by the garbage collector because of the scope they are in and lots of other things that can vary at run time.
EXAMPLE: the garbage collector may decide that there are candidates, but because it considers that there is plenty of memory still to be had that it might be too expensive time wise to flush them out at that point in time, and it will wait until memory pressure gets higher.
The garbage collector is really good now, but it isn't magic, if you are doing degenerate things, it will cause it to not work optimally. There is lots of documentation on the internet about the garbage collector settings for all the versions of the JVMs.
These un-referenced objects may just have not reached the time that the garbage collector thinks it needs them to for them to be expunged from memory, or there could be references to them held by some other object ( List ) for example that you don't realize still points to that object. This is what is most commonly referred to as a leak in Java, which is a reference leak more specifically.
I don't see any mention of OutOfMemoryError
You probably don't have a problem in your code, the garbage collection system just might not be getting put under enough pressure to kick in and deallocate objects that you think it should be cleaning up. What you think is a problem probably isn't, not unless your program is crashing with OutOfMemoryError. This isn't C, C++, Objective-C, or any other manual memory management language / runtime. You don't get to decide what is in memory or not at the detail level you are expecting you should be able to.

Check your code for finalizers, especially anything relating to IMapclient.
It could be that your MonitoringTasks are being easily collected whereas your IMapclient's are finalized, and therefore stay on the heap (though dead) until the finalizer thread runs.

The obvious answer is to add a WeakHashMap<X, Object> (and Y) to your code -- one tracking all instances of X and another tracking all instances of Y (make them static members of the class and insert every object into the map in the constructor with a null 'value'). Then you can at any time iterate over these maps to find all live instances of X and Y and see which Xs are not referenced by Ys. You might want to trigger a full GC first, to ignore objects that are dead and not yet collected.

How to keep some data as long as possible, but allow the VM to free & reload them at need?

In my application, I would like to load some amount of data into memory when first needed and keep it there in case another part of the application wants to use it. The same data would be accessed from a couple of different Activity'es, but by far not all the user could interact with. So, when not working with the relevant part of my application, I would like Android to feel free to discard the data, reloading them again at need. Note that it is unpredictable for me what the user will do, so I want Android to free the data only if hasn't been used for some time. What is a good approach to doing this?
I thought of creating a class that would be only used statically, loading the data in its static initialisation block. However, I am not sure if Dalvik would ever discard any static data stored this way. I have read something on class loaders but I have no idea what loader is used in loading my class and how it could potentially become discarded. Perhaps someone does...?
Another way I came up with is using weak reference to keep an instance of the data-holding class (non-static, obviously) but here I am afraid that the GC could decide it's useless when no Activity is currently actively operating it, even when memory is no concern at that moment. (In that case, I would like to keep the data loaded.)
The loading of my data is costly. I want, if possible, to destroy it only when the system is running out of memory or when the application exits.

It sounds like SoftReferences are what you need. These are cleared at the garbage collector's discretion when it detects that there is a memory shortfall.
If you read the class javadoc, it gives some hints about how to prevent recently used cache entries from being reclaimed.
For the record, classloaders won't help you manage instances of a class. But making the cache a static should allow the cached objects to be discarded if the cache class gets unloaded.
FOLLOWUP
My data is a solid block that would be represented by a single object.
This rather changes things. If you have a single object to cache, then LRU makes no sense. Basically it sounds like you want to hang onto the object as long as possible ... without trigger OOMEs by hanging onto it too long. This is kind of hard. Indeed, doing a perfect job is going to entail correctly predicting what the user is going to do ... which is clearly impossible.
Possibly the best strategy is to make use of the reference enqueing mechanism, and implement the queue processor to make an "intelligent" choice between letting the object die or recreating the soft link. The "intelligence" might entail looking at how much free memory there is, and / or how long it was since the object was last used. But beware!! If you get this wrong you can cause OOMEs or cause the platform to spend lots of time thrashing the garbage collector.
If I set up the cache to hold 1 object, it would be equivalent to a hard reference, wouldn't it?
Nope. If you use a SoftReference the GC will break the reference if it is running out of memory.

You can use SoftReferences. Take a look at:
http://docs.oracle.com/javase/6/docs/api/java/lang/ref/SoftReference.html
With SoftReferences you can achieve what you need:
" I want, if possible, to destroy it only when the system is running out of memory or when the application exits."

Take a look at
SoftReference gets garbage collected too early
You can also look into LruCache if your looking to cache some data in memory your app.
http://developer.android.com/reference/android/support/v4/util/LruCache.html
For a longer lived disk based cache take a look at Android Objects Cache
You can find the DiskLruCache source at https://github.com/JakeWharton/DiskLruCache/

System.gc will work in android(Andengine)

I developing a game using andengine. In J2me game when exit , i made all object as null
ie:
Image img;
Sprite s1;
When exit application ,
img=null;
s1=null;
In android i will use System.gc() or i need to make all texture, textureRegion and sprite as make as null, when exit appliaction ?

i think you should not call System.gc() explicitly. Android OS takes care of that.
"Calling System.gc() from your app is like providing electricity connection from your home to light up your complete society's lights"
I mean it slows down your app to clean all the garbages of the system.......
N_JOY.

Java garbage collection should take care of that. You don't need to do that.
However I would close open connections, file handles, etc..
System.gc() is just a hint to the JVM that garbage collection is suggested, however Java is running it at its own will.

In Android in the presence of a garbage collector, it is never good practice to manually call the GC. A GC is organized around heuristic algorithms which work best when left to their own devices. Calling the GC manually often decreases performance.
Occasionally, in some relatively rare situations, one may find that a particular GC gets it wrong, and a manual call to the GC may then improves things, performance-wise. This is because it is not really possible to implement a "perfect" GC which will manage memory optimally in all cases. Such situations are hard to predict and depend on many subtle implementation details. The "good practice" is to let the GC run by itself; a manual call to the GC is the exception, which should be envisioned only after an actual performance issue has been duly witnessed.
It's better to spend more effort in avoiding the unnecessary creation of objects (like creation of objects inside loops)..
Look at the Question Garbage collector in Android

Android proper clean up/disposing

Is there a way to "clean up" objects and other variables you create? Or are they automatically disposed of or do I have this whole concept wrong? What is the proper way to go about doing this? I am trying to avoid the GC as much as possible.

The only way to cleanup in an GC language with no memory management is the GC . You can force GC but its not recommended , the GC is pretty good , to be more proactive set objects to null for the GC to clean up.
Addition:
Also try to make objects as local as possible , that way they are GCed as they scope out.

Calling System.gc() will force Garbage Collection to happen.
There is a system counting references to objects you create. If you are looping a lot and creating lots of objects you will create periods of time where they pile up. The system will collect the garbage when your processor is not doing anything, or it will wait till you need more free memory before collection occurs. If you have been processing for some time, you will experience hiccups in your performance due to Garbage Collection happening during your processes.
Please view this page and search for "Garbage Collection"
http://developer.android.com/guide/practices/design/performance.html
NOTE: Anything created with an Application Context will live until the end of the application execution. Anything created with an Activity Context will live until the end of the activity. This two situations can cause memory leaks!

For a more complete answer specific to Android:
Make sure you review the application lifecycle for android. It will help you avoid activity leaks in Android.

For the most part they are cleaned up as long as you do not maintain a reference to the object (variable). Something's like cursor's and bitmap's though need to be closed before they can be deleted to prevent memory leaks.
I don't think you have to worry about the GC as long as your object creation is not over the top. Note: GC is a part of java. You can't avoid it.
Addendum 1: If you really are that worried about it, you could reuse variables. That way you keep object creation to a minimum, but in so doing you will lose that variable and will be unable to store a wide range of data.

Android's activities have onDestroy() method. You can use this method to close open connections or dialogs or close some pending tasks.
You could also read about Java GC to get a more proper understanding of it. I would recommend SCJP book, Garbage collection chapter. It explains well when an object becomes eligible for garbage collection.

Why do you not explicitly call finalize() or start the garbage collector?

After reading this question, I was reminded of when I was taught Java and told never to call finalize() or run the garbage collector because "it's a big black box that you never need to worry about". Can someone boil the reasoning for this down to a few sentences? I'm sure I could read a technical report from Sun on this matter, but I think a nice, short, simple answer would satisfy my curiosity.

The short answer: Java garbage collection is a very finely tuned tool. System.gc() is a sledge-hammer.
Java's heap is divided into different generations, each of which is collected using a different strategy. If you attach a profiler to a healthy app, you'll see that it very rarely has to run the most expensive kinds of collections because most objects are caught by the faster copying collector in the young generation.
Calling System.gc() directly, while technically not guaranteed to do anything, in practice will trigger an expensive, stop-the-world full heap collection. This is almost always the wrong thing to do. You think you're saving resources, but you're actually wasting them for no good reason, forcing Java to recheck all your live objects “just in case”.
If you are having problems with GC pauses during critical moments, you're better off configuring the JVM to use the concurrent mark/sweep collector, which was designed specifically to minimise time spent paused, than trying to take a sledgehammer to the problem and just breaking it further.
The Sun document you were thinking of is here: Java SE 6 HotSpot™ Virtual Machine Garbage Collection Tuning
(Another thing you might not know: implementing a finalize() method on your object makes garbage collection slower. Firstly, it will take two GC runs to collect the object: one to run finalize() and the next to ensure that the object wasn't resurrected during finalization. Secondly, objects with finalize() methods have to be treated as special cases by the GC because they have to be collected individually, they can't just be thrown away in bulk.)

Don't bother with finalizers.
Switch to incremental garbage collection.
If you want to help the garbage collector, null off references to objects you no longer need. Less path to follow= more explicitly garbage.
Don't forget that (non-static) inner class instances keep references to their parent class instance. So an inner class thread keeps a lot more baggage than you might expect.
In a very related vein, if you're using serialization, and you've serialized temporary objects, you're going to need to clear the serialization caches, by calling ObjectOutputStream.reset() or your process will leak memory and eventually die.
Downside is that non-transient objects are going to get re-serialized.
Serializing temporary result objects can be a bit more messy than you might think!
Consider using soft references. If you don't know what soft references are, have a read of the javadoc for java.lang.ref.SoftReference
Steer clear of Phantom references and Weak references unless you really get excitable.
Finally, if you really can't tolerate the GC use Realtime Java.
No, I'm not joking.
The reference implementation is free to download and Peter Dibbles book from SUN is really good reading.

As far as finalizers go:
They are virtually useless. They aren't guaranteed to be called in a timely fashion, or indeed, at all (if the GC never runs, neither will any finalizers). This means you generally shouldn't rely on them.
Finalizers are not guaranteed to be idempotent. The garbage collector takes great care to guarantee that it will never call finalize() more than once on the same object. With well-written objects, it won't matter, but with poorly written objects, calling finalize multiple times can cause problems (e.g. double release of a native resource ... crash).
Every object that has a finalize() method should also provide a close() (or similar) method. This is the function you should be calling. e.g., FileInputStream.close(). There's no reason to be calling finalize() when you have a more appropriate method that is intended to be called by you.

Assuming finalizers are similar to their .NET namesake then you only really need to call these when you have resources such as file handles that can leak. Most of the time your objects don't have these references so they don't need to be called.
It's bad to try to collect the garbage because it's not really your garbage. You have told the VM to allocate some memory when you created objects, and the garbage collector is hiding information about those objects. Internally the GC is performing optimisations on the memory allocations it makes. When you manually try to collect the garbage you have no knowledge about what the GC wants to hold onto and get rid of, you are just forcing it's hand. As a result you mess up internal calculations.
If you knew more about what the GC was holding internally then you might be able to make more informed decisions, but then you've missed the benefits of GC.

The real problem with closing OS handles in finalize is that the finalize are executed in no guaranteed order. But if you have handles to the things that block (think e.g. sockets) potentially your code can get into deadlock situation (not trivial at all).
So I'm for explicitly closing handles in a predictable orderly manner. Basically code for dealing with resources should follow the pattern:
SomeStream s = null;
...
try{
s = openStream();
....
s.io();
...
} finally {
if (s != null) {
s.close();
s = null;
}
}
It gets even more complicated if you write your own classes that work via JNI and open handles. You need to make sure handles are closed (released) and that it will happen only once. Frequently overlooked OS handle in Desktop J2SE is Graphics[2D]. Even BufferedImage.getGrpahics() can potentially return you the handle that points into a video driver (actually holding the resource on GPU). If you won't release it yourself and leave it garbage collector to do the work - you may find strange OutOfMemory and alike situation when you ran out of video card mapped bitmaps but still have plenty of memory. In my experience it happens rather frequently in tight loops working with graphics objects (extracting thumbnails, scaling, sharpening you name it).
Basically GC does not take care of programmers responsibility of correct resource management. It only takes care of memory and nothing else. The Stream.finalize calling close() IMHO would be better implemented throwing exception new RuntimeError("garbage collecting the stream that is still open"). It will save hours and days of debugging and cleaning code after the sloppy amateurs left the ends lose.
Happy coding.
Peace.

The GC does a lot of optimization on when to properly finalize things.
So unless you're familiar with how the GC actually works and how it tags generations, manually calling finalize or start GC'ing will probably hurt performance than help.

Avoid finalizers. There is no guarantee that they will be called in a timely fashion. It could take quite a long time before the Memory Management system (i.e., the garbage collector) decides to collect an object with a finalizer.
Many people use finalizers to do things like close socket connections or delete temporary files. By doing so you make your application behaviour unpredictable and tied to when the JVM is going to GC your object. This can lead to "out of memory" scenarios, not due to the Java Heap being exhausted, but rather due to the system running out of handles for a particular resource.
One other thing to keep in mind is that introducing the calls to System.gc() or such hammers may show good results in your environment, but they won't necessarily translate to other systems. Not everyone runs the same JVM, there are many, SUN, IBM J9, BEA JRockit, Harmony, OpenJDK, etc... This JVM all conform to the JCK (those that have been officially tested that is), but have a lot of freedom when it comes to making things fast. GC is one of those areas that everyone invests in heavily. Using a hammer will often times destroy that effort.