Develop Reference

Detecting out of memory errors

I would like to provide my system with a way of detecting whether out of memory exception has occurred or not. The aim for this exercise is to expose this flag through JMX and act correspondingly (e.g. by configuring a relevant alert on the monitoring system), as otherwise these errors sit unnoticed for days.
Naive approach for this would be to set an uncaught exception handler for every thread and check whether the raised exception is instance of OutOfMemoryError and set a relevant flag. However, this approach isn't realistic for the following reasons:
The exception can occur anywhere, including 3rd party libraries. There is nothing I can do to prevent them catching Throwable and keeping it for themselves.
Libraries can spawn their own threads and I have no way of enforcing uncaught exception handlers for these threads.
One of possible scenarios I see is bytecode manipulation (e.g. attaching some sort of aspect on top of OutOfMemoryError), however I am not sure if that's right approach or whether this is doable in general.
We have -XX:+HeapDumpOnOutOfMemoryError enabled, but I don't see this as a solution for this problem as it was designed for something else - and it provides no Java callback when this happens.
Has anyone done this? How would you solve it or suggest solving it? Any ideas are welcome.

You could use an out of memory warning system; this OutOfMemoryError Warning System can be an inspiration. You could configure a listener which is invoked after a certain memory threshold ( say 80%) is breached - you can use this invocation to start taking corrective measures.
We use something similar, where we suspend the component's service when the memory threshold of the component reaches 80% and start the clean up action; the component comes back only when the used memory comes below a another configurable value threshold.

There is an article based on the post that Scorpion has already given a link to.
The technique is again based on using MemoryPoolMXBean and subscribing to the "memory threshold exceeded" event, but it's slightly different from what was described in original post.
Author states that when you subscribe for the plain "memory threshold exceeded" event, there is a possibility of "false alarm". Imagine a situation when the memory consumption is above the threshold, but there will be a garbage collection performed soon and a lot of the memory is freed after that. In fact that situation is quite common in real world applications.
Fortunately, there is another threshold, "collection usage threshold", and a corresponding event, which is fired based on memory consumption right after garbage collection. When you receive that event, you can be much more confident you're running out of memory.

We have -XX:+HeapDumpOnOutOfMemoryError enabled, but I don't see this
as a solution for this problem as it was designed for something else -
and it provides no Java callback when this happens.
This flag should be all that you need. Set the output directory of the resulting heap dump file in some known location that you check regularly. Having a callback would be of no use to you. If you are out of memory, you can't guarantee that the callback code has enough memory to execute! All you can do is collect the data and use an external program to analyze why you ran out of memory. Any attempt at recovering in process can create bigger problems.
Bytecode instrumentation is possible - but hard. HPjmeter's monitoring tool has the ability to predict future OOM's (with caveats) -- but only on HP-UX/Itanium based systems. You could dedicate a daemon thread to calculating used memory in process and trigger an alert when this is exceeded, but really you're not solving the problem.

You can catch any and all uncaught exceptions with the static Thread.setDefaultUncaughtExceptionHandler. Of course, it doesn't help if someone is catching all Throwables. (I don't think anything will, though with an OOME I'd suspect you'd get a cascading effect until something outside the offending try block blew up.) Hopefully the thread would have released enough memory for the exception handler to work; OOM errors do tend to multiply as you try to deal with them.

How does finalize() work in java?

So, I recently discovered the finalize method in Java (not sure why I missed it before, but there it is). This seems like it could be the answer to a lot of the issues I'm working with, but I wanted to get a bit more information first.
Online, I found this diagram illustrating the process of garbage collection and finalize:
A couple of questions:
This takes place in a separate thread, correct?
What happens if I instantiate a new object during finalize? Is that allowed?
What happens if I call on a static method from finalize?
What happens if I establish a new reference to the object from within finalize?
I suppose I should explain why I'm interested. I work with LWJGL a lot, and it seems that if I could use finalize to cause Java objects to automatically clean up OpenGL resources, then I could do some really nice things in terms of an API.

finalize() is called by the Java Garbage Collector when it detects that no references to that particular object exists. finalize() is inherited by all Java objects through the Object class.
As far as I am aware you would have no difficulty making static method calls from a finalize() method I and you could establish a new reference to it from finalize() - however I would say this is poor programming practice.
You shouldn't rely on finalize() for clearing up, and it is better to clear up as you go. I prefer to use try, catch, finally for clearing up, rather than using finalize(). In particular, by using finalize() you will cause the JVM to hold onto all other objects that your finalizable object references, just in case it makes calls to them. This means your holding onto memory you might not need to use. More importantly, this also means you can cause the JVM to never end up disposing of objects, because they have to hold onto them incase another objects finalize method needs it e.g. a race condition.
Also, consider that it is entirely possible that GC won't be called. Therefore you can't actually guarantee that finalize() will ever be called.
Clear up resources as and when you are finished with them, and do not rely on finalize() to do it is my advice.

I don't think there are any guarantees about what thread will be used. New objects may be instantiated and static methods may be called. Establishing a new reference to your object will prevent it from being garbage collected, but the finalize method will not be called again--you don't want to do this.
Cleaning up resources is precisely what the finalize method is for, so you should be good there. A couple of warnings, though:
The method is not guaranteed to be called. If you have tied up resources that will not automatically be freed when your program stops do not depend on finalize.
When the method is called is not guaranteed. With memory tight, this will be sooner. With lots of free memory, it will be later if at all. This may suit you fine: with lots of memory you may not be all that concerned about freeing up the resources. (Though hanging on to them may interfere with other software running at the same time, in which case you would be concerned.)
My ususal solution is to have some kind of dispose method that does the clean up. I call it explicitly at some point if I can, and as soon as I can. Then I add a finalize method that just calls the dispose method. (Note that the dispose method must behave well when when called more than once! Indeed, with this kind of programming I might call dispose several times outside finalize, not being sure if previous calls were made successfully and yet wanting it to be called effectively as soon as possible.) Now, ideally, my resources are freed as soon as I no longer need them. However, if I lose track of the object with the resources, the finalize method will bail me out when memory runs short and I need the help.

First of all, remember there is no guarantee that finalization will even be run at all for all your objects. You can use it to free memory allocated in native code associated with an object, but for pure Java code most use cases are only to perform a "backup" mechanism of cleaning up resources. This means in most cases you should free resources manually and finalizers could act only a sort of helper to clean up if you forget to do it the standard way. However, you can't use them as the only or the main mechanism of cleanup. Even more generally, you shouldn't write any code whose correctness depends on finalizers being run.
Ad 1. As far as I know, there are no guarantees about what thread calls finalize(), though in practice this will probably be one of the GC threads.
Ad 2. Instantiating new objects is allowed. However, there are a number of pitfalls with handling object references in finalizers. In particular, if you store a hard reference to the object being finalized in some live object, you can prevent your about-to-be-garbage-collected object from being cleaned up. This kind of object resurrection may lead to exhausting your resources if it gets out of control. Also, watch out for exceptions in finalize() - they may halt the finalization, but there's no automatic way for your program to learn about them. You need to wrap the code in try-catch blocks and propagate the information yourself. Also, long execution time of finalizers may cause the queue of objects to build up and consume lots of memory. Some other noteworthy problems and limitations are described in this JavaWorld article.
Ad 3. There shouldn't be any issues with calling static methods from finalizers.
Ad 4. As mentioned in point 2, it is possible to prevent an object from being garbage collected (to resurrect it) by placing a reference to it in another live object during finalization. However, this is tricky behavior and probably not good practice.
To sum up, you can't rely on finalizers for cleaning up your resources. You need to handle that manually and finalizers in your case may at best be used as a backup mechanism to cover up after sloppy coding to some degreee. This means, unfortunately, your idea of making the API nicer by using finalizers to clean up OpenGL resources probably won't work.

is memory leak? why java.lang.ref.Finalizer eat so much memory

I ran a heap dump on my program. When I opened it in the memory analyzer tool, I found that the java.lang.ref.Finalizer for org.logicalcobwebs.proxool.ProxyStatement was taking up a lot of memory. Why is this so?

Some classes implement the Object.finalize() method. Objects which override this method need to called by a background thread call finalizer, and they can't be cleaned up until this happens. If these tasks are short and you don't discard many of these it all works well. However if you are creating lots of these objects and/or their finalizers take a long time, the queue of objects to be finalized builds up. It is possible for this queue to use up all the memory.
The solution is
don't use finalize()d objects if you can (if you are writing the class for the object)
make finalize very short (if you have to use it)
don't discard such objects every time (try to re-use them)
The last option is likely to be best for you as you are using an existing library.

From what I can make out, Proxool is a connection pool for JDBC connections. This suggests to me that the problem is that your application is misusing the connection pool. Instead of calling close on the statement objects, your code is probably dropping them and/or their parent connections. The Proxool is relying on finalizers to close the underlying driver-implemented objects ... but this requires those Finalizer instances. It could also mean that you are causing the connection to open / close (real) database connections more frequently than is necessary, and that would be bad for performance.
So I suggest that you check your code for leaked ResultSet, Statement and/or Connection objects, and make sure that you close them in finally blocks.
Looking at the memory dump, I expect you are concerned where the 898,527,228 bytes are going. The vast majority are retained by the Finalizer object whose id is 2aab07855e38. If you still have the dump file, take a look at what that Finalizer is referring to. It looks more problematic than the Proxool objects.

It may be late, But I had a similar issue and figured out that we need to tune up the garbage collectors, Can't keep serial and parallel GC, and G1 GC was also not working properly.
But when using concurrentMarkSweep GC we were able to stop building this queue too large.

Garbage collector and finalize() method

You people may think that within 15-20 minutes i have asked 4-5 questions on the same topic, so I may need a tutorial on this. But i am getting these questions by reading about GC.
So my question is GC will call a finalize() method on an instance only once of its life cycle even though if the same object is made uneligible to garbage collect in its finalize() method. So i wanted to know that how GC will come to know that it has executed its finalize() method once before while collecting it for the second time

Honestly, your life will be much better if you forget that finalizers exist. I've been coding Java for years and never had a reason to use a finalizer.
They're slow, not well defined (sometimes they'll never run!), and generally a PITA.
Do something along the lines of the Closeable interface instead if you are managing external resources, and use try{} finally{} blocks to clean up. Otherwise, try as much as you can to trust the language to clean up memory after itself.

Implementation dependent. Presumably the VM either has a secret bit on every object, or has a table containing objects that have already been "finalized". If I had to guess, I'd say the latter since presumably the set of already finalized objects that are still hanging around is expected to be small, so having a bit on every object in the system seems a bit wasteful.

Ah you again :)
Note that on the subject of finalizers, if you really FORCE a GC using JVMTI's ForceGargabeCollection, it is specifically stated that:
"This function does not cause finalizers to be run."
Once again, you probably do not really want to do that, but then if you have 2K+ rep and 5 questions about the GC is think that it's interesting to keep repeating that using JVMTI's ForceGarbageCollection you can really FORCE a GC.
Authoritative info as to how to force a GC:
http://java.sun.com/javase/6/docs/platform/jvmti/jvmti.html#ForceGarbageCollection

Why do you not explicitly call finalize() or start the garbage collector?

After reading this question, I was reminded of when I was taught Java and told never to call finalize() or run the garbage collector because "it's a big black box that you never need to worry about". Can someone boil the reasoning for this down to a few sentences? I'm sure I could read a technical report from Sun on this matter, but I think a nice, short, simple answer would satisfy my curiosity.

The short answer: Java garbage collection is a very finely tuned tool. System.gc() is a sledge-hammer.
Java's heap is divided into different generations, each of which is collected using a different strategy. If you attach a profiler to a healthy app, you'll see that it very rarely has to run the most expensive kinds of collections because most objects are caught by the faster copying collector in the young generation.
Calling System.gc() directly, while technically not guaranteed to do anything, in practice will trigger an expensive, stop-the-world full heap collection. This is almost always the wrong thing to do. You think you're saving resources, but you're actually wasting them for no good reason, forcing Java to recheck all your live objects “just in case”.
If you are having problems with GC pauses during critical moments, you're better off configuring the JVM to use the concurrent mark/sweep collector, which was designed specifically to minimise time spent paused, than trying to take a sledgehammer to the problem and just breaking it further.
The Sun document you were thinking of is here: Java SE 6 HotSpot™ Virtual Machine Garbage Collection Tuning
(Another thing you might not know: implementing a finalize() method on your object makes garbage collection slower. Firstly, it will take two GC runs to collect the object: one to run finalize() and the next to ensure that the object wasn't resurrected during finalization. Secondly, objects with finalize() methods have to be treated as special cases by the GC because they have to be collected individually, they can't just be thrown away in bulk.)

Don't bother with finalizers.
Switch to incremental garbage collection.
If you want to help the garbage collector, null off references to objects you no longer need. Less path to follow= more explicitly garbage.
Don't forget that (non-static) inner class instances keep references to their parent class instance. So an inner class thread keeps a lot more baggage than you might expect.
In a very related vein, if you're using serialization, and you've serialized temporary objects, you're going to need to clear the serialization caches, by calling ObjectOutputStream.reset() or your process will leak memory and eventually die.
Downside is that non-transient objects are going to get re-serialized.
Serializing temporary result objects can be a bit more messy than you might think!
Consider using soft references. If you don't know what soft references are, have a read of the javadoc for java.lang.ref.SoftReference
Steer clear of Phantom references and Weak references unless you really get excitable.
Finally, if you really can't tolerate the GC use Realtime Java.
No, I'm not joking.
The reference implementation is free to download and Peter Dibbles book from SUN is really good reading.

As far as finalizers go:
They are virtually useless. They aren't guaranteed to be called in a timely fashion, or indeed, at all (if the GC never runs, neither will any finalizers). This means you generally shouldn't rely on them.
Finalizers are not guaranteed to be idempotent. The garbage collector takes great care to guarantee that it will never call finalize() more than once on the same object. With well-written objects, it won't matter, but with poorly written objects, calling finalize multiple times can cause problems (e.g. double release of a native resource ... crash).
Every object that has a finalize() method should also provide a close() (or similar) method. This is the function you should be calling. e.g., FileInputStream.close(). There's no reason to be calling finalize() when you have a more appropriate method that is intended to be called by you.

Assuming finalizers are similar to their .NET namesake then you only really need to call these when you have resources such as file handles that can leak. Most of the time your objects don't have these references so they don't need to be called.
It's bad to try to collect the garbage because it's not really your garbage. You have told the VM to allocate some memory when you created objects, and the garbage collector is hiding information about those objects. Internally the GC is performing optimisations on the memory allocations it makes. When you manually try to collect the garbage you have no knowledge about what the GC wants to hold onto and get rid of, you are just forcing it's hand. As a result you mess up internal calculations.
If you knew more about what the GC was holding internally then you might be able to make more informed decisions, but then you've missed the benefits of GC.

The real problem with closing OS handles in finalize is that the finalize are executed in no guaranteed order. But if you have handles to the things that block (think e.g. sockets) potentially your code can get into deadlock situation (not trivial at all).
So I'm for explicitly closing handles in a predictable orderly manner. Basically code for dealing with resources should follow the pattern:
SomeStream s = null;
...
try{
s = openStream();
....
s.io();
...
} finally {
if (s != null) {
s.close();
s = null;
}
}
It gets even more complicated if you write your own classes that work via JNI and open handles. You need to make sure handles are closed (released) and that it will happen only once. Frequently overlooked OS handle in Desktop J2SE is Graphics[2D]. Even BufferedImage.getGrpahics() can potentially return you the handle that points into a video driver (actually holding the resource on GPU). If you won't release it yourself and leave it garbage collector to do the work - you may find strange OutOfMemory and alike situation when you ran out of video card mapped bitmaps but still have plenty of memory. In my experience it happens rather frequently in tight loops working with graphics objects (extracting thumbnails, scaling, sharpening you name it).
Basically GC does not take care of programmers responsibility of correct resource management. It only takes care of memory and nothing else. The Stream.finalize calling close() IMHO would be better implemented throwing exception new RuntimeError("garbage collecting the stream that is still open"). It will save hours and days of debugging and cleaning code after the sloppy amateurs left the ends lose.
Happy coding.
Peace.

The GC does a lot of optimization on when to properly finalize things.
So unless you're familiar with how the GC actually works and how it tags generations, manually calling finalize or start GC'ing will probably hurt performance than help.

Avoid finalizers. There is no guarantee that they will be called in a timely fashion. It could take quite a long time before the Memory Management system (i.e., the garbage collector) decides to collect an object with a finalizer.
Many people use finalizers to do things like close socket connections or delete temporary files. By doing so you make your application behaviour unpredictable and tied to when the JVM is going to GC your object. This can lead to "out of memory" scenarios, not due to the Java Heap being exhausted, but rather due to the system running out of handles for a particular resource.
One other thing to keep in mind is that introducing the calls to System.gc() or such hammers may show good results in your environment, but they won't necessarily translate to other systems. Not everyone runs the same JVM, there are many, SUN, IBM J9, BEA JRockit, Harmony, OpenJDK, etc... This JVM all conform to the JCK (those that have been officially tested that is), but have a lot of freedom when it comes to making things fast. GC is one of those areas that everyone invests in heavily. Using a hammer will often times destroy that effort.