Develop Reference

Java WeakReference is still holding valid reference when referent is no longer valid

I'm confused with this program:
class Point {
private final int x;
private final int y;
}
public class App
{
WeakReference<Point> newPoint() {
Point referent = new Point();
return new WeakReference<Point>(referent); // after return, stack parameter referent is invalid.
}
public static void main( String[] args ) {
App a = new App();
WeakReference<Point> wp = a.newPoint(); // wp is hold valid or invalid reference?
System.out.println(wp.get()); // not null
}
}
I knew that if weak reference is pointing to an object that's no longer alive, its get() should return null. But in my code, seems its still alive.
Where did I get wrong?

I knew that if weak reference is pointing to an object that's no longer alive, its get() should return null. But in my code, seems its still alive.
Your understanding is imprecise, especially where it relies on the idea of aliveness. Reference objects in general and WeakReference objects in particular are not directly concerned with any of the senses of aliveness that I recognize. Rather, they are concerned with reachability.
The API docs for java.lang.Reference#get() (which is not overridden by WeakReference) say this:
Returns this reference object's referent. If this reference object has been cleared, either by the program or by the garbage collector, then this method returns null.
Note well that the condition for get() returning null is rather specific: the reference object has been cleared. This is achieved for a given instance by invoking that instance's clear() method. As the doc indicates, this may be done by the garbage collector or by the application.
Among the key differences between Reference subclasses is the conditions under which the garbage collector will perform such clearing. For WeakReferences, the API docs say:
Suppose that the garbage collector determines at a certain point in time that an object is weakly reachable. At that time it will atomically clear all weak references to that object [...].
Thus, until the garbage collector determines that a given object is (only) weakly reachable, it will not clear weak references to that object. The garbage collector probably does not run at all during the brief run of your small program, and if it did run, it would be surprising for it to be timed correctly to observe the Point in question to be weakly reachable before the reference's get() method is invoked.
You could try forcing a GC run by invoking System.gc() at the appropriate place. I anticipate that doing so will result in the weak reference being cleared. That would be for demonstrative purposes only, however. Generally speaking, you should rely on Java to perform GC when appropriate, not force it.

Does JVM garbage collect objects being referenced by local variables which are no longer used? [duplicate]

This question already has answers here:
Can java finalize an object when it is still in scope?
(2 answers)
Closed 3 years ago.
As far as I know, a method's local variable is located in a stack frame in an executing thread and a reference type of a local variable only has a objects' reference, not the object itself. All of objects in JVM are located in a heap space.
I want to know that objects referenced by local variables in a method being executed are never garbage collected until the end of the method execution. (without using java.lang.ref.WeakReference and SoftReference.)
Are they garbage collected? or never? Is there compiler's optimization to this type of stuff?
(If they are never garbage collected, this means it may be needed to assign null to variables no longer used when executing big methods which take long time.)

As elaborated in Can java finalize an object when it is still in scope?, local variables do not prevent the garbage collection of referenced objects. Or, as this answer puts it, scope is a only a language concept, irrelevant to the garbage collector.
I’ll cite the relevant part of the specification, JLS §12.6.1 again:
A reachable object is any object that can be accessed in any potential continuing computation from any live thread.
Further, I extended the answer’s example to
class A {
static volatile boolean finalized;
Object b = new Object() {
#Override protected void finalize() {
System.out.println(this + " was finalized!");
finalized = true;
}
#Override public String toString() {
return "B#"+Integer.toHexString(hashCode());
}
};
#Override protected void finalize() {
System.out.println(this + " was finalized!");
}
#Override public String toString() {
return super.toString() + " with "+b;
}
public static void main(String[] args) {
A a = new A();
System.out.println("Created " + a);
for(int i = 0; !finalized; i++) {
if (i % 1_000_000 == 0)
System.gc();
}
System.out.println("finalized");
}
}
Created A#59a6e353 with B#6aaa5eb0
B#6aaa5eb0 was finalized!
finalized
A#59a6e353 with B#6aaa5eb0 was finalized!
which demonstrates that even the method with the variable in scope may detect the finalization of the referenced object. Further, being referenced from a heap variable doesn’t necessarily prevent the garbage collection either, as the B object is unreachable, as no continuing computation can access it when the object containing the reference is unreachable too.
It’s worth emphasizing that even using the object does not always prevent its garbage collection. What matters, is whether the object’s memory is needed for the ongoing operation(s) and not every access to an object’s field in source code has to lead to an actual memory access at runtime. The specification states:
Optimizing transformations of a program can be designed that reduce the number of objects that are reachable to be less than those which would naively be considered reachable. […]
Another example of this occurs if the values in an object's fields are stored in registers. The program may then access the registers instead of the object, and never access the object again. This would imply that the object is garbage.
This is not only a theoretical option. As discussed in finalize() called on strongly reachable object in Java 8, it may even happen to objects while a method is invoked on them, or in other words, the this reference may get garbage collected while an instance method is still executing.
The only ways to prevent an objects garbage collection for sure, are synchronization on the object if the finalizer also does synchronization on the object or calling Reference.reachabilityFence(object), a method added in Java 9. The late addition of the fence method demonstrates the impact of the optimizers getting better from version to version on the issue of earlier-than-wanted garbage collection. Of course, the preferred solution is to write code that does not depend on the time of garbage collection at all.

It is not quite true that all of the objects are in heap space; but it is generally true. Java has been extended to have stack-local objects, provided the JVM can detect that the object will live only as long as the stack frame.
Now for the objects on the heap, which have a local reference in a method. While the method is being processed, the stack frame associated with the method run contains the local variable references. As long as the reference can be used (which includes being still in the stack frame) the object will not be garbage collected.
Once the reference has been destroyed, and the object can no longer be reached by the running program (because there's no references that can reach it), then the garbage collector will collect it.

Meaning of ReferenceQueue

I try to understand class ReferenceQueue
It is optional constructor argument for
SoftReference
and
WeakReference
Also it is mandatory argument for PhantomReference.
According information I have read I can write some thesises
a)for PhantomReference method get always returns null
b)
for Phantom references:
1. gc detect that object can be deleted from memory
2. reference to the object puted to the ReferenceQueue
when we invoke clear or link to reference from queue becaom unreachable and gc see that
3. finalize methods invokes
4. free memory
for weak/soft references:
1. gc detect that object can be deleted from memory
2. finalize methods invokes
3. free memory
4. reference to the object puted to the queue
When can I pass second argument to XXXReference constructor?
Which help I can get?
Why PhantomReference has not constructor without ReferenceQueue ?
What the reason to have ReferenceQuee which get methods returns null always?

Maybe, the following program helps a bit:
public class SimpleGCExample {
public static void main(String[] args) throws InterruptedException {
ReferenceQueue<Object> queue=new ReferenceQueue<>();
SimpleGCExample e = new SimpleGCExample();
Reference<Object> pRef=new PhantomReference<>(e, queue),
wRef=new WeakReference<>(e, queue);
e = null;
for(int count=0, collected=0; collected<2; ) {
Reference ref=queue.remove(100);
if(ref==null) {
System.gc();
count++;
}
else {
collected++;
System.out.println((ref==wRef? "weak": "phantom")
+" reference enqueued after "+count+" gc polls");
}
}
}
#Override
protected void finalize() throws Throwable {
System.out.println("finalizing the object in "+Thread.currentThread());
Thread.sleep(100);
System.out.println("done finalizing.");
}
}
On my system, it prints
weak reference enqueued after 1 gc polls
finalizing the object in Thread[Finalizer,8,system]
done finalizing.
phantom reference enqueued after 2 gc polls
or
finalizing the object in Thread[Finalizer,8,system]
weak reference enqueued after 1 gc polls
done finalizing.
phantom reference enqueued after 2 gc polls
The order of the first two messages occasionally differs due to the multi-threading. And sometimes, the phantom reference is reported to be enqueued after three polls, indicating that it took more than the specified 100 milliseconds.
The key point is
soft and weak references are cleared and enqueued before or right when starting finalization
phantom references are enqueued after finalization, assuming that the object has not leaked the finalize method, otherwise they are enqueued after the object has become unreachable again
the presence of a (non-trivial) finalize() method causes the need of at least one additional garbage collecting cycle to detect that the object is unreachable or phantom reachable again
Since more than 99% of all objects don’t need finalization, all JVM vendors are strongly encouraged to detect when finalize() has not been overridden or is “trivial”, i.e. an empty method or a sole super.finalize() call. In these cases, the finalization step should be elided. You can easily check that this optimization happens in your JVM, by removing the finalize() method in the above example. Then it prints
weak reference enqueued after 1 gc polls
phantom reference enqueued after 1 gc polls
Since both are enqueued at once and retrieved in an arbitrary order, the order of the two messages may differ. But they are always both enqueued after one gc cycle.
It’s worth noting that the fact, that phantom references are not automatically cleared, implies that it takes another garbage collection cycle until the object’s memory really can be reused, so the above example requires at least three cycles with the non-trivial finalize() method and two without. Java 9 is going to change that, clearing phantom references automatically, so in the above example it will take two cycles with finalization and one without until the memory really can be reclaimed. Well, to be precise, in this simple example the object’s memory will never be reclaimed as the program terminates before that can happen.
The code above also demonstrates one of the intended use cases of the Reference API. We can use it to detect when an object’s reachability changed within code under our full control, e.g. using a loop within the main method. In contrast, finalize() may be called by a different, unspecified thread at an arbitrary time. The example also shows that you can draw information from the reference object without needing the get() method.
Practical applications often use subclasses of the reference classes to add more information to them. This is what happens with WeakHashMap.Entry which extends WeakReference and remembers the hash code and value. The cleanup can be done within the normal map operations, not needing any thread synchronization. This would not be possible with a finalize() method, besides the fact that the map implementation can’t push a finalize() method into the class of the keys.
This is meant with the “more flexible than finalization” term.
The WeakHashMap demonstrates how the get() method can be useful. As long as the key hasn’t been collected, it will be reported as being in the map and can be retrieved when iterating over all keys or entries.
The PhantomReference.get() method has been overwritten to always return null to prevent that an application can resurrect the referent of an enqueued reference. This is a direct consequence of the “phantom references are not automatically cleared” rule. That rule itself is questionable and it’s original intention lies in the dark. While the rule is about to be changed in the next Java version, I’m afraid that get() will continue to always return null to be backwards compatible.

1) When can I pass second argument to XXXReference constructor?
You can do it whenever you like. You should do it whenever you need the references to be processed as they are being broken.
2) Which help I can get?
I don't understand this questiom
3) Why PhantomReference has not constructor without ReferenceQueue ?
The purpose of PhantomReference is to be a more flexible alternative to regular finalization. However, in order for that to work, the references must be enqueued for the finalization-replacing code to work. (A PhantomReference that was not enqueued could not be processed.)
By contrast SoftReference and WeakReference objects are often useful without being enqueued.
4) What the reason to have ReferenceQueue which get methods returns null always?
The ReferenceQueue API doesn't have a get() method, so I guess you are talking about the PhantomReference API. The reason that there is a get() method is for compatibility with the superclass. The reason that get() is defined to return null is as follows:
"In order to ensure that a reclaimable object remains so, the referent of a phantom reference may not be retrieved: The get method of a phantom reference always returns null."
(See the javadoc.)
In other words, it is done to make it impossible to "resurrect" the referent.
UPDATE
In fact, all of the Reference classes will clear the referent before enqueuing a Reference. (Actually, the GC itself does this directly.) Application code that pulls references from a ReferenceQueue cannot use the get() method to identify the (now deleted!) referent. They must do it some other way; e.g. based on the identity of the Reference object.
What distinguishes a phantom reference is that the get() method always returns null. (So the explanation in the javadoc is ... unconvincing.)

Java: Guaranteed garbage collection using jlibs

The following snippet from RuntimeUtil.java from jlibs guarantees GC that garbage collection is done.
Since, it also uses System.gc(), i dont understand how can they guarantee that it will happen 100% of the times.
Following is the snippet:
/**
* This method guarantees that garbage collection is
* done unlike <code>{#link System#gc()}</code>
*/
public static void gc(){
Object obj = new Object();
WeakReference ref = new WeakReference<Object>(obj);
obj = null;
while(ref.get()!=null)
System.gc();
}

Its related with Strong Reference & Weak Reference with respect to Garbae Collection.
A strong reference is an ordinary Java reference, the kind you use every day.
If an object is reachable via a chain of strong references (strongly reachable), it is not eligible for garbage collection. As you don't want the garbage collector destroying objects you're working on, this is normally exactly what you want.
A weak reference, simply put, is a reference that isn't strong enough to force an object to remain in memory. Weak references allow you to leverage the garbage collector's ability to determine reachability
The gc() method in the pointed class works on this concept.
/**
* This method guarantees that garbage collection is
* done unlike <code>{#link System#gc()}</code>
*/
public static void gc(){
Object obj = new Object();
WeakReference ref = new WeakReference<Object>(obj);
obj = null;
while(ref.get()!=null)
System.gc();
}
Once a WeakReference starts returning null, the object it pointed to has become garbage and the WeakReference object is pretty much useless. This generally means that some sort of cleanup is required;
That's why they guarantee that it will happen 100% of the times.

From what I can see it should work as they
1) create an object
2) get a "weak" reference to it
3) null the reference to it, in order to mark it for garbage collection
4) and wait with the while loop until it actually disappears

A single call to System.gc() does not guarantee all objects that are eligible for garbage collection are re-claimed. See How to cause soft references to be cleared in Java?
In this case, the garbage collection is run many times till a an weak object ref is returning null. I doubt efficacy of this approach. There will be many other objects that may be garbage collected. To me it (jlib) does not have any thing to learn from.
For example the code in the library written in 2009. (extracted)
/**
* This method guarantees that garbage colleciton is
* done after JVM shutdown is initialized
*/
public static void gcOnExit(){
Runtime.getRuntime().addShutdownHook(new Thread(){
#Override
public void run(){
gc();
}
});
}
Why to you need gc when shutdown is called? The library quoted in question is pure garbage.

Long lived Java WeakReferences

I am currently trying to diagnose a slow memory leak in my application. The facts I have so far are as follows.
I have a heap dump from a 4 day run of the application.
This heap dump contains ~800 WeakReference objects which point to objects (all of the same type, which I will call Foo for the purposes of this question) retaining 40mb of memory.
Eclipse Memory Analysis Tool shows that each of the Foo objects referred to by these WeakReferences is not referred to by any other objects. My expectation is that this should make these Foo objects Weakly Reachable and thus they should be collected at the next GC.
Each of these Foo objects has a timestamp which shows that they were allocated over the course of the 4 day run. I also have logs during this time which confirm that Garbage Collection was happening.
A huge number of Foo objects are being created by my application and only a very small fraction of them are ending up in this state within the heap dump. This suggests to me that the root cause is some sort of race condition.
My application uses JNI to call through to a native library. The JNI code calls NewGlobalRef 4 times during start of day initialisation to get references to Java classes which it uses.
What could possibly cause these Foo classes to not be collected despite only being referenced by WeakReferences (according to Eclipse Memory Analyser Tool)?
EDIT1:
#mindas
The WeakReference I am using is equivalent to the following example code.
public class FooWeakRef extends WeakReference<Foo>
{
public long longA;
public long longB;
public String stringA;
public FooWeakRef(Foo xiObject, ReferenceQueue<Foo> xiQueue)
{
super(xiObject, xiQueue);
}
}
Foo does not have a finalizer and any finalizer would not be a consideration so long as the WeakRefs have not been cleared. An object is not finalizable when it is weakly reachable. See this page for details.
#kasten The weakreferences are cleared before the object is finalizable. My heap dump shows that this has not happened.
#jarnbjo I refer to the WeakReference Javadoc:
"Suppose that the garbage collector determines at a certain point in time that an object is weakly reachable. At that time it will atomically clear all weak references to that object and all weak references to any other weakly-reachable objects from which that object is reachable through a chain of strong and soft references."
This suggests to me that the GC should be detecting the fact that my Foo objects are "Weakly reachable" and "At that time" clearing the weak references.
EDIT 2
#j flemm - I know that 40mb doesn't sound like much but I am worried that 40mb in 4 days means 4000mb in 100 days. All of the docs I have read suggest that objects which are weakly reachable should not hang around for several days. I am therefore interested in any other explanations about how an object could be strongly referenced without the reference showing up in a heap dump.
I am going to try allocating some large objects when some of these dangling Foo objects are present and see whether the JVM collects them. However, this test will take a couple of days to setup and complete.
EDIT 3
#jarnbjo - I understand that I have no guarantee about when the JDK will notice that an object is weakly reachable. However, I would expect that an application under heavy load for 4 days would provide enough opportunities for the GC to notice that my objects are weakly reachable. After 4 days I am strongly suspicious that the remaining weakly references objects have been leaked somehow.
EDIT 4
#j flemm - Thats really interesting! Just to clarify, are you saying that GC is happening on your app and is not clearing Soft/Weak refs? Can you give me any more details about what JVM + GC Config you are using? My app is using a memory bar at 80% of the heap to trigger GC. I was assuming that any GC of the old gen would clear Weak refs. Are you suggesting that a GC only collects Weak refs once the memory usage is above a higher threshold? Is this higher limit configurable?
EDIT 5
#j flemm - Your comment about clearing out WeakRefs before SoftRefs is consistent with the Javadoc which states:
SoftRef: "Suppose that the garbage collector determines at a certain point in time that an object is softly reachable. At that time it may choose to clear atomically all soft references to that object and all soft references to any other softly-reachable objects from which that object is reachable through a chain of strong references. At the same time or at some later time it will enqueue those newly-cleared soft references that are registered with reference queues."
WeakRef: "Suppose that the garbage collector determines at a certain point in time that an object is weakly reachable. At that time it will atomically clear all weak references to that object and all weak references to any other weakly-reachable objects from which that object is reachable through a chain of strong and soft references. At the same time it will declare all of the formerly weakly-reachable objects to be finalizable. At the same time or at some later time it will enqueue those newly-cleared weak references that are registered with reference queues."
For clarity, are you saying that the Garbage Collector runs when your app has more than 50% free memory and in this case it does not clear WeakRefs? Why would the GC run at all when your app has >50% free memory? I think your app is probably just generating a very low amount of garbage and when the collector runs it is clearing WeakRefs but not SoftRefs.
EDIT 6
#j flemm - The other possible explanation for your app's behaviour is that the young gen is being collected but that your Weak and Soft refs are all in the old gen and are only cleared when the old gen is being collected. For my app I have stats showing that the old gen is being collected which should mean that WeakRefs get cleared.
EDIT 7
I am starting a bounty on this question. I am looking for any plausible explanations for how WeakRefs could fail to be cleared while GC is happening. If the answer is that this is impossible I would ideally like to be pointed at the appropriate bits of OpenJDK which show WeakRefs being cleared as soon as an object is determined to be weakly reachable and that weak reachability is resolved every time GC runs.

I have finally got round to checking the Hotspot JVM source code and found the following code.
In referenceProcessor.cpp:
void ReferenceProcessor::process_discovered_references(
BoolObjectClosure* is_alive,
OopClosure* keep_alive,
VoidClosure* complete_gc,
AbstractRefProcTaskExecutor* task_executor) {
NOT_PRODUCT(verify_ok_to_handle_reflists());
assert(!enqueuing_is_done(), "If here enqueuing should not be complete");
// Stop treating discovered references specially.
disable_discovery();
bool trace_time = PrintGCDetails && PrintReferenceGC;
// Soft references
{
TraceTime tt("SoftReference", trace_time, false, gclog_or_tty);
process_discovered_reflist(_discoveredSoftRefs, _current_soft_ref_policy, true,
is_alive, keep_alive, complete_gc, task_executor);
}
update_soft_ref_master_clock();
// Weak references
{
TraceTime tt("WeakReference", trace_time, false, gclog_or_tty);
process_discovered_reflist(_discoveredWeakRefs, NULL, true,
is_alive, keep_alive, complete_gc, task_executor);
}
The function process_discovered_reflist has the following signature:
void
ReferenceProcessor::process_discovered_reflist(
DiscoveredList refs_lists[],
ReferencePolicy* policy,
bool clear_referent,
BoolObjectClosure* is_alive,
OopClosure* keep_alive,
VoidClosure* complete_gc,
AbstractRefProcTaskExecutor* task_executor)
This shows that WeakRefs are being unconditionally cleared by ReferenceProcessor::process_discovered_references.
Searching the Hotspot code for process_discovered_reference shows that the CMS collector (which is what I am using) calls this method from the following call stack.
CMSCollector::refProcessingWork
CMSCollector::checkpointRootsFinalWork
CMSCollector::checkpointRootsFinal
This call stack looks like it is invoked every time a CMS collection is run.
Assuming this is true, the only explanation for a long lived weakly referenced object would be either a subtle JVM bug or if the GC had not been run.

You might want to check if you have leaked classloader issue. More on this topic you could find in this blog post

You need to clarify on what is the link between Foo and WeakReference. The case
class Wrapper<T> extends WeakReference<T> {
private final T referent;
public Wrapper(T referent) {
super(t);
this.referent = referent;
}
}
is very different from just
class Wrapper<T> extends WeakReferece<T> {
public Wrapper(T referent) {
super(t);
}
}
or its inlined version, WeakReference<Foo> wr = new WeakReference<Foo>(foo).
So I assume your case is not like I described in my first code snippet.
As you have said you are working with JNI, you might want to check if you have any unsafe finalizers. Every finalizer should have finally block calling super.finalize() and it's easy to slip.
You probably need to tell us more about the nature of your objects to offer better ideas.

Try SoftReference instead. Javadoc says: All soft references to softly-reachable objects are guaranteed to have been cleared before the virtual machine throws an OutOfMemoryError.
WeakReference doesn't have such guarantees, which makes them more suitable for caches, but sometimes SoftReferences are better.

#iirekm No: WeakReferences are 'weaker' than SoftReferences, meaning that a WeakReference will always be garbage collected before a SoftReference.
More info in this post: Understanding Java's Reference classes: SoftReference, WeakReference, and PhantomReference
Edit: (after reading comments) Yes surely Weak References are 'Weaker' than SoftReferences, typo. :S
Here's some use cases to throw further light on the subject:
SoftReference: In-memory cache (Object stays alive until VM deems that there's not enough heap mem)
WeakReference: Auto-clearing Listeners (Object should be cleared on next GC cycle after deemed being Weakly reachable)
PhantomReference: Avoiding out-of-memory errors when handling unusually large objects (When scheduled in reference queue, we know that host object is to be cleared, safe to allocate another large object). Think of it as a finalize() alternative, without the ability to bring dead objects back to life (as you potentially could with finalize)
This being said, nothing prevents the VM (please correct me if I'm wrong) to let the Weakly reachable objects stay alive as long as it is not running out of memory (as in the orig. author's case).
This is the best resource I could find on the subject: http://www.pawlan.com/monica/articles/refobjs/
Edit 2: Added "to be" in front of cleared in PhantomRef

I am not acquainted with Java, but you may be using a generational garbage collector, which will keep your Foo and FooWeakRef objects alone (not collected) as long as
they passed in an older generation
there is enough memory to allocate new objects in younger generations
Does the log that indicates that garbage collection occurred discriminates between major and minor collections?

For non-believers who claim that weak references are cleared before soft references:
import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.SoftReference;
import java.lang.ref.WeakReference;
import java.util.HashMap;
import java.util.Map;
public class Test {
/**
* #param args
*/
public static void main(String[] args) {
ReferenceQueue<Object> q = new ReferenceQueue<Object>();
Map<Reference<?>, String> referenceToId = new HashMap<Reference<?>, String>();
for(int i=0; i<100; ++i) {
Object obj = new byte [10*1024*1024]; // 10M
SoftReference<Object> sr = new SoftReference<Object>(obj, q);
referenceToId.put(sr, "soft:"+i);
WeakReference<Object> wr = new WeakReference<Object>(obj, q);
referenceToId.put(wr, "weak:"+i);
for(;;){
Reference<?> ref = q.poll();
if(ref == null) {
break;
}
System.out.println("cleared reference " + referenceToId.get(ref) + ", value=" + ref.get());
}
}
}
}
If your run it with either -client or -server, you'll see that soft references are always cleared before weak references, which also agrees with Javadoc: http://download.oracle.com/javase/1.4.2/docs/api/java/lang/ref/package-summary.html#reachability
Typically soft/weak references are used in connection with Maps to make kinds of caches. If keys in your Map are compared with == operator, (or unoverriden .equals from Object), then it's best to use Map which operates on SoftReference keys (eg from Apache Commons) - when the object 'disappears' no other object will ever be equal in the '==' sense to the old one. If keys of your Map are compared with advanced .equals() operator, like String or Date, many other objects may match to the 'disappearing' one, so it's better to use the standard WeakHashMap.