Develop Reference

Java memory usage and object not eligible for gc

I have a problem of memory usage in my application, it start some task that full the memory. I expect that, when the task finish, the gc will clean the memory, but it never happen.
I wish to say that the code is witten very bad (like my English :( ). This makes some extraction from db and it make a massive usage of variables, store they in lists and at the end write all values to a file.
First of all I tried to write values to file frequently, but nothing change. I read that an object became eligible for the gc only if it has no reference to nothing, so, it is possible to see (maybe in debug) the tree reference of an object? In this way i can nullify the parent object at the end of the task. I know that the solution is to re-write the code, but it is very complex and we don't have much effort to do that, so we need a temporary solution.
Thank you
Bye

OutOfMemoryError:GC Overhead limit exceeded

In one of our java application we have got
OutOfMemoryError:GC Overhead limit exceeded.
We have used HashMaps in someplaces for storing some data.From logs we can I identify that its reproducing at the same place.
I wanted to ask if Garbage Collector spends more time in clearing up the hashmaps?
Upon looking at the code( i cant share here ), I have found that that there is a Hashmap created like
Hashmap topo = new HashMap();
but this hashmap is never used.
Is this a kind of memory leak in my application ?
If this Hashmap is created inside a method which is doing some processing and it is not used elsewhere also this method is accessed my multiple threads say 20 .Then in such a case would it impact,creating Hashmap as above, Garbage collector to spend more time in recovering heap and throw OOME.
Please let me know if you need some more details.

n one of our java application we have got OutOfMemoryError:GC Overhead
limit exceeded. We have used HashMaps in someplaces for storing some
data.From logs we can I identify that its reproducing at the same
place.
If the Hashmap is simply ever building and most likely marked as static, which means you keep adding things to this hashmap and never delete. Then one fine day it will lead to OutOfMemoryError.
I wanted to ask if Garbage Collector spends more time in clearing up
the hashmaps?
Garbage collector spends time on the objects which are not referenced, weakly referenced, soft referenced. Wherever it find such objects, depending on the need it will clear them.
Upon looking at the code( i cant share here ), I have found that that there is a Hashmap
created like Hashmap topo = new HashMap(); , but this hashmap is never used. Is this a
kind of memory leak in my application ?
if this Hashmap is created inside a method which is doing some
processing and it is not used elsewhere also this method is accessed
my multiple threads say 20 . Then in such a case would it
impact,creating Hashmap as above, Garbage collector to spend more time
in recovering heap and throw OOME.
If it is hashmap local to a methid, and the method exits after doing some processing, then it should be garbage collected as soon as method exits. As the hashmap is local to the method, so each thread will have a separate copy of this map and once thread finishes the method execution, map is eligible for GC.

You need to look for long-lifetime objects & structures, which might be the actual problem, rather than wildly grasping at some clueless manager's idea of potential problem.
See:
How to find memory leaks using visualvm
How to find a Java Memory Leak
Look out especially for static/ or application-lifetime Maps or Lists, which are added to during the lifetime rather than just at initialization. It will most likely be one, or several, of these that are accumulating.
Note also that inner classes (Listeners, Observers) can capture references to their containing scope & prevent these from being GC'ed indefinitely.

Please let me know if you need some more details.
You need some more details. You need to profile your application to see what objects are consuming the heap space.
Then, if some of the sizeable objects are no longer actually being used by your application, you have a memory leak. Look at the references to these objects to find out why they're still being held in memory when they're no longer useful, and then modify your code to no longer hold these references.
Alternatively, you may find that all of the objects in memory are what you would expect as your working set. Then either you need to increase the heap size, or refactor your application to work with a smaller working set (e.g. streaming events one at a time rather than reading an entire list; storing the last seesion details in the database rather than memory; etc.).

How to keep some data as long as possible, but allow the VM to free & reload them at need?

In my application, I would like to load some amount of data into memory when first needed and keep it there in case another part of the application wants to use it. The same data would be accessed from a couple of different Activity'es, but by far not all the user could interact with. So, when not working with the relevant part of my application, I would like Android to feel free to discard the data, reloading them again at need. Note that it is unpredictable for me what the user will do, so I want Android to free the data only if hasn't been used for some time. What is a good approach to doing this?
I thought of creating a class that would be only used statically, loading the data in its static initialisation block. However, I am not sure if Dalvik would ever discard any static data stored this way. I have read something on class loaders but I have no idea what loader is used in loading my class and how it could potentially become discarded. Perhaps someone does...?
Another way I came up with is using weak reference to keep an instance of the data-holding class (non-static, obviously) but here I am afraid that the GC could decide it's useless when no Activity is currently actively operating it, even when memory is no concern at that moment. (In that case, I would like to keep the data loaded.)
The loading of my data is costly. I want, if possible, to destroy it only when the system is running out of memory or when the application exits.

It sounds like SoftReferences are what you need. These are cleared at the garbage collector's discretion when it detects that there is a memory shortfall.
If you read the class javadoc, it gives some hints about how to prevent recently used cache entries from being reclaimed.
For the record, classloaders won't help you manage instances of a class. But making the cache a static should allow the cached objects to be discarded if the cache class gets unloaded.
FOLLOWUP
My data is a solid block that would be represented by a single object.
This rather changes things. If you have a single object to cache, then LRU makes no sense. Basically it sounds like you want to hang onto the object as long as possible ... without trigger OOMEs by hanging onto it too long. This is kind of hard. Indeed, doing a perfect job is going to entail correctly predicting what the user is going to do ... which is clearly impossible.
Possibly the best strategy is to make use of the reference enqueing mechanism, and implement the queue processor to make an "intelligent" choice between letting the object die or recreating the soft link. The "intelligence" might entail looking at how much free memory there is, and / or how long it was since the object was last used. But beware!! If you get this wrong you can cause OOMEs or cause the platform to spend lots of time thrashing the garbage collector.
If I set up the cache to hold 1 object, it would be equivalent to a hard reference, wouldn't it?
Nope. If you use a SoftReference the GC will break the reference if it is running out of memory.

You can use SoftReferences. Take a look at:
http://docs.oracle.com/javase/6/docs/api/java/lang/ref/SoftReference.html
With SoftReferences you can achieve what you need:
" I want, if possible, to destroy it only when the system is running out of memory or when the application exits."

Take a look at
SoftReference gets garbage collected too early
You can also look into LruCache if your looking to cache some data in memory your app.
http://developer.android.com/reference/android/support/v4/util/LruCache.html
For a longer lived disk based cache take a look at Android Objects Cache
You can find the DiskLruCache source at https://github.com/JakeWharton/DiskLruCache/

How to solve OutOfMemoryError?

For a project for school I have to program different kind of algorithms. The problem is, I got a working algorithm. But I have to run it several times and after some time it gives me the following errors:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
I know what the error means, but is it possible to let Java search for empty space during the run? I know it uses a lot of space which isn't used at some point. It sets a lot of object to null during the application run and create a lot of new ones, because of this it runs out of memory.
So concrete: is it possible to let the JVM free some space that is set to null? Or free some space in the time the program is running? I know I can set the JVM to more space, but sooner or later I will run to the same problem.
If you need my IDE (in case it is IDE specific) it is Eclipse.

Please google 'garbage collection'. Java is always looking to reuse space from objects that you aren't using. If you run out of memory, you either need to use -Xmx to configure for more memory, or you have to fix your code to retain fewer objects. You may find that a profiler like jvisualvm would help you find wasteful memory usage.

If you're using an Oracle/Sun JVM, I'd recommend that you download Visual VM 1.3.3, install all the plugins, and start it up. It'll show you what's happening in every heap generation, threads, CPU, objects, etc. It can tell you which class is taking up the most heap space.
You'll figure it out quickly if you have data.

I would use a memory profiler to determine where the memory is being used. Setting to null rarely helps. The GC will always run and free as much space as possible before you get an OOME.

Q: "is it possible to let the JVM free some space that is set to null? Or free some space in the time the program is running?"
A: Yes, use a call to System.gc() will do this, but this will not likely solve your problem as the system does this automatically from time to time. You need to find the object that is using all the memory and fix it in your code. Likely a list that is never cleared and only ever added to.
I actually encountered this issue while implementing a particularly complicated algorithm that required a massive data structure. I had to come and post a question on this website. It turned out I had to use a completely different type of object altogether in order to avoid the memory error.
Here is that question.

GC will reclaim 'unused' memory automatically, so yes, it is possible to free some space at runtime, but it's crucial to understand what's classified as possible to be reclaimed.
Basically an object's space can be reclaimed (garbage collected) if the object itself is unreachable - there are no references to it. When you say 'setting space to null' you're most likely removing just one link (reference) to the object by setting it to null. This will allow to reclaim the object only if that was the only link (reference)
Object First= new Object(); //first object
Object Second= new Object(); //second object
Object SecondPrim=Second; //second reference to second object
First=null;
// First memory will be reclaimed (sooner or later)
Second=null;
// there is still a reference to second object via SecondPrim
// second object will not be reclaimed
Hope this helps. As for checking what's exactly going on I would second advice to profile your program.

How to handle OutOfMemoryError in Java? [duplicate]

This question already has answers here:
How to deal with "java.lang.OutOfMemoryError: Java heap space" error?
(31 answers)
Closed 9 years ago.
I have to serialize around a million items and I get the following exception when I run my code:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at org.girs.TopicParser.dump(TopicParser.java:23)
at org.girs.TopicParser.main(TopicParser.java:59)
How do I handle this?

I know that the official Java answer is "Oh noes! Out of memories! I give in!". This is all rather frustrating for anyone who has programmed in environments where running out of memory is not allowed to be a fatal error (for example, writing an OS, or writing apps for non-protected OSes).
The willingness to surrender is necessary - you can't control every aspect of Java memory allocation, so you can't guarantee that your program will succeed in low-memory conditions. But that doesn't mean you must go down without a fight.
Before fighting, though, you could look for ways to avoid the need. Perhaps you can avoid Java serialization, and instead define your own data format which does not require significant memory allocation to create. Serialization allocates a lot of memory because it keeps a record of objects it has seen before, so that if they occur again it can reference them by number instead of outputting them again (which could lead to an infinite loop). But that's because it needs to be general-purpose: depending on your data structure, you might be able to define some text/binary/XML/whatever representation which can just be written to a stream with very little need to store extra state. Or you might be able to arrange that any extra state you need is stored in the objects all along, not created at serialization time.
If your application does one operation which uses a lot of memory, but mostly uses much less, and especially if that operation is user-initiated, and if you can't find a way to use less memory or make more memory available, then it might be worth catching OutOfMemory. You could recover by reporting to the user that the problem is too big, and inviting them to trim it down and try again. If they've just spend an hour setting up their problem, you do not want to just bail out of the program and lose everything - you want to give them a chance to do something about it. As long as the Error is caught way up the stack, the excess memory will be unreferenced by the time the Error is caught, giving the VM at least a chance to recover. Make sure you catch the error below your regular event-handling code (catching OutOfMemory in regular event handling can result in busy loops, because you try to display a dialog to the user, you're still out of memory, and you catch another Error). Catch it only around the operation which you've identified as the memory-hog, so that OutOfMemoryErrors you can't handle, that come from code other than the memory hog, are not caught.
Even in a non-interactive app, it might make sense to abandon the failed operation, but for the program itself to carry on running, processing further data. This is why web servers manage multiple processes such that if one page request fails for lack of memory, the server itself doesn't fall over. As I said at the top, single-process Java apps can't make any such guarantees, but they can at least be made a bit more robust than the default.
That said, your particular example (serialization) may not be a good candidate for this approach. In particular, the first thing the user might want to do on being told there's a problem is save their work: but if it's serialization which is failing, it may be impossible to save. That's not what you want, so you might have to do some experiments and/or calculations, and manually restrict how many million items your program permits (based on how much memory it is running with), before the point where it tries to serialize.
This is more robust than trying to catch the Error and continue, but unfortunately it's difficult to work out the exact bound, so you would probably have to err on the side of caution.
If the error is occurring during deserialization then you're on much firmer ground: failing to load a file should not be a fatal error in an application if you can possibly avoid it. Catching the Error is more likely to be appropriate.
Whatever you do to handle lack of resources (including letting the Error take down the app), if you care about the consequences then it's really important to test it thoroughly. The difficulty is that you never know exactly what point in your code the problem will occur, so there is usually a very large number of program states which need to be tested.

Ideally, restructure your code to use less memory. For example, perhaps you could stream the output instead of holding the whole thing in memory.
Alternatively, just give the JVM more memory with the -Xmx option.

You should not handle it in code. OutOfMemory should not be caught and handled. Instead start your JVM with a bigger heapspace
java -Xmx512M
should do the trick.
See here for more details

Everyone else has already covered how to give Java more memory, but because "handle" could conceivably mean catch, I'm going to quote what Sun has to say about Errors:
An Error is a subclass of Throwable
that indicates serious problems that a
reasonable application should not try
to catch. Most such errors are
abnormal conditions.
(emphasis mine)

You get an OutOfMemoryError because your program requires more memory than the JVM has available. There is nothing you can specifically do at runtime to help this.
As noted by krosenvold, your application may be making sensible demands for memory but it just so happens that the JVM is not being started with enough (e.g. your app will have a 280MB peak memory footprint but the JVM only starts with 256MB). In this case, increasing the size allocated will solve this.
If you feel that you are supplying adequate memory at start up, then it is possible that your application is either using too much memory transiently, or has a memory leak. In the situation you have posted, it sounds like you are holding references to all of the million items in memory at once, even though potentially you are dealing with them sequentially.
Check what your references are like for items that are "done" - you should deference these as soon as possible so that they can be garbage collected. If you're adding a million items to a collection and then iterating over that collection, for example, you'll need enough memory to store all of those object instances. See if you can instead take one object at a time, serialise it and then discard the reference.
If you're having trouble working this out, posting a pseudo-code snippet would help.

In addition to some of the tips that have been give to you, as review the memory lacks and
also start the JVM with more memory (-Xmx512M).
Looks like you have a OutOfMemoryError cause your TopicParser is reading a line that probably is pretty big (and here is what you should avoid), you can use the FileReader (or, if the encoding is an issue, an InputStreamReader wrapping a FileInputStream). Use its read(char[]) method with a reasonably sized char[] array as a buffer.
Also finally to investigate a little why is the OutOfMemoryError you can use
-XX:+HeapDumpOnOutOfMemoryError
Flag in the JVM to get a dump heap information to disk.
Good luck!

Interesting - you are getting an out of memory on a readline. At a guess, you are reading in a big file without linebreaks.
Instead of using readline to get the stuff out of the file as one single big long string, write stuff that understands the input a bit better, and handles it in chunks.
If you simply must have the whole file in a single big long string ... well, get better at coding. In general, trying to handle mutimegabyte data by stuffing it all into a single array of byte (or whatever) is a good way to lose.
Go have a look at CharacterSequence.

Use the transient keyword to mark fields in the serialized classes which can be generated from existing data.
Implement writeObject and readObject to help with reconstructing transient data.

After you follow the suggestion of increasing heap space (via -Xmx) but sure to use either JConsole or JVisualVM to profile your applications memory usage. Make sure that memory usage does not continuously grow. If so you'll still get the OutOfMemoryException, it'll just take longer.

You can increase the size of the memory java uses with the -Xmx-option, for instance:
java -Xmx512M -jar myapp.jar
Better is to reduce the memory-footprint of your app. You serialize millions of items? Do you need to keep all of them in memory? Or can you release some of them after using them? Try to reduce the used objects.

Start java with a larger value for option -Xmx, for instance -Xmx512m

There's no real way of handling it nicely. Once it happens you are in the unknown territory. You can tell by the name - OutOfMemoryError. And it is described as:
Thrown when
the Java Virtual Machine cannot allocate an object because it is out of
memory, and no more memory could be made available by the garbage
collector
Usually OutOfMemoryError indicates that there is something seriously wrong with the system/approach (and it's hard to point a particular operation that triggered it).
Quite often it has to do with ordinary running out of heapspace. Using the -verbosegc and mentioned earlier -XX:+HeapDumpOnOutOfMemoryError should help.
You can find a nice and concise summary of the problem at javaperformancetuning

Before taking any dangerous, time-consuming or strategic actions, you should establish exactly what in your program is using up so much of the memory. You may think you know the answer, but until you have evidence in front of you, you don't. There's the possibility that memory is being used by something you weren't expecting.
Use a profiler. It doesn't matter which one, there are plenty of them. First find out how much memory is being used up by each object. Second, step though iterations of your serializer, compare memory snapshots and see what objects or data are created.
The answer will most likely be to stream the output rather than building it in memory. But get evidence first.

I have discovered an alternate, respecting all other views that we should not try to catch the memory out of exception, this is what I've learned in recent time.
catch (Throwable ex){
if (!(ex instanceof ThreadDeath))
{
ex.printStackTrace(System.err);
}}
for your reference: OutOfMemoryError
any feedback is welcome.
Avishek Arang