I have a problem of memory usage in my application, it start some task that full the memory. I expect that, when the task finish, the gc will clean the memory, but it never happen.
I wish to say that the code is witten very bad (like my English :( ). This makes some extraction from db and it make a massive usage of variables, store they in lists and at the end write all values to a file.
First of all I tried to write values to file frequently, but nothing change. I read that an object became eligible for the gc only if it has no reference to nothing, so, it is possible to see (maybe in debug) the tree reference of an object? In this way i can nullify the parent object at the end of the task. I know that the solution is to re-write the code, but it is very complex and we don't have much effort to do that, so we need a temporary solution.
Thank you
Bye
I am developing a programming contest manager in JAVA.
Concept of Contest Manager:
The main concept of Contest Manager is summarized below. If you have idea about it, you can skip lines before the picture.
Server runs on Judge PC.
All contestants be connected with Judge as client.
Contestants are provided hard copies of problem statements and they write solution on C++. They submit their solution using Contest Manger Software written in JAVA.
Judge have some input data in file and corresponding output data in file for every problem.
When a contestant submits his solution, Judge server runs it against the provided inputs by Judge.
Judge server then matches the output of the contestant with the correct output provided before.
Then Judge server gives a verdict on the basis of the matching result like Accepted, Wrong Answer, Compile Error, Time Limit Exceeded, etc.
Each problem has a predefined time limit. That means the submitted solution must run within a certain time period. (Usually it ranges from 1 second to 15 second)
The verdict of a submitted solution would be visible to all contestants. The picture below would clear the scenario. This is the submission queue and it is visible to all the contestants and judge.
Problem Background:
In the picture, you can see a red marked area where the time elapsed by every submitted solution is written in milliseconds. I could do easily by having the following code:
long start,end;
start = new Date().getTime();
int verdictCode = RunProgram(fileEXE, problem.inputFile, fileSTDOUT, problem.timeLimit);
end = new Date().getTime();
submission.timeElapsed = end - start;
Here, RunProgram function runs the submitted solution (program) and generates output file against an input file. If you need the details of it, ask me later, I would describe.
Main Problem:
However, There is another type of verdict called Memory Limit Exceeded
which is not implemented here. I want to implement that. But getting no idea how to do it. I googled it. Somebody tell about profiling, but I am not getting how to do it properly and Do not know can it serve my purpose or not.
That means, there would be a column named Memory Elapsed like Time Elapsed.
It is possible to do the thing because Online Judges like Codeforces are already showing it. But my question is, Is it possible to do the same in JAVA?
If yes, then how?
If no, then how could you be sure?
Note:
The software has some dependency. It must run on windows platform.
I think you are asking about measuring statistics on native programs written in C++, correct?. You can't measure the memory usage of other programs in the OS with Java, you can only get memory information about the current JVM in Java. To measure memory usage or things like CPU usage of other processes you would need a platform-dependent solution (native code which you run with JNI). Luckily people have already implemented things like this, so that you can use plain Java objects to do what you want without having to write any C/JNI code. Check out Hyperic Sigar library for an easy way to do what you want.
I think you want the Runtime class.
Runtime runtime = Runtime.getRuntime();
System.out.println("Free memory: " + runtime.getFreeMemory() + " / " + rutime.getMaxMemory());
For a project for school I have to program different kind of algorithms. The problem is, I got a working algorithm. But I have to run it several times and after some time it gives me the following errors:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
I know what the error means, but is it possible to let Java search for empty space during the run? I know it uses a lot of space which isn't used at some point. It sets a lot of object to null during the application run and create a lot of new ones, because of this it runs out of memory.
So concrete: is it possible to let the JVM free some space that is set to null? Or free some space in the time the program is running? I know I can set the JVM to more space, but sooner or later I will run to the same problem.
If you need my IDE (in case it is IDE specific) it is Eclipse.
Please google 'garbage collection'. Java is always looking to reuse space from objects that you aren't using. If you run out of memory, you either need to use -Xmx to configure for more memory, or you have to fix your code to retain fewer objects. You may find that a profiler like jvisualvm would help you find wasteful memory usage.
If you're using an Oracle/Sun JVM, I'd recommend that you download Visual VM 1.3.3, install all the plugins, and start it up. It'll show you what's happening in every heap generation, threads, CPU, objects, etc. It can tell you which class is taking up the most heap space.
You'll figure it out quickly if you have data.
I would use a memory profiler to determine where the memory is being used. Setting to null rarely helps. The GC will always run and free as much space as possible before you get an OOME.
Q: "is it possible to let the JVM free some space that is set to null? Or free some space in the time the program is running?"
A: Yes, use a call to System.gc() will do this, but this will not likely solve your problem as the system does this automatically from time to time. You need to find the object that is using all the memory and fix it in your code. Likely a list that is never cleared and only ever added to.
I actually encountered this issue while implementing a particularly complicated algorithm that required a massive data structure. I had to come and post a question on this website. It turned out I had to use a completely different type of object altogether in order to avoid the memory error.
Here is that question.
GC will reclaim 'unused' memory automatically, so yes, it is possible to free some space at runtime, but it's crucial to understand what's classified as possible to be reclaimed.
Basically an object's space can be reclaimed (garbage collected) if the object itself is unreachable - there are no references to it. When you say 'setting space to null' you're most likely removing just one link (reference) to the object by setting it to null. This will allow to reclaim the object only if that was the only link (reference)
Object First= new Object(); //first object
Object Second= new Object(); //second object
Object SecondPrim=Second; //second reference to second object
First=null;
// First memory will be reclaimed (sooner or later)
Second=null;
// there is still a reference to second object via SecondPrim
// second object will not be reclaimed
Hope this helps. As for checking what's exactly going on I would second advice to profile your program.
I have written a program in Java which has 5 threads. In the run() I have a while loop which will loop over and over and this loop will loop a lot of times.
While the program is running, it is gradually eating ram bringing the program to a crawl. Is there anyway I can stop it eating all my ram?
Edit:
Actually just thinking about it, it probably is because the loop is creating lots of objects. Should I = null those objects at the end of the while loop??
If you are creating new objects and saving them in some collection in your loop that would easily fill up memory very quickly.
What you need to do is make sure you aren't saving any extra objects that you don't need.
Also, you should optimize the logic in your threads to consume the least amount of memory.
While the program is running, it is
gradually eating ram bringing the
program to a crawl. Is there anyway I
can stop it eating all my ram?
Sure. Change your code so that it uses less RAM.
Specific to #Michael Borgwardt's answer, try to minimize the creation of new objects. If you can re-use the same object instead of creating new ones every time, you can save some memory there.
But as others have said, without seeing your code, we're just guessing.
You might need a memory analyzer to understand what is "eating RAM". There are many tools available. One commercial tool is JProfiler.
A simple and free tool is to use Sun Profiler supplied with Java VisualVM. If you have JDK 6 installed you probably already have this program installed on your computer. Run the C:\Program Files\Java\jdk1.6.0_16\bin\jvisualvm.exe. Connect to the running Java process and press the Profiler tab and Memory button. You can now see what objects you have and how much memory they are using. Press refresh to update the view.
It's normal for each one of your thread to have an infinite while loop, but simply having the while loops does not use up more RAM.
new Thread(new Runnable() {
public void run() {
try {
while (true) {
// do some work
}
} catch(InterruptedException ex) {}
}
}).start();
You're doing something to consume the RAM, so you must debug your application and find the cause of the increased RAM usage.
Update:
You don't have to make the objects null at the end of the loop... if you're not holding on to a reference of those objects then the garbage collector should clean them up. What are you doing with the objects after you create them? How much RAM are you using up?
This question already has answers here:
How to deal with "java.lang.OutOfMemoryError: Java heap space" error?
(31 answers)
Closed 9 years ago.
I have to serialize around a million items and I get the following exception when I run my code:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at org.girs.TopicParser.dump(TopicParser.java:23)
at org.girs.TopicParser.main(TopicParser.java:59)
How do I handle this?
I know that the official Java answer is "Oh noes! Out of memories! I give in!". This is all rather frustrating for anyone who has programmed in environments where running out of memory is not allowed to be a fatal error (for example, writing an OS, or writing apps for non-protected OSes).
The willingness to surrender is necessary - you can't control every aspect of Java memory allocation, so you can't guarantee that your program will succeed in low-memory conditions. But that doesn't mean you must go down without a fight.
Before fighting, though, you could look for ways to avoid the need. Perhaps you can avoid Java serialization, and instead define your own data format which does not require significant memory allocation to create. Serialization allocates a lot of memory because it keeps a record of objects it has seen before, so that if they occur again it can reference them by number instead of outputting them again (which could lead to an infinite loop). But that's because it needs to be general-purpose: depending on your data structure, you might be able to define some text/binary/XML/whatever representation which can just be written to a stream with very little need to store extra state. Or you might be able to arrange that any extra state you need is stored in the objects all along, not created at serialization time.
If your application does one operation which uses a lot of memory, but mostly uses much less, and especially if that operation is user-initiated, and if you can't find a way to use less memory or make more memory available, then it might be worth catching OutOfMemory. You could recover by reporting to the user that the problem is too big, and inviting them to trim it down and try again. If they've just spend an hour setting up their problem, you do not want to just bail out of the program and lose everything - you want to give them a chance to do something about it. As long as the Error is caught way up the stack, the excess memory will be unreferenced by the time the Error is caught, giving the VM at least a chance to recover. Make sure you catch the error below your regular event-handling code (catching OutOfMemory in regular event handling can result in busy loops, because you try to display a dialog to the user, you're still out of memory, and you catch another Error). Catch it only around the operation which you've identified as the memory-hog, so that OutOfMemoryErrors you can't handle, that come from code other than the memory hog, are not caught.
Even in a non-interactive app, it might make sense to abandon the failed operation, but for the program itself to carry on running, processing further data. This is why web servers manage multiple processes such that if one page request fails for lack of memory, the server itself doesn't fall over. As I said at the top, single-process Java apps can't make any such guarantees, but they can at least be made a bit more robust than the default.
That said, your particular example (serialization) may not be a good candidate for this approach. In particular, the first thing the user might want to do on being told there's a problem is save their work: but if it's serialization which is failing, it may be impossible to save. That's not what you want, so you might have to do some experiments and/or calculations, and manually restrict how many million items your program permits (based on how much memory it is running with), before the point where it tries to serialize.
This is more robust than trying to catch the Error and continue, but unfortunately it's difficult to work out the exact bound, so you would probably have to err on the side of caution.
If the error is occurring during deserialization then you're on much firmer ground: failing to load a file should not be a fatal error in an application if you can possibly avoid it. Catching the Error is more likely to be appropriate.
Whatever you do to handle lack of resources (including letting the Error take down the app), if you care about the consequences then it's really important to test it thoroughly. The difficulty is that you never know exactly what point in your code the problem will occur, so there is usually a very large number of program states which need to be tested.
Ideally, restructure your code to use less memory. For example, perhaps you could stream the output instead of holding the whole thing in memory.
Alternatively, just give the JVM more memory with the -Xmx option.
You should not handle it in code. OutOfMemory should not be caught and handled. Instead start your JVM with a bigger heapspace
java -Xmx512M
should do the trick.
See here for more details
Everyone else has already covered how to give Java more memory, but because "handle" could conceivably mean catch, I'm going to quote what Sun has to say about Errors:
An Error is a subclass of Throwable
that indicates serious problems that a
reasonable application should not try
to catch. Most such errors are
abnormal conditions.
(emphasis mine)
You get an OutOfMemoryError because your program requires more memory than the JVM has available. There is nothing you can specifically do at runtime to help this.
As noted by krosenvold, your application may be making sensible demands for memory but it just so happens that the JVM is not being started with enough (e.g. your app will have a 280MB peak memory footprint but the JVM only starts with 256MB). In this case, increasing the size allocated will solve this.
If you feel that you are supplying adequate memory at start up, then it is possible that your application is either using too much memory transiently, or has a memory leak. In the situation you have posted, it sounds like you are holding references to all of the million items in memory at once, even though potentially you are dealing with them sequentially.
Check what your references are like for items that are "done" - you should deference these as soon as possible so that they can be garbage collected. If you're adding a million items to a collection and then iterating over that collection, for example, you'll need enough memory to store all of those object instances. See if you can instead take one object at a time, serialise it and then discard the reference.
If you're having trouble working this out, posting a pseudo-code snippet would help.
In addition to some of the tips that have been give to you, as review the memory lacks and
also start the JVM with more memory (-Xmx512M).
Looks like you have a OutOfMemoryError cause your TopicParser is reading a line that probably is pretty big (and here is what you should avoid), you can use the FileReader (or, if the encoding is an issue, an InputStreamReader wrapping a FileInputStream). Use its read(char[]) method with a reasonably sized char[] array as a buffer.
Also finally to investigate a little why is the OutOfMemoryError you can use
-XX:+HeapDumpOnOutOfMemoryError
Flag in the JVM to get a dump heap information to disk.
Good luck!
Interesting - you are getting an out of memory on a readline. At a guess, you are reading in a big file without linebreaks.
Instead of using readline to get the stuff out of the file as one single big long string, write stuff that understands the input a bit better, and handles it in chunks.
If you simply must have the whole file in a single big long string ... well, get better at coding. In general, trying to handle mutimegabyte data by stuffing it all into a single array of byte (or whatever) is a good way to lose.
Go have a look at CharacterSequence.
Use the transient keyword to mark fields in the serialized classes which can be generated from existing data.
Implement writeObject and readObject to help with reconstructing transient data.
After you follow the suggestion of increasing heap space (via -Xmx) but sure to use either JConsole or JVisualVM to profile your applications memory usage. Make sure that memory usage does not continuously grow. If so you'll still get the OutOfMemoryException, it'll just take longer.
You can increase the size of the memory java uses with the -Xmx-option, for instance:
java -Xmx512M -jar myapp.jar
Better is to reduce the memory-footprint of your app. You serialize millions of items? Do you need to keep all of them in memory? Or can you release some of them after using them? Try to reduce the used objects.
Start java with a larger value for option -Xmx, for instance -Xmx512m
There's no real way of handling it nicely. Once it happens you are in the unknown territory. You can tell by the name - OutOfMemoryError. And it is described as:
Thrown when
the Java Virtual Machine cannot allocate an object because it is out of
memory, and no more memory could be made available by the garbage
collector
Usually OutOfMemoryError indicates that there is something seriously wrong with the system/approach (and it's hard to point a particular operation that triggered it).
Quite often it has to do with ordinary running out of heapspace. Using the -verbosegc and mentioned earlier -XX:+HeapDumpOnOutOfMemoryError should help.
You can find a nice and concise summary of the problem at javaperformancetuning
Before taking any dangerous, time-consuming or strategic actions, you should establish exactly what in your program is using up so much of the memory. You may think you know the answer, but until you have evidence in front of you, you don't. There's the possibility that memory is being used by something you weren't expecting.
Use a profiler. It doesn't matter which one, there are plenty of them. First find out how much memory is being used up by each object. Second, step though iterations of your serializer, compare memory snapshots and see what objects or data are created.
The answer will most likely be to stream the output rather than building it in memory. But get evidence first.
I have discovered an alternate, respecting all other views that we should not try to catch the memory out of exception, this is what I've learned in recent time.
catch (Throwable ex){
if (!(ex instanceof ThreadDeath))
{
ex.printStackTrace(System.err);
}}
for your reference: OutOfMemoryError
any feedback is welcome.
Avishek Arang