Develop Reference

Simple CLI Java programme keeps eating up more and more memory

I've wrote this test programme.
import java.io.Console;
public class ConsoleTest{
public static void main(String[] args) {
// TODO Auto-generated method stub
Console console = System.console();
System.out.println("ConsoleTest initialising... \n Enter command : ");
String line = "";
if(console!=null){
while ( (line = console.readLine())!= null ){
System.gc();
System.out.println("your input : " + line);
if("exit".equalsIgnoreCase(line)){
System.out.println("Bye ~");
System.exit(0);
}else{
System.out.println("Enter command : ");
}
}
}else{
System.out.println("No console found");
}
}
}
I saw heap memory usage constantly going up even while I was writing this question, doing nothing to the programme.
Also, everytime I click on the Perform GC button, windows taskmanager shows that the programme is using a bit MORE memory.
Is there something wrong with the code ? or is this a normal behaviour ?
EDIT
Later I found that the memory heap usage graph fluctuated regularly on a longer time scale. I don't know why it does that but that's another matter I suppose.
If I could ask one more thing, I saw a huge difference between heap usage in Java Monitor and memory usage shown in Windows taskmanager.
Heap usage in Java Monitor showed about 300MB for Eclipse process.
In Windows taskmanager, 900MB - ish.
Is this difference because the heap usage did not include stack memory usage ?

The amount of memory you're talking about is negligible. The JVM isn't bothering to deal with releasing the small amount. If you want to see more interesting behavior, allocate and then drop references to large blocks before asking for garbage collection.

I saw heap memory usage constantly going up even while I was writing this question, doing nothing to the programme.
It is most likely the monitoring agent that is causing this spontaneous increase in memory usage. In other words, it is caused by your monitoring.
Also, everytime I click on the Perform GC button, windows taskmanager shows that the programme is using a bit MORE memory.
It is possibly the same issue. Other possibilities are:
clicking the GC button is not collecting all heaps, and/or
the GC is not giving back the reclaimed memory to the operating system.
Is there something wrong with the code? or is this a normal behaviour?
This is normal behaviour.
The only thing wrong with your code is that it is calling System.gc(). This is a bad idea in most circumstances.

'Catching' OutOfMemoryError completely solves out-of-memory issue?

I was getting OutOfMemoryError messages in LogCat and my app was crashing because the errors were uncaught.
Two things were causing the OutOfMemoryError:
1. Reading a large text file into a string.
2. Sending that string to my TextView.
Simply adding a catch to these two things not only catches the OutOfMemoryError but appears to completely solve the out-of-memory problem.
No more crashing and no more error messages in LogCat. The app just runs perfectly.
How is this possible? What exactly is happening?
With this code, I was getting the error messages & app crashing:
try
{
myString = new Scanner(new File(myFilePath)).useDelimiter("\\A").next();
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
myTextView.setText(myString);
Just by 'catching' the OutOfMemoryError, no more error messages in LogCat and no crashing:
try
{
myString = new Scanner(new File(myFilePath)).useDelimiter("\\A").next();
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
catch(OutOfMemoryError e)
{
}
try
{
myTextView.setText(myString);
}
catch(OutOfMemoryError e)
{
}

I guess your string isn't loaded completely, or even if it is (it may throw the error just after adding the text), what happens depends on the current memory available for your app so catching OutOfMemoryError isn't a viable solution here.
If you really want to load a large string file and display it in an EditText, I recommend you to load only a small part of the file (let's say 50kB), and implement some kind of paging, like a button which loads the next 50kB. You can also load additional lines as the user scrolls through the EditText.

If you catch the OutOfMemoryError, the garbage collector tries to free up the memory previously used and thus the application can carry on if the application will let the garbage collector do its job (i.e. the application has no longer a reference to that large string of yours).
However, catching an OutOfMemoryError is far from fool-proof. See Catching java.lang.OutOfMemoryError?.

When you catch the exception the JVM tries to recover from it by calling the Garbage Collector and scraping the objects that are not used anymore.
This might solve the problem in your case. But imagine that the problem appears because of bad coding and memory leaks all over your code. Catching will not solve the problem because GC will not collect any objects. GC will kick in more frequently and the performance of your application will drop until it becomes unusable.
Basically, this error happens when the JVM cannot allocate more memory on heap for new objects. Catching the exception and letting the GC to clean up and release memory might be a solution but you are never absolutely sure that you are in recoverable state. I would use the catch block to recover from the error, log it and close the application. If you want to solve the memory problem in this case, do it properly and initialize the JVM with more memory (using the java argument -Xmx)

out of memory exception when compiling files

Compilefile.this.compileThread = new Thread() {
#Override
public void run() {
try {
synchronized (this) {
Application.getDBHandler().setAutoCommit(false);
MIBParserUtils.getDefaultMibsMap();
compileSelectedFiles();
Application.getDBHandler().CommitTrans();
Application.getDBHandler().setAutoCommit(true);
}
}
catch(OutOfMemoryError exp) {
JOptionPane.showMessageDialog(null, "Compilation Stopped.. Insufficient Memory!!!");
CompileMib.this.compileThread.interrupt();
System.gc();
dispose();
NmsLogger.writeDebugLog(exp);
}
finally {
}
}
I tried to compile some files within a thread. The UI selects more than 200 files to compile. During compilation an OutOfMemoryError occurred due to in sufficient memory in Eclipse. I want to stop the thread and display a message box and dispose the compile window in my application. I wrote the below code but its not working. Can I catch the exception and handle it, or is there a better solution?

can i handle the exception in catch block?
You can certainly catch an OOME. But successfully recovering is another thing entirely. This Answer discusses some of the issues: https://stackoverflow.com/a/1692421/139985.
Another thing to consider is that the OOME might be being thrown on a different thread:
The compileSelectedFiles() method or one of the other methods could be doing the work on another thread and throwing OOME there.
The OOME could be being thrown on one of Eclipse's background threads.
In either case, that catch obviously won't catch it.
It is worth noting that calling System.gc() after an OOME is a waste of time. I can guarantee that it won't release any memory that wouldn't be released anyway. All you are doing is suggesting that the JVM waste time on something that won't help. If you are lucky, the JVM will ignore the suggestion.
My advice would be to just increase Eclipse's heap size by altering the -Xmx JVM parameter in the eclipse.ini file.

There is almost always no reliable way to recover from OOM, as anything that you try to put into catch block can require more memory, which is not available. And GC has already tried its best before OOM is thrown, so no point in asking him again.
As always, you can either increase the amount of memory available to your application via Xmx option, or fix your application for it not to require that much memory.
One more possible source of error is memory leak. In this case there is only 1 course of action: find it and fix it. Plumbr can help in that.

Have you tried adding the following to your eclipse.ini (located in same folder as eclipse.exe):
-Xmx1024m
This increases the heap space available to Eclipse. If your issue is during compilation, this may solve it. It gives 1GB of memory as heap space limit. Try -Xmx512m if you don't want to allocate quite so much space though.

Cleaning Up a Program that Continuously Builds a String out of Webpage

I'm looking to either clean up or make more efficient my program, but I'm not sure how. Here's what my program does:
I take a webpage, turn it into a string, scan it for a keyword, if I don't find the keyword, I rebuild the string taking care of any possible page edits (effectively refreshing the webpage), and this program goes on forever. If ever the webpage is not available, it just runs the main program again. It's a monitoring program.
I can imagine this constant string building to consume memory over time, similar to a memory leak. Anyway to fix this cleverly?
Also, can I print a message to the windows command prompt in a java program?

As long as your code releases references to the objects that it holds, then the garbage collector will free up the memory for you. You don't have to do anything else.
To print to the console, its as simple as this.
System.out.println("Hello World");
or
System.out.printf("Hello %s%n", "World");

Database connections and OutOfMemoryError: Java Heap Space

Last summer, I made a Java application that would parse some PDF files and get the information they contain to store them in a SQLite database.
Everything was fine and I kept adding new files to the database every week or so without any problems.
Now, I'm trying to improve my application's speed and I wanted to see how it would fare if I parsed all the files I have from the last two years in a new database. That's when I started getting this error: OutOfMemoryError: Java Heap Space. I didn't get it before because I was only parsing about 25 new files per week, but it seems like parsing 1000+ files one after the other is a lot more demanding.
I partially solved the problem: I made sure to close my connection after every call to the database and the error went away, but at a huge cost. Parsing the files is now unbearably slow. As for my ResultSets and Statements / PreparedStatements, I'm already closing them after every call.
I guess there's something I don't understand about when I should close my connection and when I should keep re-using the same one. I thought that since auto-commit is on, it commits after every transaction (select, update, insert, etc.) and the connection releases the extra memory it was using. I'm probably wrong since when I parse too many files, I end up getting the error I'm mentioning.
An easy solution would be to close it after every x calls, but then again I won't understand why and I'm probably going to get the same error later on. Can anyone explain when I should be closing my connections (if at all except when I'm done)? If I'm only supposed to do it when I'm done, then can someone explain how I'm supposed to avoid this error?
By the way, I didn't tag this as SQLite because I got the same error when I tried running my program on my online MySQL database.
Edit
As it has been pointed out by Deco and Mavrav, maybe the problem isn't my Connection. Maybe it's the files, so I'm going to post the code I use to call the function to parse the files one by one:
public static void visitAllDirsAndFiles(File dir){
if (dir.isDirectory()){
String[] children = dir.list();
for (int i = 0; i < children.length; i++){
visitAllDirsAndFiles(new File(dir, children[i]));
}
}
else{
try{
// System.out.println("File: " + dir);
BowlingFilesReader.readFile(dir, playersDatabase);
}
catch (Exception exc){
System.out.println("Other exception in file: " + dir);
}
}
}
So if I call the method using a directory, it recursively calls the function again using the File object I just created. My method then detects that it's a file and calls BowlingFilesReader.readFile(dir, playersDatabase);
The memory should be released when the method is done I think?

Your first instinct on open resultsets and connections was good, though maybe not entirely the cause. Let's start with your database connection first.
Database
Try using a database connection pooling library, such as the Apache Commons DBCP (BasicDataSource is a good place to start): http://commons.apache.org/dbcp/
You will still need to close your database objects, but this will keep things running smoothly on the database front.
JVM Memory
Increase the size of the memory you give to the JVM. You may do so by adding -Xmx and a memory amount after, such as:
-Xmx64m <- this would give the JVM 64 megs of memory to play with
-Xmx512m <- 512 megs
Be careful with your numbers, though, throwing more memory at the JVM will not fix memory leaks. You may use something like JConsole or JVisualVM (included in your JDK's bin/ folder) to observe how much memory you are using.
Threading
You may increase the speed of your operations by threading them out, assuming the operation you are performing to parse these records is threadable. But more information might be necessary to answer that question.
Hope this helps.

As it happens with Garbage colleciton I dont think the memory would be immediately recollected for the subsequent processes and threads.So we cant entirely put our eggs in that basket.To begin with put all the files in a directory and not in child directories of the parent. Then load the file one by one by iterating like this
File f = null;
for (int i = 0; i < children.length; i++){
f = new File(dir, children[i]);
BowlingFilesReader.readFile(f, playersDatabase);
f = null;
}
So we are invalidating the reference so that the file object is released and will be picked up in the subsequent GC. And to check the limits test it by increasing the no. of files start with 100, 200 ..... and then we will know at what point OME is getting thrown.
Hope this helps.