I have heard people many times use the term "Resource leak". I'm sure this is a global phenomenon, though for the purpose of this answer, I will stick to those in Java. Take for example the following code :
public void append(String text) throws IOException
{
BufferedWriter buffWriter = new BufferedWriter(new FileWriter("tf2rocks.imnotkidding", true));
buffWriter.write(event);
buffWriter.close();
}
In the above snippet, there is a resource leak, as if an IOException was thrown by write(), close() will never be called.
Now my question is : What exactly is a resource leak? How can they cause harm to me?
If each java program is executed in it's own instance of the JVM, in an enclosed environment, how exactly can these "resource leaks" cause me harm? Is it possible for other malicious programs to take advantage of this?
Classes implementing java.io.Closeable (since JDK 1.5) and java.lang.AutoCloseable (since JDK 1.7) are considered to represent external resources, which should be closed using method close(), when they are no longer needed. All operating system have limits on the number of sockets, file handles.etc. that can be opened at particular instance of time. If you don't close the resources then you are unnecessary keeping them open and if you continue to keep opening more and more resources without closing them then after some time operating system will not be able to allocate more resources.
You are right with your example. Think from OS perspective, where you have predefined file/sockets handles that you could create i.e. say this many files can be open at a time. Now if you keep on opening multiple files, you may go out of number of files that can be open.
So although, its JVM but JVM alone can't run without the help from OS. File handler/descriptor is just one of the example.
If it's memory, then you don't have you worry as it will be handled by GC automatically but GC wont take care of resources.
Related
What is the meaning of the word "resource" in java?
Why we have to close it after usage,even though garbage collector runs in jvm ?
Why we have to write resource cleanup code in finally block?
A resource is something that has a limited ammount, such as DB connections and file descriptors. The GC frees memory, but you still have to release resources such as DB connections, open files, etc..., to allow other threads to use them.
By the way, it's better to free the resources immediately after you finish using them, and not just in the finalize method, which might take a long time until it is called by the GC.
A database connection, a thread, a file handle, a socket - all are finite resources.
The operating system you run on only allows so many threads - 1 MB overhead per thread. You're limited by the available RAM. Same for file handles and sockets.
Database connections are interesting because they involve client and server. If the client gc's the connection, what tells the server to close the connection? If you fail to close in finally blocks, you'll soon find out that your database server will run out of connections under heavy load.
Finalize is not the right way to go. Don't depend on the VM to call it. Write a close() method and call it in a finally block when your method is done with the resource. Close in the narrowest scope possible.
Say you have a file, you can write to it and not close the resource and eventually it will be closed by the GC. The problem is that while the file is open you can't delete it in windows, and in Linux you can delete it but it doesn't free any space. If you want to delete a file you don't want to be waiting until the GC feels like running perhaps hours later.
What is the meaning of the word "resource" in java?
The typical Java application manipulates several types of resources such as files, streams, sockets, and database connections.
Why we have to write resource cleanup code in finally block?
The Oracle article presents the Java 7 answer to the automatic resource management problem.
Such resources must be handled with great care, because they acquire system resources for their operations. Thus, you need to ensure that they get freed even in case of errors.
Indeed, incorrect resource management is a common source of failures in production applications, with the usual pitfalls being database connections and file descriptors remaining opened after an exception has occurred somewhere else in the code.
This leads to application servers being frequently restarted when resource exhaustion occurs, because operating systems and server applications generally have an upper-bound limit for resources.
Use Java 7 The try-with-resources Statement
I've always heard that resources in java must be closed after use or these resources will get exhausted. Is it really a matter of concern for small programs that use very few resources(like 1 or 2 file readers/ buffered readers and all)? Doesn't the JVM keep track of the resources being used by a program? And wouldn't it release all those resources once the program exits? If not, why would the JVM keep these resources blocked even after the program exit?
These resources are indeed closed upon a normal JVM exit. However, you may not always know how your method is being called, and it could perhaps be called 2000 times externally by another programmer, and those resources will start to stack up.
In addition, certain non-mainstream OSs may run into the issue that if the JVM were to halt abnormally(via Runtime.getRuntime().halt() or a significant internal error/inconsistency within the JVM) then resources could remain open(due to cleanup code not being run), potentially unusable until rebooted or manually released. Even on mainstream systems sockets could remain open for multiple minutes.
When the process exits, the JVM exits. What happens then is that the OS will free up any resources previously used by the JVM. This includes things like memory, file handles etc.
Small programms might live with resource leaks, because the OS might handle this after the process ended. But this is considered non portable.
The bigger issue is that the OS the JVM is running in won't know how to free resources allocated on a remote machine or if there are even resources allocated that need to be freed.
It is not JVM but OS who releases resources allocated to the JVM process after it exited.
I built my first Java program which is built on top of the Interactive Brokers Java API. That may or may not be important. I just extended the main API classes with a couple new classes.
The program is making data queries to a remote server. When the server responds, I log the received data to a local MySQL data base. Once the program finishes logging the data, the program will make the next data request.
I am having a problem after leaving the program running for some time, after making a couple hundred server requests. I will see this error, then the program doesn't continue to execute:
java.lang.OutOfMemoryError: Java heap space
I did some research, and from what I read, I conclude that the program is creating many new variables, and not destroying old worthless ones. Since I am using Netbeans for development, I used the Netbeans profiler to inspect if this was the case. See the picture here:
After running the program for quite some time, more and more of the memory is used up by Byte. So it seems that my theory is still true.
I don't really know where to go from here. There is no reference to a class or specific variable, just a variable type. How can pinpoint where the problem is coming from?
UPDATE
I corrected a specific problem that was mentioned by BigMike in the comments. Previiously, I was creating many Statements in the JDBC MySQL Connector API, and I was calling .execute() to execute the statements, but I wasn't closing the statement with .close().
I made sure the add the statement.close() call after each execution, and the program runs much better now. By looking at the RAM usage for this program, it seems to solved the problem. I am also not seeing the Java heap space error anymore, which is nice.
Thanks!
It's very hard to say what might be wrong by simply that.
It might have to do with Streams that you are opening that aren't being closed when you no longer need them.
Double check methods that allocate resources (reading from files, database, etc), especially if they read data into streams, and make sure you close those streams in a finally clause.
Apart from that, you can try and profile what methods are being called more often, etc, to try and narrow down the problem to a specific part of your code.
I found a site with a reasonable explanation of how Garbage Collection works, and what can cause OutOfMemoryErrors:
http://www.kdgregory.com/index.php?page=java.outOfMemory
If you read through that, there's a specific reference to high allocation of Object[] and byte[], that might point you in the right direction.
Generally speaking, this comes about for one of two reasons:
There is a memory leak in the application, such that the application fails to release items for garbage collection, leading to the JVM running out of memory over time.
The application attempted a one-off operation that would require more memory than is available, leading to the JVM running out of memory due to the operation.
Since your output seems to indicate that the bulk of the memory is consumed by literally a million plus small byte arrays, my guess is that #1 is probably the culprit; however, to verify this, restart your application and watch it's memory consumption over time. It will bounce up and down, but really you only need to watch the trend of consumption. If the consumption average continues to climb over time, you have a memory leak.
To solve this issue, you typically need the source code, and need to find the parts of the code where the troubling objects are being created, used, and then "stored" far beyond the last time that they will ever be used. The solution is to correct the code to no longer store them. HashMaps, Lists, and other Collections are often accomplices in memory leak problems.
If you lack the source code, you can attempt to measure the trend of the memory consumption, and schedule shutdowns and restarts of the application to effectively "reset the clock" such that you choose your downtime instead of watching the application choose it for you.
If it is a one-off operation (not likely considering your data) then you won't see an upward trend in memory consumption until the triggering event occurs. In such a case, with access to the source code, you should protect your application from processing data that grows very far outside of normal operating parameters. For example, reading a message from the network typically takes only a few KB, but in exceptional circumstances a client might transmit forever. In such a case, kill the message processing and throw the message away with an error if you exceed a maximum message size limit of 10 MB.
Without access to the source code in the latter scenario, the only hope is to identify the incoming upset, hunt down the source of the errant transmission, and attempt to manipulate it to prevent the overload of output.
The variations on how to approach these techniques are vast, but now you have a few ideas.
Java applications may use IO streams, sockets or database connections that should be closed when they are no longer needed.
However, applications may be terminated (e.g. by killing the process). Will all used resources be freed in this case? Who will free them: OS or JRE?
The JVM will release all active resources upon termination; however, this does not ensure that the other end will free the resource too, so explicitly closing resources is in every programmer's best interest.
An alternative to explicitly closing streams exists in Java 7, called the try-resource "statement", which is the equivalent of closing a resource in a finally block after a try block. More info on that can be found here.
If your software doesn't take care of resource management properly, the following will happen:
at runtime: the JVM will attempt during the duration of your program to close open streams if they're are seemingly unused (during garbage collection cycles),
at your program's termination point: the JVM should close open streams of all kinds left open by your program,
at the JVM's process termination point: the operating system will take care of releasing anything that hasn't been properly released by the JVM when it exists (hopefully, or this OS has some serious issues...).
As mentioned by Vulcan, none of this ensures that they are properly dealt with on the other end, obviously.
Note that the 3rd bullet point is a rather generic thing: most operating systems will take care of this, and it doesn't relate to the Java Platform's internals. It's about the OS managing its processes and resources on its own.
More Info
See also:
Cleanup of resources on an abnormal Exit, also on StackOverflow
How do I implement graceful termination in Java
Java Theory and Practice: Good Housekeeping Practices, at the IBM Developer Series
This question already has answers here:
Why would you ever implement finalize()?
(21 answers)
Closed 5 years ago.
This is mostly out of curiosity.
I was wandering if anyone has encountered any good usage for Object.finalize() except for debugging/logging/profiling purposes ?
If you haven't encountered any what would you say a good usage would be ?
If your Java object uses JNI to instruct native code to allocate native memory, you need to use finalize to make sure it gets freed.
Late to the party here but thought I would still chime in:
One of the best uses I have found for finalizers is to call explicit termination methods which, for what ever reason, were not called. When this occurs, we also log the issue because it is a BUG!
Because:
There is no guarantee that finalizers will be executed promptly (or technically at all), per the language specification
Execution is largely dependent on the JVM implementation
Execution can sometimes be delayed if the GC has a lower thread priority
This leaves only a handful of tasks that they can address without much risk.
close external connections (db, socket etc)
close open files. may be even try to write some additional information.
logging
if this class runs external processes that should exist only while object exists you can try to kill them here.
But it is just a fallback that is used is "normal" mechanism did not work. Normal mechanism should be initiated explicitly.
Release resources that should be released manually in normal circumstances, but were not released for some reason. Perhaps with write a warning to the log.
I use it to write back data to a database when using soft references for caching database-backed objects.
I see one good use for finalize(): freeing resources that are available in large amounts and are not exclusive.
For example, by default there are 1024 file handles available for a Linux process and about 10000 for Windows. This is pretty much, so for most applications if you open a file, you don't have to call .close() (and use the ugly try...finally blocks), and you'll be OK - finally() will free it for you some time later. However for some pieces of code (like intensive server applications), releasing resources with .close() is a must, otherwise finally() may be called too late for you and you may run out of file handles.
The same technique is used by Swing - operating system resources for displaying windows and drawing aren't released by any .close() method, but just by finalize(), so you don't have to worry about all .close() or .dispose() methods like in SWT for example.
However, when there is very limited number of resources, or you must 'lock' resource to use it, also remember to 'unlock' it. For example if you create a file lock on a file, remember also to remove this lock, otherwise nobody else will be able to read or write this file and this can lead to deadlocks - then you can't rely on finalize() to remove this lock for you - you must do it manually at the right place.