During a refactoring job, I discovered that if I close my outputstream using "try-with-resources" - I always get a MessagingException from java's SMTPTransport. It always complains that the socket was closed.
The code which I identified to have the problems is this:
try (LineOutputStream los = new LineOutputStream(os);) {
los.writeln(signatureHeaderLine);
Enumeration hdrLines = getNonMatchingHeaderLines(ignoreList);
while (hdrLines.hasMoreElements()) {
String notIgnoredLine = (String) hdrLines.nextElement();
los.writeln(notIgnoredLine);
}
los.writeln();
// Send signed mail to waiting DATA command
os.write(osBody.toByteArray());
os.flush();
} catch (MessagingException me) {
// Deal with it
} catch (Exception e) {
// Deal with it
}
The code above is part of an override of MimeMessage.writeTo(OutputStream, String[]) And the issue comes when `issueSendCommand' and 'sendCommand' is eventually called from SMTPTransport.
So does this mean my sockets should remain open all the time? I know from non-technical viewpoint, it doesn't feel right to close the socket since I will be writing messages through it. But I was trying to understand whether this would cause memory leak by any chance in the future.
Regards,
I believe the problem comes because you are using the OutputStream os outside the try-with-resource statement.
The try-with-resource statement ensures that all initialized AutoClosable resources will be closed after execution of the try block. At the moment com.sun.mail.util.LineOutputStream is closed, also the OutputStream os (passed to its constructor) will be closed. Any access to os after the try-with-resource statement acts on a already closed OutputStream.
edit There is an exception, when the OutputStream has a close() method without an effect. Which is the case for example for the ByteArrayOutputStream.
Closing a ByteArrayOutputStream has no effect. The methods in this class can be called after the stream has been closed without generating an IOException.
a snippet to demonstrate
private static void demoMethod(OutputStream os) throws IOException {
try (LineOutputStream los = new LineOutputStream(os)) {
los.writeln("signatureHeaderLine");
los.writeln();
os.write("foo".getBytes());
System.out.println("within try-block");
} catch (Exception e) {
System.out.println(e);
}
os.write("bar".getBytes());
System.out.println("after try-block");
}
calling the method demoMethod with a ByteArrayOutputStream
ByteArrayOutputStream os = new ByteArrayOutputStream();
demoMethod(os);
gives the output
within try-block
after try-block
So the ByteArrayOutputStream can be used even after calling close() on it (which is implicitly called by the LineOutputStream.close() invoked by try-with-resource code).
Doing the same with a FileOutputStream
FileOutputStream os = new FileOutputStream("/tmp/dummy.out");
demoMethod(os);
throws an exception, because the FileOutputStream has been close at the end of the try-with-resource statement.
within try-block
Exception in thread "main" java.io.IOException: Stream Closed
at java.base/java.io.FileOutputStream.writeBytes(Native Method)
at java.base/java.io.FileOutputStream.write(FileOutputStream.java:342)
at Main.demoMethod(Main.java:24)
at Main.main(Main.java:12)
I have next function:
static void write()
{
try {
File file = new File ("flip.out");
BufferedWriter out = new BufferedWriter(new FileWriter(file));
out.write(sMax);
System.out.println(sMax);//This command it works
out.close();
} catch (Exception e) {
e.printStackTrace();
}
}
The problem is that my program doesn't write anything in my file.
Few things to rectify -
Why create two different instances of File object
File file = new File ("flip.out");
BufferedWriter out = new BufferedWriter(new FileWriter("flip.out"));
All you need to do is
File file = new File ("flip.out");
BufferedWriter out = new BufferedWriter(new FileWriterfile(file ) ));
Next put your close call in finally statement rather than try block. Why? Because of IOException occurs resource will not be closed and if resource do not get closed your changes may not be reflected in the file.
Next it is a good programming practice not to catch Runtime exceptions. So do not use Exception as a ploymorphic type to catch your exception. Use whatever is being thrown like IOException in your case.
Now there might be various reasons why noting is being written in the file. As you are not getting and Exception one of the reason why this might be happening because your static function is not getting called or the string/object sMax(whatever that is) is empty.
Also the file(if not already present) will be created in the current directory. So if there are multiple instance is your code where your are creating files with same name then make sure you are checking the right one.
You have to flush the stream in order for what's in the memory to get written to the drive. What you wrote to BufferedWriter is sitting in a byte array waiting for the rest of it to be filled up before actually writing it to the disk. This helps with performance, but means you have to flush the stream in case you don't fill up that buffer. Here is how you do that:
static void write() throws IOException {
BufferedWriter out = new BufferedWriter(new FileWriter("flip.out"));
try {
out.write(sMax);
out.flush();
} catch (Exception e) {
// probably could ditch this and
// just the exception bubble up and
// handle it higher up.
e.printStackTrace();
} finally {
out.close();
}
}
So if it makes it to the flush() we know we wrote everything to the stream we wanted. However, if we ever get an exception we make sure we close the stream regardless of success or exception. And finally our stream is outside the try statement because the only exception ever thrown by Writers/OutputStreams during construction is FileNotFoundException which means the file never got opened in the first place so we don't have to close it.
can you call out.flush() before closing.
that will make sure any content in buffer is written to file immediately.
At some point of my server application I want to stop some threads that are performing I/O blocking operations.
For instance, one of them have the following run() method:
public void run() {
System.out.println("GWsocket thread running");
int len;
byte [] buffer = new byte[1500];
try {
this.in = new DataInputStream(this.socket.getInputStream());
this.out = new DataOutputStream(this.socket.getOutputStream());
running = true;
while (running){
len = in.read (buffer);
if (len < 0)
running = false;
else
parsepacket (buffer, len);
}
}catch (IOException ex) {
System.out.println("GWsocket catch IOException: "+ex);
}finally{
try {
System.out.println("Closing GWsocket");
fireSocketClosure();
in.close();
out.close();
socket.close();
}catch (IOException ex) {
System.out.println("GWsocket finally IOException: "+ex);
}
}
}
If I Want to stop a thread running this code, what should I do?
Here they show how to do (How do I stop a thread that waits for long periods (e.g., for input)?), but I don't understand what they mean with:
For this technique to work, it's critical that any method that catches
an interrupt exception and is not prepared to deal with it immediately
reasserts the exception. We say reasserts rather than rethrows,
because it is not always possible to rethrow the exception. If the
method that catches the InterruptedException is not declared to throw
this (checked) exception, then it should "reinterrupt itself" with the
following incantation: Thread.currentThread().interrupt();
Can anyone give me some hints? Some code examples would be very appreciated.
A solution, described by Peter Lawrey and also seen here is to close the socket.
With nio, You could also use a SocketChannel which is interruptible and would allow the application of the standard interrupt model of Java.
A call to the interrupt method of your Thread object would throw an InterruptedException which would stop even your blocking IO operation.
You can add a method like
public void close() throws IOException {
this.socket.close();
}
and any blocking IO operations will throw a SocketException.
You might like to set a flag like closed which you can check if an exception thrown was to be expected or not.
BTW: You cannot be sure that you will get discrete packets on a read. It is far better to read what you need and use BufferedInputStream for efficiency. Only read blocks if you don't need to parse the contents e.g. copying from a socket to a file.
e.g. your read could get just one byte or get the end of one packet and the start of another.
This question already has answers here:
Is it necessary to close each nested OutputStream and Writer separately?
(7 answers)
Closed 6 years ago.
Note: This question and most of its answers date to before the release of Java 7. Java 7 provides Automatic Resource Management functionality for doing this easilly. If you are using Java 7 or later you should advance to the answer of Ross Johnson.
What is considered the best, most comprehensive way to close nested streams in Java? For example, consider the setup:
FileOutputStream fos = new FileOutputStream(...)
BufferedOS bos = new BufferedOS(fos);
ObjectOutputStream oos = new ObjectOutputStream(bos);
I understand the close operation needs to be insured (probably by using a finally clause). What I wonder about is, is it necessary to explicitly make sure the nested streams are closed, or is it enough to just make sure to close the outer stream (oos)?
One thing I notice, at least dealing with this specific example, is that the inner streams only seem to throw FileNotFoundExceptions. Which would seem to imply that there's not technically a need to worry about closing them if they fail.
Here's what a colleague wrote:
Technically, if it were implemented right, closing the outermost
stream (oos) should be enough. But the implementation seems flawed.
Example:
BufferedOutputStream inherits close() from FilterOutputStream, which defines it as:
155 public void close() throws IOException {
156 try {
157 flush();
158 } catch (IOException ignored) {
159 }
160 out.close();
161 }
However, if flush() throws a runtime exception for some reason, then
out.close() will never be called. So it seems "safest" (but ugly) to
mostly worry about closing FOS, which is keeping the file open.
What is considered to be the hands-down best, when-you-absolutely-need-to-be-sure, approach to closing nested streams?
And are there any official Java/Sun docs that deal with this in fine detail?
When closing chained streams, you only need to close the outermost stream. Any errors will be propagated up the chain and be caught.
Refer to Java I/O Streams for details.
To address the issue
However, if flush() throws a runtime exception for some reason, then out.close() will never be called.
This isn't right. After you catch and ignore that exception, execution will pick back up after the catch block and the out.close() statement will be executed.
Your colleague makes a good point about the RuntimeException. If you absolutely need the stream to be closed, you can always try to close each one individually, from the outside in, stopping at the first exception.
In the Java 7 era, try-with-resources is certainly the way to go. As mentioned in several previous answers, the close request propagates from the outermost stream to the innermost stream. So a single close is all that is required.
try (ObjectInputStream ois = new ObjectInputStream(new FileInputStream(f))) {
// do something with ois
}
There is however a problem with this pattern. The try-with-resources is not aware of the inner FileInputStream, so if the ObjectInputStream constructor throws an exception, the FileInputStream is never closed (until the garbage collector gets to it). The solution is...
try (FileInputStream fis = new FileInputStream(f); ObjectInputStream ois = new ObjectInputStream(fis)) {
// do something with ois
}
This is not as elegant, but is more robust. Whether this is actually a problem will depend on what exceptions can be thrown during construction of the outer object(s). ObjectInputStream can throw IOException which may well get handled by an application without terminating. Many stream classes only throw unchecked exceptions, which may well result in termination of the application.
It is a good practice to use Apache Commons to handle IO related objects.
In the finally clause use IOUtils
IOUtils.closeQuietly(bWriter);
IOUtils.closeQuietly(oWritter);
Code snippet below.
BufferedWriter bWriter = null;
OutputStreamWriter oWritter = null;
try {
oWritter = new OutputStreamWriter( httpConnection.getOutputStream(), "utf-8" );
bWriter = new BufferedWriter( oWritter );
bWriter.write( xml );
}
finally {
IOUtils.closeQuietly(bWriter);
IOUtils.closeQuietly(oWritter);
}
I usually do the following. First, define a template-method based class to deal with the try/catch mess
import java.io.Closeable;
import java.io.IOException;
import java.util.LinkedList;
import java.util.List;
public abstract class AutoFileCloser {
// the core action code that the implementer wants to run
protected abstract void doWork() throws Throwable;
// track a list of closeable thingies to close when finished
private List<Closeable> closeables_ = new LinkedList<Closeable>();
// give the implementer a way to track things to close
// assumes this is called in order for nested closeables,
// inner-most to outer-most
protected final <T extends Closeable> T autoClose(T closeable) {
closeables_.add(0, closeable);
return closeable;
}
public AutoFileCloser() {
// a variable to track a "meaningful" exception, in case
// a close() throws an exception
Throwable pending = null;
try {
doWork(); // do the real work
} catch (Throwable throwable) {
pending = throwable;
} finally {
// close the watched streams
for (Closeable closeable : closeables_) {
if (closeable != null) {
try {
closeable.close();
} catch (Throwable throwable) {
if (pending == null) {
pending = throwable;
}
}
}
}
// if we had a pending exception, rethrow it
// this is necessary b/c the close can throw an
// exception, which would remove the pending
// status of any exception thrown in the try block
if (pending != null) {
if (pending instanceof RuntimeException) {
throw (RuntimeException) pending;
} else {
throw new RuntimeException(pending);
}
}
}
}
}
Note the "pending" exception -- this takes care of the case where an exception thrown during close would mask an exception we might really care about.
The finally tries to close from the outside of any decorated stream first, so if you had a BufferedWriter wrapping a FileWriter, we try to close the BuffereredWriter first, and if that fails, still try to close the FileWriter itself. (Note that the definition of Closeable calls for close() to ignore the call if the stream is already closed)
You can use the above class as follows:
try {
// ...
new AutoFileCloser() {
#Override protected void doWork() throws Throwable {
// declare variables for the readers and "watch" them
FileReader fileReader =
autoClose(fileReader = new FileReader("somefile"));
BufferedReader bufferedReader =
autoClose(bufferedReader = new BufferedReader(fileReader));
// ... do something with bufferedReader
// if you need more than one reader or writer
FileWriter fileWriter =
autoClose(fileWriter = new FileWriter("someOtherFile"));
BufferedWriter bufferedWriter =
autoClose(bufferedWriter = new BufferedWriter(fileWriter));
// ... do something with bufferedWriter
}
};
// .. other logic, maybe more AutoFileClosers
} catch (RuntimeException e) {
// report or log the exception
}
Using this approach you never have to worry about the try/catch/finally to deal with closing files again.
If this is too heavy for your use, at least think about following the try/catch and the "pending" variable approach it uses.
The colleague raises an interesting point, and there are grounds for arguing either way.
Personally, I would ignore the RuntimeException, because an unchecked exception signifies a bug in the program. If the program is incorrect, fix it. You can't "handle" a bad program at runtime.
This is a surprisingly awkward question. (Even assuming the acquire; try { use; } finally { release; } code is correct.)
If the construction of the decorator fails, then you wont be closing the underlying stream. Therefore you do need to close the underlying stream explicitly, whether in the finally after use or, more diifcult after successfully handing over the resource to the decorator).
If an exception causes execution to fail, do you really want to flush?
Some decorators actually have resources themselves. The current Sun implementation of ZipInputStream for instance has non-Java heap memory allocated.
It has been claimed that (IIRC) two thirds of the resources uses in the Java library are implemented in a clearly incorrect manner.
Whilst BufferedOutputStream closes even on an IOException from flush, BufferedWriter closes correctly.
My advice: Close resources as directly as possible and don't let them taint other code. OTOH, you can spend too much time on this issue - if OutOfMemoryError is thrown it's nice to behave nicely, but other aspects of your program are probably a higher priority and library code is probably broken in this situation anyway. But I'd always write:
final FileOutputStream rawOut = new FileOutputStream(file);
try {
OutputStream out = new BufferedOutputStream(rawOut);
... write stuff out ...
out.flush();
} finally {
rawOut.close();
}
(Look: No catch!)
And perhaps use the Execute Around idiom.
The Java SE 7 try-with-resources doesn't seem to be mentioned. It eliminates needing to explicitly do a close completely, and I quite like the idea.
Unfortunately, for Android development this sweet only becomes available by using Android Studio (I think) and targeting Kitkat and above.
Also you dont have to close all nested streams
check this
http://ckarthik17.blogspot.com/2011/02/closing-nested-streams.html
I use to close streams like this, without nesting try-catch in finally blocks
public class StreamTest {
public static void main(String[] args) {
FileOutputStream fos = null;
BufferedOutputStream bos = null;
ObjectOutputStream oos = null;
try {
fos = new FileOutputStream(new File("..."));
bos = new BufferedOutputStream(fos);
oos = new ObjectOutputStream(bos);
}
catch (Exception e) {
}
finally {
Stream.close(oos,bos,fos);
}
}
}
class Stream {
public static void close(AutoCloseable... array) {
for (AutoCloseable c : array) {
try {c.close();}
catch (IOException e) {}
catch (Exception e) {}
}
}
}
Sun's JavaDocs include RuntimeExceptions in their documentation, as shown by InputStream's read(byte[], int, int) method; documented as throwing NullPointerException and IndexOutOfBoundsException.
FilterOutputStream's flush() is only documented as throwing IOException, thus it doesn't actually throw any RuntimeExceptions. Any that could be thrown would most likely be wrapped in an IIOException.
It could still throw an Error, but there's not much you can do about those; Sun recommends that you don't try to catch them.
So I have a an application which is running on top of gridgain and does so quite successfully for about 12-24 hours of stress testing before it starts to act funny. After this period of time the application will suddenly start replying to all queries with the exception java.nio.channels.ClosedByInterruptException (full stack trace is at http://pastie.org/664717
The method that is failing from is (edited to use #stephenc feedback)
public static com.vlc.edge.FileChannel createChannel(final File file) {
FileChannel channel = null;
try {
channel = new FileInputStream(file).getChannel();
channel.position(0);
final com.vlc.edge.FileChannel fileChannel = new FileChannelImpl(channel);
channel = null;
return fileChannel;
} catch (FileNotFoundException e) {
throw new VlcRuntimeException("Failed to open file: " + file, e);
} catch (IOException e) {
throw new VlcRuntimeException(e);
} finally {
if (channel != null) {
try {
channel.close();
} catch (IOException e){
// noop
LOGGER.error("There was a problem closing the file: " + file);
}
}
}
}
and the calling function correctly closes the object
private void fillContactBuffer(final File signFile) {
contactBuffer = ByteBuffer.allocate((int) signFile.length());
final FileChannel channel = FileUtils.createChannel(signFile);
try {
channel.read(contactBuffer);
} finally {
channel.close();
}
contactBuffer.rewind();
}
The application basically serves as a distributed file parser so it does a lot of these types of operations (will typically open about 10 such channels per query per node). It seems that after a certain period it stops being able to open files and I'm at a loss to explain why this could be happening and would greatly appreciate any one who can tell me what could be causing this and how I could go about tracking it down and fixing it. If it is possibly related to file handle exhaustion, I'd love to hear any tips for finding out for sure... i.e. querying the JVM while it's running or using linux command line tools to find out more information about what handles are currently open.
update: I've been using command line tools to interrogate the output of lsof and haven't been able to see any evidence that file handles are being held open... each node in the grid has a very stable profile of openned files which I can see changing as the above code is executed... but it always returns to a stable number of open files.
Related to this question: Freeing java file handles
There are a couple of scenarios where file handles might not be being closed:
There might be some other code that opens files.
There might be some other bit of code that calls createChannel(...) and doesn't call fillContactBuffer(...)
If channel.position(0) throws an exception, the channel won't be closed. The fix is to rearrange the code so that the following statements are inside the try block.
channel.position(0);
return new FileChannelImpl(channel);
EDIT: Looking at the stack trace, it seems that the two methods are in different code-bases. I'd point the finger of blame at the createChannel method. It is potentially leaky, even if it is not the source of your problems. It needs an in internal finally clause to make sure that the channel is closed in the event of an exception.
Something like this should do the trick. Note that you need to make sure that the finally block does not closes the channel on success!
public static com.vlc.edge.FileChannel createChannel(final File file) {
final FileChannel channel = null;
try {
channel = new FileInputStream(file).getChannel();
channel.position(0);
FileChannel res = new FileChannelImpl(channel);
channel = null;
return res;
} catch (FileNotFoundException e) {
throw new VlcRuntimeException("Failed to open file: " + file, e);
} catch (IOException e) {
throw new VlcRuntimeException(e);
} finally {
if (channel != null) {
try {
channel.close();
} catch (...) {
...
}
}
}
}
FOLLOWUP much later
Given that file handle leakage has been eliminated as a possible cause, my next theory would be that the server side is actually interrupting its own threads using Thread.interrupt(). Some low-level I/O calls respond to an interrupt by throwing an exception, and the root exception being thrown here looks like one such exception.
This doesn't explain why this is happening, but at a wild guess I'd say that it was the server-side framework trying to resolve an overload or deadlock problem.