Develop Reference

Reading InputStream from webservice response

What I am trying to do seems to be a quite simple thing, get an InputStream from a Jersey webservice which is returned from a class RestResponse. But I don't get the stream to my client:
public class RestResponse {
private InputStream responseStream;
public RestResponse(InputStream responseBodyStream) throws IOException{
this.responseStream = responseBodyStream;
//here I can get the stream contents from this.responseStream
}
public InputStream getResponseStream() throws IOException {
//here stream content is empty, if called from outside
//only contains content, if called from constructor
return this.responseStream;
}
}
public class HttpURLConnectionClient{
public RestResponse call(){
try{
....
InputStream in = httpURLConnection.getInputStream();
RestResponse rr = new RestResponse(in);
}finally{
in.close(); <- this should be the suspect
}
}
}
RestResponse rr = httpURLConnectionClient.call()//call to some url
rr.getResponseStream(); //-> stream content is empty
Any ideas, what I am missing? Is is not possible to just pipe the stream through?

Certain types of InputStream can only be read once in Java. Based on your comment above, it appears that you are using the InputStream when you pipe it to System.out. Try commenting out the call to System.out and see if you can access your InputStream. Also make sure that the stream is not being consumed anywhere else in the code before the point where you need it.
Update:
It appears that your actual problem was being caused by closing the InputStream before you got a chance to use it. So the solution is to keep the stream open until you need it, and close it afterwards.
Typically, it is not a good design practice to open a stream and keep it open for a long time, because then the underlying resource won't be available to anyone else who needs it. So you should open the stream, and consume it only when you actually need it.

Alternative ways to write a file in Java

I have next function:
static void write()
{
try {
File file = new File ("flip.out");
BufferedWriter out = new BufferedWriter(new FileWriter(file));
out.write(sMax);
System.out.println(sMax);//This command it works
out.close();
} catch (Exception e) {
e.printStackTrace();
}
}
The problem is that my program doesn't write anything in my file.

Few things to rectify -
Why create two different instances of File object
File file = new File ("flip.out");
BufferedWriter out = new BufferedWriter(new FileWriter("flip.out"));
All you need to do is
File file = new File ("flip.out");
BufferedWriter out = new BufferedWriter(new FileWriterfile(file ) ));
Next put your close call in finally statement rather than try block. Why? Because of IOException occurs resource will not be closed and if resource do not get closed your changes may not be reflected in the file.
Next it is a good programming practice not to catch Runtime exceptions. So do not use Exception as a ploymorphic type to catch your exception. Use whatever is being thrown like IOException in your case.
Now there might be various reasons why noting is being written in the file. As you are not getting and Exception one of the reason why this might be happening because your static function is not getting called or the string/object sMax(whatever that is) is empty.
Also the file(if not already present) will be created in the current directory. So if there are multiple instance is your code where your are creating files with same name then make sure you are checking the right one.

You have to flush the stream in order for what's in the memory to get written to the drive. What you wrote to BufferedWriter is sitting in a byte array waiting for the rest of it to be filled up before actually writing it to the disk. This helps with performance, but means you have to flush the stream in case you don't fill up that buffer. Here is how you do that:
static void write() throws IOException {
BufferedWriter out = new BufferedWriter(new FileWriter("flip.out"));
try {
out.write(sMax);
out.flush();
} catch (Exception e) {
// probably could ditch this and
// just the exception bubble up and
// handle it higher up.
e.printStackTrace();
} finally {
out.close();
}
}
So if it makes it to the flush() we know we wrote everything to the stream we wanted. However, if we ever get an exception we make sure we close the stream regardless of success or exception. And finally our stream is outside the try statement because the only exception ever thrown by Writers/OutputStreams during construction is FileNotFoundException which means the file never got opened in the first place so we don't have to close it.

can you call out.flush() before closing.
that will make sure any content in buffer is written to file immediately.

Store raw HTML content from URL then get a InputStream from memory (not using a connection)

I have a kind of tricky problem involving multi-threading. What I do is that I use a thread pool (ExecutorService) that is tasked with opening connections and putting them in a LinkedBlockingQueue.
So far I have used:
//run method in "getter threads"
public void run() {
try {
URL url = new URL(url_s); //url_s is given as a constructor argument
//if I am correct then url.openStream will wait until we have the content
InputStream stream = url.openStream();
Request req = new Request(); //a class with two variables:
req.html_stream = new InputSource(stream);
req.source = stream;
//this is a class variable (LinkedBlockingQueue<Request>)
blocking_queue.put(req);
} catch (Exception ex) {
logger.info("Getter thread died from an exeption",ex);
return;
}
}
I then have consumer thread (java.lang.Thread) that takes these InputSources and InputStreams and does:
public void run() {
while(running) {
try {
logger.info("waiting for data to eat");
Request req = blocking_queue.take();
if(req.html_stream != null)
eat_data(req);
} catch (Exception ex) {
logger.error(ex);
return;
}
}
}
Where eat_data calls an external library that takes InputSource. The library uses a singleton instance to do the processing so I cant put this step in the "getter" threads.
When I tested this code for small amounts of data it worked fine, but when I supplied it with several thousands of URLs I started to have real problems. Its not easy to find out exactly what is wrong, but I suspect that the connections time out before the consumer thread get to them, sometimes even causing deadlock.
I implemented it this way because it was so easy to go from url.openStream() to InputSource but I realize that I really must store the data locally for this to work.
How do I get from url.openStream() to some object I can store in my LinkedBlockingQueue (all data in memory) that I can later turn into an InputSoruce when my consumer thread has time to process it?

You can copy the contents of the URL into a ByteArrayOutputStream and close the URL Stream. Then store the ByteArrayInputStream in the queue.
Pseudo Code :
InputStream in = null;
try {
in = url.openStream();
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
IOUtils.copy(in, buffer);
ByteArrayInputStream bin = new ByteArrayInputStream(buffer.toByteArray());
queue.put(bin);
}
References :
java.io.ByteArrayInputStream
java.io.ByteArrayOutputStream
org.apache.commons.io.IOUtils.IOUtils

How / when to delete a file in java?

The problem is, user clicks a button in JSP, which will export the displayed data. So what i am doing is, creating a temp. file and writing the contents in it [ resultSet >> xml >> csv ], and then writing the contents to ServletResponse. After closing the respons output stream, i try to delete the file, but every time it returns false.
code;
public static void writeFileContentToResponse ( HttpServletResponse response , String fileName ) throws IOException{
ServletOutputStream responseoutputStream = response.getOutputStream();
File file = new File(fileName);
if (file.exists()) {
file.deleteOnExit();
DataInputStream dis = new DataInputStream(new FileInputStream(
file));
response.setContentType("text/csv");
int size = (int) file.length();
response.setContentLength(size);
response.setHeader("Content-Disposition",
"attachment; filename=\"" + file.getName() + "\"");
response.setHeader("Pragma", "public");
response.setHeader("Cache-control", "must-revalidate");
if (size > Integer.MAX_VALUE) {
}
byte[] bytes = new byte[size];
dis.read(bytes);
FileCopyUtils.copy(bytes, responseoutputStream );
}
responseoutputStream.flush();
responseoutputStream.close();
file.delete();
}
i have used 'file.deleteOnExit();' and file.delete(); but none of them is working.

file.deleteOnExit() isn't going to produce the result you want here - it's purpose is to delete the file when the JVM exits - if this is called from a servlet, that means to delete the file when the server shuts down.
As for why file.delete() isn't working - all I see in this code is reading from the file and writing to the servlet's output stream - is it possible when you wrote the data to the file that you left the file's input stream open? Files won't be deleted if they're currently in use.
Also, even though your method throws IOException you still need to clean up things if there's an exception while accessing the file - put the file operations in a try block, and put the stream.close() into a finally block.

Don't create that file.
Write your data directly from your resultset to your CSV responseoutputStream.
That saves time, memory, diskspace and headache.
If you realy need it, try using File.createTempFile() method.
These files will be deleted when your VM stops normaly if they haven't been deleted before.

I'm assuming you have some sort of concurrency issue going on here. Consider making this method non-static, and use a unique name for your temp file (like append the current time, or use a guid for a filename). Chances are that you're opening the file, then someone else opens it, so the first delete fails.

as I see it, you are not closing the DataInputStream dis - this results to the false status, when you do want to delete file. Also, you should handle the streams in try-catch-finally block and close them within finally. The code is a bit rough, but it is safe:
DataInputStream dis = null;
try
{
dis = new DataInputStream(new FileInputStream(
file));
... // your other code
}
catch(FileNotFoundException P_ex)
{
// catch only Exceptions you want, react to them
}
finally
{
if(dis != null)
{
try
{
dis.close();
}
catch (IOException P_ex)
{
// handle exception, again react only to exceptions that must be reacted on
}
}
}

How are you creating the file. You probably need to use createTempFile.
You should be able to delete a temporary file just fine (No need for deleteOnExit). Are you sure the file isn't in use, when you are trying to delete it? You should have one file per user request (That is another reason you should avoid temp files and store everything in memory).

you can try piped input and piped output stream. those buffers need two threads one to feed the pipe (exporter) and the other (servlet) to consume data from the pipe and write it to the response output stream

You really don't want to create a temporary file for a request. Keep the resulting CSV in memory if at all possible.
You may need to tie the writing of the file in directly with the output. So parse a row of the result set, write it out to response stream, parse the next row and so on. That way you only keep one row in memory at a time. Problem there is that the response could time out.
If you want a shortcut method, take a look at Display tag library. It makes it very easy to show a bunch of results in a table and then add pre-built export options to said table. CSV is one of those options.

You don't need a temporary file. The byte buffer which you're creating there based on the file size may also cause OutOfMemoryError. It's all plain inefficient.
Just write the data of the ResultSet immediately to the HTTP response while iterating over the rows. Basically: writer.write(resultSet.getString("columnname")). This way you don't need to write it to a temporary file or to gobble everything in Java's memory.
Further, most JDBC drivers will by default cache everything in Java's memory before giving anything to ResultSet#next(). This is also inefficient. You'd like to let it give the data immediately row-by-row by setting the Statement#setFetchSize(). How to do it properly depends on the JDBC driver used. In case of for example MySQL, you can read it up in its JDBC driver documentation.
Here's a kickoff example, assuming that you're using MySQL:
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
response.setContentType("text/csv");
response.setCharacterEncoding("UTF-8");
Connection connection = null;
Statement statement = null;
ResultSet resultSet = null;
PrintWriter writer = response.getWriter();
try {
connection = database.getConnection();
statement = connection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
statement.setFetchSize(Integer.MIN_VALUE);
resultSet = statement.executeQuery("SELECT col1, col2, col3 FROM tbl");
while (resultSet.next()) {
writer.append(resultSet.getString("col1")).append(',');
writer.append(resultSet.getString("col2")).append(',');
writer.append(resultSet.getString("col3")).println();
// Note: don't forget to escape quotes/commas as per RFC4130.
}
} catch (SQLException e) {
throw new ServletException("Retrieving CSV rows from DB failed", e);
} finally {
if (resultSet != null) try { resultSet.close; } catch (SQLException logOrIgnore) {}
if (statement != null) try { statement.close; } catch (SQLException logOrIgnore) {}
if (connection != null) try { connection.close; } catch (SQLException logOrIgnore) {}
}
}
That's it. This way effectlvely only one database row is been kept in the memory all the time.

Correct way to close nested streams and writers in Java [duplicate]

This question already has answers here:
Is it necessary to close each nested OutputStream and Writer separately?
(7 answers)
Closed 6 years ago.
Note: This question and most of its answers date to before the release of Java 7. Java 7 provides Automatic Resource Management functionality for doing this easilly. If you are using Java 7 or later you should advance to the answer of Ross Johnson.
What is considered the best, most comprehensive way to close nested streams in Java? For example, consider the setup:
FileOutputStream fos = new FileOutputStream(...)
BufferedOS bos = new BufferedOS(fos);
ObjectOutputStream oos = new ObjectOutputStream(bos);
I understand the close operation needs to be insured (probably by using a finally clause). What I wonder about is, is it necessary to explicitly make sure the nested streams are closed, or is it enough to just make sure to close the outer stream (oos)?
One thing I notice, at least dealing with this specific example, is that the inner streams only seem to throw FileNotFoundExceptions. Which would seem to imply that there's not technically a need to worry about closing them if they fail.
Here's what a colleague wrote:
Technically, if it were implemented right, closing the outermost
stream (oos) should be enough. But the implementation seems flawed.
Example:
BufferedOutputStream inherits close() from FilterOutputStream, which defines it as:
155 public void close() throws IOException {
156 try {
157 flush();
158 } catch (IOException ignored) {
159 }
160 out.close();
161 }
However, if flush() throws a runtime exception for some reason, then
out.close() will never be called. So it seems "safest" (but ugly) to
mostly worry about closing FOS, which is keeping the file open.
What is considered to be the hands-down best, when-you-absolutely-need-to-be-sure, approach to closing nested streams?
And are there any official Java/Sun docs that deal with this in fine detail?

When closing chained streams, you only need to close the outermost stream. Any errors will be propagated up the chain and be caught.
Refer to Java I/O Streams for details.
To address the issue
However, if flush() throws a runtime exception for some reason, then out.close() will never be called.
This isn't right. After you catch and ignore that exception, execution will pick back up after the catch block and the out.close() statement will be executed.
Your colleague makes a good point about the RuntimeException. If you absolutely need the stream to be closed, you can always try to close each one individually, from the outside in, stopping at the first exception.

In the Java 7 era, try-with-resources is certainly the way to go. As mentioned in several previous answers, the close request propagates from the outermost stream to the innermost stream. So a single close is all that is required.
try (ObjectInputStream ois = new ObjectInputStream(new FileInputStream(f))) {
// do something with ois
}
There is however a problem with this pattern. The try-with-resources is not aware of the inner FileInputStream, so if the ObjectInputStream constructor throws an exception, the FileInputStream is never closed (until the garbage collector gets to it). The solution is...
try (FileInputStream fis = new FileInputStream(f); ObjectInputStream ois = new ObjectInputStream(fis)) {
// do something with ois
}
This is not as elegant, but is more robust. Whether this is actually a problem will depend on what exceptions can be thrown during construction of the outer object(s). ObjectInputStream can throw IOException which may well get handled by an application without terminating. Many stream classes only throw unchecked exceptions, which may well result in termination of the application.

It is a good practice to use Apache Commons to handle IO related objects.
In the finally clause use IOUtils
IOUtils.closeQuietly(bWriter);
IOUtils.closeQuietly(oWritter);
Code snippet below.
BufferedWriter bWriter = null;
OutputStreamWriter oWritter = null;
try {
oWritter = new OutputStreamWriter( httpConnection.getOutputStream(), "utf-8" );
bWriter = new BufferedWriter( oWritter );
bWriter.write( xml );
}
finally {
IOUtils.closeQuietly(bWriter);
IOUtils.closeQuietly(oWritter);
}

I usually do the following. First, define a template-method based class to deal with the try/catch mess
import java.io.Closeable;
import java.io.IOException;
import java.util.LinkedList;
import java.util.List;
public abstract class AutoFileCloser {
// the core action code that the implementer wants to run
protected abstract void doWork() throws Throwable;
// track a list of closeable thingies to close when finished
private List<Closeable> closeables_ = new LinkedList<Closeable>();
// give the implementer a way to track things to close
// assumes this is called in order for nested closeables,
// inner-most to outer-most
protected final <T extends Closeable> T autoClose(T closeable) {
closeables_.add(0, closeable);
return closeable;
}
public AutoFileCloser() {
// a variable to track a "meaningful" exception, in case
// a close() throws an exception
Throwable pending = null;
try {
doWork(); // do the real work
} catch (Throwable throwable) {
pending = throwable;
} finally {
// close the watched streams
for (Closeable closeable : closeables_) {
if (closeable != null) {
try {
closeable.close();
} catch (Throwable throwable) {
if (pending == null) {
pending = throwable;
}
}
}
}
// if we had a pending exception, rethrow it
// this is necessary b/c the close can throw an
// exception, which would remove the pending
// status of any exception thrown in the try block
if (pending != null) {
if (pending instanceof RuntimeException) {
throw (RuntimeException) pending;
} else {
throw new RuntimeException(pending);
}
}
}
}
}
Note the "pending" exception -- this takes care of the case where an exception thrown during close would mask an exception we might really care about.
The finally tries to close from the outside of any decorated stream first, so if you had a BufferedWriter wrapping a FileWriter, we try to close the BuffereredWriter first, and if that fails, still try to close the FileWriter itself. (Note that the definition of Closeable calls for close() to ignore the call if the stream is already closed)
You can use the above class as follows:
try {
// ...
new AutoFileCloser() {
#Override protected void doWork() throws Throwable {
// declare variables for the readers and "watch" them
FileReader fileReader =
autoClose(fileReader = new FileReader("somefile"));
BufferedReader bufferedReader =
autoClose(bufferedReader = new BufferedReader(fileReader));
// ... do something with bufferedReader
// if you need more than one reader or writer
FileWriter fileWriter =
autoClose(fileWriter = new FileWriter("someOtherFile"));
BufferedWriter bufferedWriter =
autoClose(bufferedWriter = new BufferedWriter(fileWriter));
// ... do something with bufferedWriter
}
};
// .. other logic, maybe more AutoFileClosers
} catch (RuntimeException e) {
// report or log the exception
}
Using this approach you never have to worry about the try/catch/finally to deal with closing files again.
If this is too heavy for your use, at least think about following the try/catch and the "pending" variable approach it uses.

The colleague raises an interesting point, and there are grounds for arguing either way.
Personally, I would ignore the RuntimeException, because an unchecked exception signifies a bug in the program. If the program is incorrect, fix it. You can't "handle" a bad program at runtime.

This is a surprisingly awkward question. (Even assuming the acquire; try { use; } finally { release; } code is correct.)
If the construction of the decorator fails, then you wont be closing the underlying stream. Therefore you do need to close the underlying stream explicitly, whether in the finally after use or, more diifcult after successfully handing over the resource to the decorator).
If an exception causes execution to fail, do you really want to flush?
Some decorators actually have resources themselves. The current Sun implementation of ZipInputStream for instance has non-Java heap memory allocated.
It has been claimed that (IIRC) two thirds of the resources uses in the Java library are implemented in a clearly incorrect manner.
Whilst BufferedOutputStream closes even on an IOException from flush, BufferedWriter closes correctly.
My advice: Close resources as directly as possible and don't let them taint other code. OTOH, you can spend too much time on this issue - if OutOfMemoryError is thrown it's nice to behave nicely, but other aspects of your program are probably a higher priority and library code is probably broken in this situation anyway. But I'd always write:
final FileOutputStream rawOut = new FileOutputStream(file);
try {
OutputStream out = new BufferedOutputStream(rawOut);
... write stuff out ...
out.flush();
} finally {
rawOut.close();
}
(Look: No catch!)
And perhaps use the Execute Around idiom.

The Java SE 7 try-with-resources doesn't seem to be mentioned. It eliminates needing to explicitly do a close completely, and I quite like the idea.
Unfortunately, for Android development this sweet only becomes available by using Android Studio (I think) and targeting Kitkat and above.

Also you dont have to close all nested streams
check this
http://ckarthik17.blogspot.com/2011/02/closing-nested-streams.html

I use to close streams like this, without nesting try-catch in finally blocks
public class StreamTest {
public static void main(String[] args) {
FileOutputStream fos = null;
BufferedOutputStream bos = null;
ObjectOutputStream oos = null;
try {
fos = new FileOutputStream(new File("..."));
bos = new BufferedOutputStream(fos);
oos = new ObjectOutputStream(bos);
}
catch (Exception e) {
}
finally {
Stream.close(oos,bos,fos);
}
}
}
class Stream {
public static void close(AutoCloseable... array) {
for (AutoCloseable c : array) {
try {c.close();}
catch (IOException e) {}
catch (Exception e) {}
}
}
}

Sun's JavaDocs include RuntimeExceptions in their documentation, as shown by InputStream's read(byte[], int, int) method; documented as throwing NullPointerException and IndexOutOfBoundsException.
FilterOutputStream's flush() is only documented as throwing IOException, thus it doesn't actually throw any RuntimeExceptions. Any that could be thrown would most likely be wrapped in an IIOException.
It could still throw an Error, but there's not much you can do about those; Sun recommends that you don't try to catch them.