Develop Reference

System.out.print() method doesn't flush [duplicate]

I have a java application that takes input via a Scanner reading System.in, and gives output via System.out. The scanner is always active, it does not terminate without using Ctrl+C via the terminal or ending the process in an IDE.
I'm running into some behavior that leads me to believe System.out is not flushing properly.
In my code there are the lines:
System.out.print(",\\" + '\n');
System.out.print(" " + someString);
(someString does not contain a newline character)
When I execute this code via the terminal, the whitespace and someString are not printed to the terminal. However in my IDE's console, it is. (IntelliJ)
If I change the second statement to println instead of print, it works fine, but it does append a new line between one execution of the code and the next, which is not workable here. (Maybe there's something I can do with a carriage return?)
This sounds a lot like the output isn't being flushed, as its only System.out.print that has the trouble. However, adding System.out.flush() after the print statement does not cause it to print.

This sounds a lot like the output isn't being flushed, as its only System.out.print that has the trouble.
Typically System.out and System.err are configured differently. (System.err is typically not buffered, and System.out is typically buffered.) However, the javadocs do not specify the flushing behavior of either streams. This could explain the differences in behavior between the (real) console and running in an IDE.
For info, here is how the streams are initialized in Java 8:
FileInputStream fdIn = new FileInputStream(FileDescriptor.in);
FileOutputStream fdOut = new FileOutputStream(FileDescriptor.out);
FileOutputStream fdErr = new FileOutputStream(FileDescriptor.err);
setIn0(new BufferedInputStream(fdIn));
setOut0(newPrintStream(fdOut, props.getProperty("sun.stdout.encoding")));
setErr0(newPrintStream(fdErr, props.getProperty("sun.stderr.encoding")));
private static PrintStream ewPrintStream(FileOutputStream fos, String enc) {
if (enc != null) {
try {
return new PrintStream(new BufferedOutputStream(fos, 128),
true, enc);
} catch (UnsupportedEncodingException uee) {}
}
return new PrintStream(new BufferedOutputStream(fos, 128), true);
}
As you can see, System.out is initialized as buffered with autoflush enabled.
However, adding System.out.flush() after the print statement does not cause it to print.
Are you sure about that? A flush() should flush any buffered output.
I suggest that the problem is actually somewhere else; e.g. the print or flush calls are not happening ... for some reason.
It is also possible that some of your problems are due to this:
System.out.print(",\\" + '\n');
As #javaguy points out, a newline character is a platform specific line separator. On some platforms, the console requires something different. The simplest platform independent way to tell the console to do a line break is:
System.out.println(",\\");
Or putting it all together:
System.out.println(",\\");
System.out.print(" " + someString);
System.out.flush(); // This is necessary ... and should work.

Assuming that you are running the program in Windows, the carriage
return (\r) and line feed (\n) together need to be added to print
the new line as below:
System.out.print(",\\" + '\r\n');
System.out.print(" " + someString);

System.out.println() places the cursor to the next line after execution. so you don't need to add "\n". That's probably why a line of space is between the lines.
Just do this:
System.out.println(",\\");
System.out.println(" " + someString);

Why I couldn't write all the characters to the file with java FileWriter? [duplicate]

I have a Java program that reads some text from a file, line by line, and writes new text to an output file. But not all the text I write to my BufferedWriter appears in the output file after the program has finished. Why is that?
The details: the program takes a CSV text document and converts it into SQL commands to insert the data into a table. The text file has more than 10000 lines which look similar to following:
2007,10,9,1,1,1006134,19423882
The program seems to work fine except it just stops in the file randomly half way through creating a new SQL statement having printed it into the SQL file. It looks something like:
insert into nyccrash values (2007, 1, 2, 1, 4, 1033092, 259916);
insert into nyccrash values (2007, 1, 1, 1, 1, 1020246, 197687);
insert into nyccrash values (2007, 10, 9, 1
This happens after about 10000 lines but several hundred lines before the end of the file. Where the break happens is between a 1 and a ,. However, the characters doesn't seem important because if I change the 1 to a 42 the last thing written to the new file is 4, which is cutting off the 2 from that integer. So it seems like the reader or writer must just be dying after writing/reading a certain amount.
My Java code is as follows:
import java.io.*;
public class InsertCrashData
{
public static void main (String args[])
{
try
{
//Open the input file.
FileReader istream = new FileReader("nyccrash.txt");
BufferedReader in = new BufferedReader(istream);
//Open the output file.
FileWriter ostream = new FileWriter("nyccrash.sql");
BufferedWriter out = new BufferedWriter(ostream);
String line, sqlstr;
sqlstr = "CREATE TABLE nyccrash (crash_year integer, accident_type integer, collision_type integer, weather_condition integer, light_condition integer, x_coordinate integer, y_coordinate integer);\n\n";
out.write(sqlstr);
while((line = in.readLine())!= null)
{
String[] esa = line.split(",");
sqlstr = "insert into nyccrash values ("+esa[0]+", "+esa[1]+", "+esa[2]+", "+esa[3]+", "+esa[4]+", "+esa[5]+", "+esa[6]+");\n";
out.write(sqlstr);
}
}
catch(Exception e)
{
System.out.println(e);
}
}
}

You need to close your OutputStream which will flush the remainder of your data:
out.close();
The default buffer size for BufferedWriter is 8192 characters, large enough to easily hold hundreds of lines of unwritten data.

You must close() your BufferedWriter. You must close() your BufferedWriter because it IS-A Writer and thus implements AutoCloseable, which means (emphasis added) it is
A resource that must be closed when it is no longer needed.
Some people say you must first call flush() for your BufferedWriter before calling close(). They are wrong. The documentation for BufferedWriter.close() notes that it "Closes the stream, flushing it first" (emphasis added).
The documented semantics of flushing (flush()) are
Flushes this stream by writing any buffered output to the underlying stream
So, you must close, and close will flush any buffered output.
Your output file does not include all the text you wrote to your BufferedWriter because it stored some of that text in a buffer. The BufferedWriter never emptied that buffer, passing it through to the file, because you never told it to do so.
Since Java 7, the best way to ensure an AutoCloseable resource, such as a BufferedWriter, is closed when it is not longer need is to use automatic resource management (ARM), also known as try-with-resources:
try (BufferedWriter out = new BufferedWriter(new FileWriter(file))) {
// writes to out here
} catch (IOException ex) {
// handle ex
}
You must also close your BufferedReader when it is no longer need, so you should have nested try-with-resources blocks:
try (BufferedReader in = new BufferedReader(new FileReader("nyccrash.txt")) {
try (BufferedWriter out = new BufferedWriter(new FileWriter("nyccrash.sql"))) {
// your reading and writing code here
}
} catch (IOException ex) {
// handle ex
}
Do not be tempted (as other answers here suggest) just to call close() at the end of your method, when your code has "finished" using the writer. That will not work if your writing code throws an exception, and in particular if it throws an IOException.

A resource that must be closed when it is no longer needed.
finally {
out.close();//this would resolve the issue
}
Some things to consider:
BufferedWriter.close() flushes the buffer to the underlying stream, so if you forget to flush() and don't close, your file may not have all the text you wrote to it.
BufferedWriter.close() also closes the wrapped Writer. When that's a FileWriter, this will ultimately close a FileOutputStream and tell the OS that you're done writing to the file.
The garbage collector will automatically call close(), not on the BufferedWriter or the wrapped FileWriter, but on the FileOuputStream. So the OS will be happy, but you have to wait for the GC.
However, you always want to release OS resources as soon as you no longer need them. This goes for open files, database connections, print queues ... anything. Trust me on this one.
BufferedWriter.close() does clear up the internal character buffer, so that memory will be available for garbage collection, even while the BufferedWriter itself remains in scope.
So, Always close your resources (not just files) when you're done with them.
If you really want a peek under the covers, most of the Java API's source is available. BufferedWriter is here.

Your code does not appear to be closing the writer after you've finished writing to it. Add an out.close() (preferably in a finally block) and it should work properly.

you dint close your BufferedWriter.close it inside a finally block
finally {
out.close();//this would resolve the issue
}

Always close your resources (not just files) when you're done with them.
finally {
out.close();//this would resolve the issue
}
There might be situations when you want to flush the buffer without closing the file. In these situations you can use the flush-method.

Since you're using BufferedWriter you can also flush the buffer when appropriate:
out.flush()
This will write the rest of the buffer to the actual file. Close-method also flushes the buffer and closes the file.
out.close()
There might be situations when you want to flush the buffer without closing the file. In these situations you can use the flush-method.
You can also use BuffredWriter's newline-method instead of adding \n to the end of the line. Newline-method uses system specific line separator so your code works on different platforms.
out.newLine()

According to documentation it is no use calling flush() method. If you intent to use FileWriter then flush() would help you out.
Basically in this case, you just need to close, BufferedWriter.close() only. This will flush the remainder of your data.
create finally block and put the close method inside so that it will put all data without missing.
finally {
out.close();
}

java on linux - Have to press CtrlD twice

String str = "";
BufferedReader bfr = new BufferedReader(new InputStreamReader(System.in));
int tempint = 0;
try {
//The program cannot get out from this while loop!
while((tempint = bfr.read()) != -1){
str += Character.toString((char)tempint);
}
}
catch(IOException ioe) {
System.out.println(ioe);
}
//Print the input
System.out.println(str);
This is the code I wrote for reading user's input from standard input. This piece of code is extracted as playinput.jar
And I have written a script play to run this jar. But after I run ./play in terminal(linux) and finish my input, which does not contain enter, I have to press Ctrl+D twice to get the input printed out.
The same problem occurred when I ran another script called check, which will simply invoke ./play and send some input via stdin. After I ran ./check, it just hanged there and the input cannot be printed out.
Could anyone help fix this problem? Thank you:)

That's just how the Linux terminal works. It has nothing to do with your Java code.
If you test it out with a command like cat > textfile, you will find that unless you are at the beginning of a line, ^D doesn't immediately end the file as you might expect it to. (I don't know all the details of this behavior, but that's the gist of it.)
The convention for Linux is that a text file always ends with a newline. You can run into problems like this if you don't follow the convention.
However, I'm not sure about your problem with the program hanging when you send it data using redirection. That part is more surprising to me, since it's not interactive so the terminal behavior shouldn't be an issue.

How do I force Windows line endings in Java app?

I have a Java app that writes to a file with:
BufferedWriter bw = new BufferedWriter(new FileWriter(outputFile));
bw.write(line + lineTermination);
Line termination is defined as:
\r\n
I get the odd, mysterious blank line inserted into my file.
I get no extra lines if I change my code to:
bw.write(line);
bw.newLine();
But I want to force a specific line ending, not use System property. Clients specifically request a line ending character - some even have |. Its not a viable fix to just use \n.
Here is an snippet of the data with missing line:
"KABE","14/01/11","14:35","14:56","1987","US","SS","CO","MARRIED WITH CHILDREN","","EINE SCHRECKLICH NETTE FAMILIE","","N","10","","12","O'NEILL ED","13","SAGAL KATEY"
"PRO7","14/01/11","14:35","14:55","2001","US","SS","CO","SCRUBS","","SCRUBS DIE ANFAENGER","","C","10","BERNSTEIN ADAM","12","BRAFF ZACH","13","CHALKE SARAH"
Thanks for your time :-)

You can call
System.setProperty("line.separator", "\r\n");
in order to set the system property inside your code.

Launching wkhtmltopdf from Runtime.getRuntime().exec(): never terminates?

I'm launching wkhtmltopdf from within my Java app (part of a Tomcat server, running in debug mode within Eclipse Helios on Win7 64-bit): I'd like to wait for it to complete, then Do More Stuff.
String cmd[] = {"wkhtmltopdf", htmlPathIn, pdfPathOut};
Process proc = Runtime.getRuntime().exec( cmd, null );
proc.waitFor();
But waitFor() never returns. I can still see the process in the Windows Task Manager (with the command line I passed to exec(): looks fine). AND IT WORKS. wkhtmltopdf produces the PDF I'd expect, right where I'd expect it. I can open it, rename it, whatever, even while the process is still running (before I manually terminate it).
From the command line, everything is fine:
c:\wrk>wkhtmltopdf C:\Temp\foo.html c:\wrk\foo.pdf
Loading pages (1/6)
Counting pages (2/6)
Resolving links (4/6)
Loading headers and footers (5/6)
Printing pages (6/6)
Done
The process exits just fine, and life goes on.
So what is it about runtime.exec() that's causing wkhtmltopdf to never terminate?
I could grab proc.getInputStream() and look for "Done", but that's... vile. I want something that is more general.
I've calling exec() with and without a working directory. I've tried with and without an empty "env" array. No joy.
Why is my process hanging, and what can I do to fix it?
PS: I've tried this with a couple other command line apps, and they both exhibit the same behavior.
Further exec woes.
I'm trying to read standard out & error, without success. From the command line, I know there's supposed to be something remarkably like my command line experience, but when I read the input stream returned by proc.getInputStream(), I immediately get an EOL (-1, I'm using inputStream.read()).
I checked the JavaDoc for Process, and found this
The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the [b]subprocess to block, and even deadlock[/b].
Emphasis added. So I tried that. The first 'read()' on the Standard Out inputStream blocked until I killed the process...
WITH WKHTMLTOPDF
With the generic command line ap & no params so it should "dump usage and terminate", it sucks out the appropriate std::out, then terminates.
Interesting!
JVM version issue? I'm using 1.6.0_23. The latest is... v24. I just checked the change log and don't see anything promising, but I'll try updating anyway.
Okay. Don't let the Input Streams fill or they'll block. Check. .close() can also prevent this, but isn't terribly bright.
That works in general (including the generic command line apps I've tested).
In specific however, it falls down. It appears that wkhtmltopdf is using some terminal manipulation/cursor stuff to do an ASCII-graphic progress bar. I believe this is causing the inputStream to immediately return EOF rather than giving me the correct values.
Any ideas? Hardly a deal-breaker, but it would definitely be Nice To Have.

I had the same exact issue as you and I solved it. Here are my findings:
For some reason, the output from wkhtmltopdf goes to STDERR of the process and NOT STDOUT. I have verified this by calling wkhtmltopdf from Java as well as perl
So, for example in java, you would have to do:
//ProcessBuilder is the recommended way of creating processes since Java 1.5
//Runtime.getRuntime().exec() is deprecated. Do not use.
ProcessBuilder pb = new ProcessBuilder("wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
Process process = pb.start();
BufferedReader errStreamReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//not "process.getInputStream()"
String line = errStreamReader.readLine();
while(line != null)
{
System.out.println(line); //or whatever else
line = reader.readLine();
}
On a side note, if you spawn a process from java, you MUST read from the stdout and stderr streams (even if you do nothing with it) because otherwise the stream buffer will fill and the process will hang and never return.
To futureproof your code, just in case the devs of wkhtmltopdf decide to write to stdout, you can redirect stderr of the child process to stdout and read only one stream like this:
ProcessBuilder pb = new ProcessBuilder("wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader inStreamReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
Actually, I do this in all the cases where I have to spawn an external process from java. That way I don't have to read two streams.
You should also read the streams of the spawned process in different threads if you dont want your main thread to block, since reading from streams is blocking.
Hope this helps.
UPDATE: I raised this issue in the project page and was replied that this is by design because wkhtmltopdf supports giving the actual pdf output in STDOUT. Please see the link for more details and java code.

A process has 3 streams: input, output and error. you can read both output and error stream at the same time using separate processes. see this question and its accepted answer and also this one for example.

You should read from the streams in a different thread.

final Semaphore semaphore = new Semaphore(numOfThreads);
final String whktmlExe = tmpwhktmlExePath;
int doccount = 0;
try{
File fileObject = new File(inputDir);
for(final File f : fileObject.listFiles()) {
if(f.getAbsolutePath().endsWith(".html")) {
doccount ++;
if(doccount >500 ) {
LOG.info(" done with conversion of 1000 docs exiting ");
break;
}
System.out.println(" inside for before "+semaphore.availablePermits());
semaphore.acquire();
System.out.println(" inside for after "+semaphore.availablePermits() + " ---" +f.getName());
new java.lang.Thread() {
public void run() {
try {
String F_ = f.getName().replaceAll(".html", ".pdf") ;
ProcessBuilder pb = new ProcessBuilder(whktmlExe , f.getAbsolutePath(), outPutDir + F_ .replaceAll(" ", "_") );//"wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader errStreamReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line = errStreamReader.readLine();
while(line != null)
{
System.err.println(line); //or whatever else
line = errStreamReader.readLine();
}
System.out.println("after completion for ");
} catch (Exception e) {
e.printStackTrace();
}finally {
System.out.println(" in finally releasing ");
semaphore.release();
}
}
}.start();
}
}
}catch (Exception ex) {
LOG.error(" *** Error in pdf generation *** ", ex);
}
while (semaphore.availablePermits() < numOfThreads) {//till all threads finish
LOG.info( " Waiting for all threads to exit "+ semaphore.availablePermits() + " --- " +( numOfThreads - semaphore.availablePermits()));
java.lang.Thread.sleep(10000);
}