Hi I am trying to execute external program from Java program and read the stdout message in real time, without waiting for the program to exit. However, i found that there are different stdout behaviour in different .exe program, and I don't know how to handle it.
Example 1:
server1.exe is a console program. When i run it, it will continuously listening on a port. When a client is connected to it, it will generate 1 line of stdout output every 1 second. It will not exit unless i press "ctrl-C".
In a command prompt, I run this:
server1.exe > stdout.out 2> stderr.err
When client is connected to it, I found that stdout.out file will be updated in real time. Even though server1.exe is still running, I can open stdout.out file and read the stdout output in real time.
Example 2:
Similar to server1.exe, server2.exe is also a console program. When i run it, it will also continuously listening on a port. When client is connected to it, it will generate 1 line of stdout output every 1 second. It will not exit unless i press "ctrl-C".
In a command prompt, I run this:
server2.exe > stdout.out 2> stderr.err
Even though client has connected to server2.exe, I found that stdout.out file is empty. As long as server2.exe is still running, no stdout is written to stdout.out file. That file is not updated in real time. When i press ctrl-C, it suddenly write many lines of output to stdout.out file.
Assuming that i press ctrl-C at t=11, it will write all stdout output from t=1 until t=11 into the stdout.out file. Before this, at t=10, the stdout.out file is empty.
The program in example 2 is giving me problem because I am unable to read the stdout in real time in my Java program. My java program is as below:
process = Runtime.getRuntime().exec(command);
input = new BufferedReader(new InputStreamReader(process.getInputStream()));
String inputtext = null;
while ((inputtext = input.readLine()) != null)
{
//print out the text in Real Time, when the .exe program is still running
}
May i know why the program in example 2 will not generate stdout output unless I press ctrl-C?
The strange thing is, when i run that program in console window manually, I can see the stdout output printed on the console window every 1 second. But when I try to read it from Java using inputtext = input.readLine(), inputtext will be null as long as the program is still running (I have tested it by printing out inputtext). When I press ctrl-C, the BufferedReader will suddenly be filled with all the pending stdout output.
How can I read stdout of server2.exe in real time?
The way you describe things, there is some buffering happening in your second server. The server might decide to buffer output internally, unless it is connected to a live interactive console window.
While there may be ways to work around this, I would address this in the server2 source code. Whenever that application writes its once-per-second output, it should flush its output streams afterwards. Perhaps there is some option to enable that behaviour. If there isn't, and if the sources of that program are outside your control, kindly ask the developers to add flushing, in order to allow for better integration.
For short: You need to flush the buffers.
System.out.flush()
You need to do this after every chunk of relevant data written on these streams, try doing it after every line print.
Related
In a Clojure program, how do you read from standard out? I want to do that, or pipe the standard output, to an input stream that I create. The standard output in Clojure is a java.io.PrintWriter .
I have a Samza job, started by a Clojure program. There's also an nrepl server to which I can remotely connect. After connecting, I need to be able to tap into and tail standard out (to which jobs write their output).
1) As per this SO question, with-out-str (see here) lets us temporarily bind *out* (to a java.io.StringWriter), so that your executed code writes to a string. But that doesn't let me tap into the existing *out*.
2) If you look at clojure.java.shell (see here), it gets the JVM's Runtime and exec's a Process on it. From that process, you can get its standard output stream. But again, that's not the default standard out (*out*) I'm looking for.
3) This SO question gets close to what I'm trying to do. But again, I'm connecting to an existing process, and want to tail out its standard output.
Is this possible in Clojure (see here)? Has anyone solved this?
Process output is not a publish subscribe model, so in effect when a process puts a character into it's output buffer, exactly one process gets to pull it off that buffer. If you have a program that was started by a shell that shell process if reading it's output and writing it to a terminal (or reading and ignoring it). If you attach your process after the process that started it and start trying to grab the data, you will most likely not get anything because the parent process will get it first. I just tried this from two terminals:
Terminal 1:
cat
Terminal 2:
ps -ef | grep cat
tail -f /proc/24547/fd/2
Terminal 1:
hello
Terminal 2:
< nothing >
The string "hello" printed to terminal 1, the process that started it.
It's tempting then to say "well what if nobody reads the output, then it will be there for me to get". While this sounds good it runs into the problem that these are fixed sized buffers, so as soon as the output buffer is full the process that is trying to write to it blocks (is prevented from running at all) until someone reads the output to unblock it.
The general solution is to pipe the process you want to tail later to the tee command which writes the output to a file and passes it to whatever was reading it.
command-to-watch arg1 arg2 | tee logfile.potentially-huge
Though if you go this route you should rotate the log file before your disk fills. Make sure you empty the log file with exactly this command
echo > logfile.potentially-huge
or use your program to make a truncate call to the file. simply deleting the file will remove it's name from the log directory without deleting it, it will silently continue to grow taking up disk space and the new file will get no output ever.
This is basically why we built log libraries like log4j (in the 90s) and syslog (in the 80s).
If you still want to get hackish crazy on this, turn to tmux, it can do anything, and changes the way people work with text. In all seriousness you should change the way the other process creates it's output to make it easier to get.
i have strange behavior i have csh file that run java process something like this :
run_server.csh
#!/usr/bin/tcsh
java -Dtest=testparam -cp ${TEST}/lib/device.jar:${TEST}/conf:${TEST}/lib/commons-logging-1.1.1.jar com.device.server
when i run it like this :
run_server.csh& in the putty shell in linux
im getting this massage :
[2] + Suspended (tty output) run_server.csh
Although when i run it without the ampersand the server is running and outputting its log massages into the stdout but the problem is when i press ctr+c the process is killed
another strange thing is if i write wrapper script to run_server.csh
that looked like this run_server_wrapper.csh:
#!/usr/bin/tcsh
run_server.csh &
sleep 5
it does run the server as bg process and the run_server_wrapper.csh it self
getting the massage :
[2] + Suspended (tty output) run_server_wrapper.csh
what is the problem here ?
What's happening is that you're putting the process in the background, but it's still writing to the terminal. The terminal driver doesn't approve of that sort of behavior, and suspends the process when it tries to write to the TTY. The obvious answer, of course, is to stop doing that. If you're putting run_server.csh into the background, you should also redirect the output, like so:
run_server.csh > /path/to/serverlog 2>&1 &
If you want the program to run in the background and you still want to see the output, the usual solution is to redirect the output to a log file as I showed, and then monitor the log file with the tail command.
tail -f /path/to/serverlog
If you really, really want to have something running in the background and still able to write to the TTY, turn off the tostop flag.
stty -tostop
One addition to This isn't my real name's useful answer:
If the process you put into the background needs user input, then it will be suspended until you provide input.
I am mostly concerned with Linux but answers involving windows are welcome.
When I use System.out.println or cout<< what is actually happening and what happens when I do a cout in a gui application (does it go anywhere)?
One case that I am interested in is the Netbeans IDE. When I run a java program in Netbeans what makes it possible for the IDE to "steal" the output from the program and display it?
Update/Sidenote
http://www.linfo.org/standard_output.html
One of the features of standard output is that it has a default
destination but can easily be redirected (i.e., diverted) to another
destination. That default destination is the display screen on the
computer that initiated the program. Because the standard streams are
plain text, they are by definition human readable.
What is meant by "initiate the program"?
I'm not very familiar with how the execution of a program begins but in the case of my netbeans example it's pretty clear that the IDE initiated the program. So what does that mean? When the program is being setup to be executed is there some meta data that is floating around letting the OS know that Netbeans is initiating the program?
When the program gets executed, three special file descriptors: stdin, stdout and stderr are associated to some device to determine how input and output is managed. If you execute a program from a terminal shell, stdin is associated to the keyboard, stdout and stderr to the terminal window. When you execute the program in a development environment usually stdout and stderr are displayed in some special console tabs. In other situations the output goes to some log file or maybe get discarded...
System.out and cout are the objects representing the stdout stream in Java and C++.
In UNIX, I'm supposed to write a Java file that will print "EXIT 1" to the standard error, and then exit with a status of 1.
Here is my approach..
System.err.println("EXIT 1");
System.exit(1);
Is this what I'm supposed to do?
If so, how am I supposed to use it in the Unix shells? When I compile and run it in the bash, it just prints "EXIT 1" (so it does the same thing as System.out.println, why should I use "err"?). What is the "standard error" here?
Every running program has these three streams:
Standard input (stdin), which normally comes from the keyboard. Exposed as System.in
Standard out (stdout), which normally goes to the console. Exposed as
System.out
Standard error (stderr), which normally also goes to the console. Exposed as System.err
Your program is correct – it does print to stderr. But under normal circumstances, the stderr stream goes to the console just like the stdout stream, so they are visually indistinguishable.
However, the reason you should use stderr instead of stdout for error messages, is redirection. That means that you send stderr to a file instead of the console. Meanwhile, stdout will be unaffected, because the two streams are independent.
For example, you can do this in bash, cmd, PowerShell, etc:
$ java Program 2> errors.txt
Now, all output with System.err.println() will end up in errors.txt, while System.out.println() will still go to the screen. This can help with debugging.
There are three data streams associated with nearly every process:
Standard Input: This is the stream of input into a program, either from a terminal, a console, piped output from another process, or some other means.
Standard Error: This is where all debugging and error messages should go. This is so that this sort of information can easily be separately captured from the regular output of a program. Web servers do this, by sending error messages to an error_log file via stderr, while the normal log file would be e. g. access_log.
Standard Output: This is the where all typical, expected output that a user running a program should expect to see said output appear.
Standard Output (stdout) and Standard Error (stderr) are nearly always the first and second output streams coming from a process, respectively. This allows me to do something like /path/to/my/neat/program > logs/program.log 2> logs/program.err and have output and errors nicely sorted.
Here is my problem:
I have a .bat file which has a number of commands. When I invoke this bat file from my java program using processbuilder it starts executing the commands in .bat file line by line. However when it reaches the last line which executes a perl program it hangs and the program never ends. It hangs there forever. Any idea why this might be happening. Just for your information there is no problem with the perl code and I'm able to execute
Does your perl program write to standard output?
I suspect the problem is that a Java process redirects the child process' standard output and standard error output to a pipe.
The receiving end of that pipe is connected to the java program, so any child process invoked from java that writes to standard output / error will block if you fail to read its output.
You must either eliminate the perl program's output or read the child process' output (Process.getInputStream() and/or Process.getErrorStream()) from within your java program.
Hey I was able to fix this problem. All I had to do was to include a statement in my java code which closes the output stream of the process. Without which it was waiting on the command line for the perl program to be executed forever. I'm not exactly sure about the behavior, any information would be help