Develop Reference

Can Clojure Capture Standard Out of Existing Process?

In a Clojure program, how do you read from standard out? I want to do that, or pipe the standard output, to an input stream that I create. The standard output in Clojure is a java.io.PrintWriter .
I have a Samza job, started by a Clojure program. There's also an nrepl server to which I can remotely connect. After connecting, I need to be able to tap into and tail standard out (to which jobs write their output).
1) As per this SO question, with-out-str (see here) lets us temporarily bind *out* (to a java.io.StringWriter), so that your executed code writes to a string. But that doesn't let me tap into the existing *out*.
2) If you look at clojure.java.shell (see here), it gets the JVM's Runtime and exec's a Process on it. From that process, you can get its standard output stream. But again, that's not the default standard out (*out*) I'm looking for.
3) This SO question gets close to what I'm trying to do. But again, I'm connecting to an existing process, and want to tail out its standard output.
Is this possible in Clojure (see here)? Has anyone solved this?

Process output is not a publish subscribe model, so in effect when a process puts a character into it's output buffer, exactly one process gets to pull it off that buffer. If you have a program that was started by a shell that shell process if reading it's output and writing it to a terminal (or reading and ignoring it). If you attach your process after the process that started it and start trying to grab the data, you will most likely not get anything because the parent process will get it first. I just tried this from two terminals:
Terminal 1:
cat
Terminal 2:
ps -ef | grep cat
tail -f /proc/24547/fd/2
Terminal 1:
hello
Terminal 2:
< nothing >
The string "hello" printed to terminal 1, the process that started it.
It's tempting then to say "well what if nobody reads the output, then it will be there for me to get". While this sounds good it runs into the problem that these are fixed sized buffers, so as soon as the output buffer is full the process that is trying to write to it blocks (is prevented from running at all) until someone reads the output to unblock it.
The general solution is to pipe the process you want to tail later to the tee command which writes the output to a file and passes it to whatever was reading it.
command-to-watch arg1 arg2 | tee logfile.potentially-huge
Though if you go this route you should rotate the log file before your disk fills. Make sure you empty the log file with exactly this command
echo > logfile.potentially-huge
or use your program to make a truncate call to the file. simply deleting the file will remove it's name from the log directory without deleting it, it will silently continue to grow taking up disk space and the new file will get no output ever.
This is basically why we built log libraries like log4j (in the 90s) and syslog (in the 80s).
If you still want to get hackish crazy on this, turn to tmux, it can do anything, and changes the way people work with text. In all seriousness you should change the way the other process creates it's output to make it easier to get.

daemonize java program - go background after input some data

I tried to find a solution for the following use case (on Linux):
Start the program, show some information to the stdout, input some information such as username/password.
The program validate the username/password, then goes to background and run as a daemon.
I did not find a way to do this in Java. There are several sulotions to daemonize java program (such as jsvc, or this: http://barelyenough.org/blog/2005/03/java-daemon/ ). But seems they all do not work for this situation, because the program just goes to background from the beginning, there is no chance to input information before it goes to background.

I don't believe that there's a way to do this purely in java. You could make it work by writing an init script which accepted the command line parameters before spawning the java process in the background. You could use -D command line arguments to pass the user input to the java process.

How to handle different stdout behaviour in external program?

Hi I am trying to execute external program from Java program and read the stdout message in real time, without waiting for the program to exit. However, i found that there are different stdout behaviour in different .exe program, and I don't know how to handle it.
Example 1:
server1.exe is a console program. When i run it, it will continuously listening on a port. When a client is connected to it, it will generate 1 line of stdout output every 1 second. It will not exit unless i press "ctrl-C".
In a command prompt, I run this:
server1.exe > stdout.out 2> stderr.err
When client is connected to it, I found that stdout.out file will be updated in real time. Even though server1.exe is still running, I can open stdout.out file and read the stdout output in real time.
Example 2:
Similar to server1.exe, server2.exe is also a console program. When i run it, it will also continuously listening on a port. When client is connected to it, it will generate 1 line of stdout output every 1 second. It will not exit unless i press "ctrl-C".
In a command prompt, I run this:
server2.exe > stdout.out 2> stderr.err
Even though client has connected to server2.exe, I found that stdout.out file is empty. As long as server2.exe is still running, no stdout is written to stdout.out file. That file is not updated in real time. When i press ctrl-C, it suddenly write many lines of output to stdout.out file.
Assuming that i press ctrl-C at t=11, it will write all stdout output from t=1 until t=11 into the stdout.out file. Before this, at t=10, the stdout.out file is empty.
The program in example 2 is giving me problem because I am unable to read the stdout in real time in my Java program. My java program is as below:
process = Runtime.getRuntime().exec(command);
input = new BufferedReader(new InputStreamReader(process.getInputStream()));
String inputtext = null;
while ((inputtext = input.readLine()) != null)
{
//print out the text in Real Time, when the .exe program is still running
}
May i know why the program in example 2 will not generate stdout output unless I press ctrl-C?
The strange thing is, when i run that program in console window manually, I can see the stdout output printed on the console window every 1 second. But when I try to read it from Java using inputtext = input.readLine(), inputtext will be null as long as the program is still running (I have tested it by printing out inputtext). When I press ctrl-C, the BufferedReader will suddenly be filled with all the pending stdout output.
How can I read stdout of server2.exe in real time?

The way you describe things, there is some buffering happening in your second server. The server might decide to buffer output internally, unless it is connected to a live interactive console window.
While there may be ways to work around this, I would address this in the server2 source code. Whenever that application writes its once-per-second output, it should flush its output streams afterwards. Perhaps there is some option to enable that behaviour. If there isn't, and if the sources of that program are outside your control, kindly ask the developers to add flushing, in order to allow for better integration.

For short: You need to flush the buffers.
System.out.flush()
You need to do this after every chunk of relevant data written on these streams, try doing it after every line print.

Running a continuous batch process in Java

I have a batch file which performs the operation of listening to the microphone and converting it to text (i am using pocket sphinx).
The command I am using to run the batch file is pocketsphinx_continuous.exe -dict <dict name> -lm <language model> -hmm <acoustic model location>. The batch file starts off and keeps listening to the microphone. Whenever we finish speaking a sentence it converts it into text on the command prompt. Since this continuously running we terminate this task by Ctrl-C.
I was trying to make this into a standalone Java application. I wanted to run this batch file through Java, so i used Runtime.getRuntime().exec("cmd /c start pocketsphinx_continuous.exe ...") with all the parameters. However strangely, it starts the batch process in a separate command prompt but immediately exits. I tried to use process.waitfor(), however it simply starts executing the batch process and then terminates. I havent called process.destroy, so i am unable to figure out why it exits the batch process.
The other question is since the batch file is running continuously, after every spoken sentence has been transcribed , I wish to get the output in my Java application. I know i can redirect the batch process to a file and then read the file, was just wondering if there is a more direct process. Could you please help me figure out where I am making a mistake.

You should use Process.getInputStream() and Process.getErrorStream() to find out what messages it prints out.
If it is exiting instantly, you might need to get the exit code (Process.waitFor()) and check the error logs from the error stream.
Also, if you can post some of your code we might be able to help. In general, these problems are due to incorrectly configured paths or command strings.
A possible fix would be to use the ProcessBuilder from Java 1.5 thusly:
// Note the lack of "cmd \c" - you shouldn't need to run it in a cmd window!
ProcessBuilder pb = new ProcessBuilder("pocketsphinx_continuous.exe",
"dict", "1121.dic",
"-lm", "1121.lm",
"-hmm", "hub4/hmm");
Process p = pb.start();
// TODO for you:
// 1. Create Threads to handle the input
// 2. Store the Process instance so that you can call p.destroy() later.
// 3. If interested, have a Thread doing p.waitFor() to get the exit code.
As Joachim Sauer mentioned in the comments, this Javaworld article explains a lot of the gotchas associated with the Process API, so have a read through. I haven't had a chance to play with the improvements made in JDK7 to the Process API, and by the looks of things Oracle are improving it again for JDK8 (JEP 102).

You can also use Timer and TimerTask to schedule your batch scan process in background. You can use this to specify infinite running task.

Permission error while trying to access an (server) program started by a Java program

I am starting a server application (normally to be started from the Unix command line) by using Runtime.getRuntime().exec("path/mmserver"). My problem is now that as long as my Java program, which started that server runs, the server is correctly accessible (from command line and other programs). But when my Java program exits the sever is not accessible anymore (the process of the server is still running). I just get such a error message when trying to access the server: "Error: permission_error(flush_output(user_output),write,stream,user_output,errno(32))".
The server is a blackbox for me.
I am just looking for other ways to start a new process. And maybe someone has a hint why I get that permission error (even if one doesn't know what that server exactly is ... you rather won't know it).

I'm guessing your server program is trying to write to standard output or perhaps standard error (System.out / System.err in Java terms) which it implicitly inherited from your Java program but which turn into pumpkins when your Java program goes away.
A simple solution might be for your Java program to exec a shell script which starts your server as a background process (using START (Windows) or & (Unix)) with explicitly redirected I/O streams.
The Java library has recently gotten some nice updates to the Process class (I think) that allow you to do a lot of fiddling with the streams, but I don't have much experience there so I can't offer a detailed suggestion.
EDIT: My suggestion from the middle paragraph. Untested, sorry!
File server-runner.sh:
#!/bin/bash
/path/mmserver >/dev/null &
You'll need to chmod +x server-runner.sh, of course.
Then, from your Java program, you exec the script server-runner.sh rather than your mmserver.
If you want to kill mmserver, you'll have to find it in ps -ux and use kill on the process number.