Develop Reference

How to write a byte array to OutputStream of process builder (Java)

byte[] bytes = value.getBytes();
Process q = new ProcessBuilder("process","arg1", "arg2").start();
q.getOutputStream().write(bytes);
q.getOutputStream().flush();
System.out.println(q.getInputStream().available());
I'm trying to stream file contents to an executable and capture the output but the output(InputStream) is always empty. I can capture the output if i specify the the file location but not with streamed input.
How might I overcome this?

Try wrapping your streams with BufferedInputStream() and BufferedOutputStream():
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getOutputStream%28%29
Implementation note: It is a good idea for the output stream to be buffered.
Implementation note: It is a good idea for the input stream to be buffered.
Even with buffered streams, it is still possible for the buffer to fill if you're dealing with large amounts of data, you can deal with this by starting a separate thread to read from q.getInputStream(), so you can still be reading from the process while writing to the process.

Perhaps the program you execute only starts its work when it detects the end of its input data. This is normally done by waiting for an EOF (end-of-file) symbol. You can send this by closing the output stream to the process:
q.getOutputStream().write(bytes);
q.getOutputStream().close();
Try this together with waiting for the process.

I dont know if something else may also be wrong here, but the other process ("process") does not even have time to respond, you are not waiting for it (the method available() does not block). To try this out you can first insert a sleep(2000) after the flush(), and if that works you should switch to query'ing q.getInputStream().available() multiple times with short pauses in between.

I think, you have to wait, until the process finished.
I implemented something like this this way:
public class ProcessReader {
private static final int PROCESS_LOOP_SLEEP_MILLIS = 100;
private String result;
public ProcessReader(Process process) {
BufferedReader resultReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder resultOutput = new StringBuilder();
try {
while (!checkProcessTerminated(process, resultReader, resultOutput)) {
}
} catch (Exception ex1) {
throw new RuntimeException(ex1);
}
result = resultOutput.toString();
}
public String getResult(){
return result;
}
private boolean checkProcessTerminated(Process process, BufferedReader resultReader, StringBuilder resultOutput) throws Exception {
try {
int exit = process.exitValue();
return true;
} catch (IllegalThreadStateException ex) {
Thread.sleep(PROCESS_LOOP_SLEEP_MILLIS);
} finally {
while (resultReader.ready()) {
String out = resultReader.readLine();
resultOutput.append(out).append("\n");
}
}
return false;
}
}
I just removed now some specific code, that you dont need, but it should work, try it.
Regards

Java, Junit - Capture the standard input / Output for use in a unit test [duplicate]

This question already has answers here:
JUnit test for System.out.println()
(14 answers)
Closed 8 years ago.
I'm writing integration tests using JUnit to automate the testing of a console based application. The application is homework but this part isn't the homework. I want to automate these tests to be more productive -- I don't want to have to go back and retest already tested parts of the application. (Standard reasons to use Unit tests)
Anyway, I can't figure out or find an article on capturing the output so that I can do assertEquals on it nor providing automated input. I don't care if the output/input goes to the console/output pane. I only need to have the test execute and verify the the output is what is expected given the input.
Anyone have an article or code to help out with this.

Use System.setOut() (and System.setErr()) to redirect the output to an arbitrary printstream - which can be one that you read from programmatically.
For example:
final ByteArrayOutputStream myOut = new ByteArrayOutputStream();
System.setOut(new PrintStream(myOut));
// test stuff here...
final String standardOutput = myOut.toString();

The System class has methods setIn(), setOut() and setErr() that allow you to set the standard input, output and error streams, e.g. to a ByteArrayOutputStream that you can inspect at will.

Here is the solution in place of ByteArrayOutputStream. It does not add anything to the idea of System.setOut. Rather, I want to share the implementation that is better than capturing everything into ByteArrayOutputStream. I prefer to capture only selected information and let all log messages to appear in the console as they are logged rather than capturing everything into a balckbox (of which size?) for later processing.
/**
* Once started, std output is redirected to this thread.
* Thread redirects all data to the former system.out and
* captures some strings.*/
static abstract class OutputCaputre extends Thread {
// overrdie these methods for System.err
PrintStream getDownstream() { return System.out;}
void restoreDownstream() { System.setOut(downstream);}
// will be called for every line in the log
protected abstract void userFilter(String line);
final PrintStream downstream;
public final PipedInputStream pis;
private final PipedOutputStream pos;
OutputCaputre() throws IOException {
downstream = getDownstream();
pos = new PipedOutputStream();
pis = new PipedInputStream(pos);
System.setOut(new PrintStream(pos));
start();
}
public void run() {
try {
BufferedReader br = new BufferedReader(new InputStreamReader(pis));
// once output is resotred, we must terminate
while (true) {
String line = br.readLine();
if (line == null) {
return;
}
downstream.println(line);
userFilter(line);
}
} catch (IOException e) {
e.printStackTrace();
}
}
public void terminate() throws InterruptedException, IOException {
restoreDownstream(); // switch back to std
pos.close(); // there will be no more data - signal that
join(); // and wait until capture completes
}
};
Here is an example of using the class:
OutputCaputre outputCapture = new OutputCaputre() {
protected void userFilter(String line) {
downstream.println("Capture: " + line);
}
};
System.out.println("do you see me captured?");
// here is your test
outputCapture.terminate(); // finally, stop capturing

design for a wrapper around command-line utilities

im trying to come up with a design for a wrapper for use when invoking command line utilities in java. the trouble with runtime.exec() is that you need to keep reading from the process' out and err streams or it hangs when it fills its buffers. this has led me to the following design:
public class CommandLineInterface {
private final Thread stdOutThread;
private final Thread stdErrThread;
private final OutputStreamWriter stdin;
private final History history;
public CommandLineInterface(String command) throws IOException {
this.history = new History();
this.history.addEntry(new HistoryEntry(EntryTypeEnum.INPUT, command));
Process process = Runtime.getRuntime().exec(command);
stdin = new OutputStreamWriter(process.getOutputStream());
stdOutThread = new Thread(new Leech(process.getInputStream(), history, EntryTypeEnum.OUTPUT));
stdOutThread.setDaemon(true);
stdOutThread.start();
stdErrThread = new Thread(new Leech(process.getErrorStream(), history, EntryTypeEnum.ERROR));
stdErrThread.setDaemon(true);
stdErrThread.start();
}
public void write(String input) throws IOException {
this.history.addEntry(new HistoryEntry(EntryTypeEnum.INPUT, input));
stdin.write(input);
stdin.write("\n");
stdin.flush();
}
}
And
public class Leech implements Runnable{
private final InputStream stream;
private final History history;
private final EntryTypeEnum type;
private volatile boolean alive = true;
public Leech(InputStream stream, History history, EntryTypeEnum type) {
this.stream = stream;
this.history = history;
this.type = type;
}
public void run() {
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String line;
try {
while(alive) {
line = reader.readLine();
if (line==null) break;
history.addEntry(new HistoryEntry(type, line));
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
my issue is with the Leech class (used to "leech" the process' out and err streams and feed them into history - which acts like a log file) - on the one hand reading whole lines is nice and easy (and what im currently doing), but it means i miss the last line (usually the prompt line). i only see the prompt line when executing the next command (because there's no line break until that point).
on the other hand, if i read characters myself, how can i tell when the process is "done" ? (either complete or waiting for input)
has anyone tried something like waiting 100 millis since the last output from the process and declaring it "done" ?
any better ideas on how i can implement a nice wrapper around things like runtime.exec("cmd.exe") ?

Use PlexusUtils it is used by Apache Maven 2 to execute all external processes.

I was looking for the same thing myself, and I found a Java port of Expect, called ExpectJ. I haven't tried it yet, but it looks promising

I would read the input in with the stream and then write it into a ByteArrayOutputStream. The byte array will continue to grow until there are no longer any available bytes to read. At this point you will flush the data to history by converting the byte array into a String and splitting it on the platform line.separator. You can then iterate over the lines to add history entries. The ByteArrayOutputStream is then reset and the while loop blocks until there is more data or the end of stream is reached (probably because the process is done).
public void run() {
ByteArrayOutputStream out = new ByteArrayOutputStream();
int bite;
try {
while((bite = stream.read()) != -1) {
out.write(bite);
if (stream.available() == 0) {
String string = new String(out.toByteArray());
for (String line : string.split(
System.getProperty("line.separator"))) {
history.addEntry(new HistoryEntry(type, line));
}
out.reset();
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
This will make sure you pick up that last line of input and it solves your problem of knowing when the stream is ended.

What is the best way, if possible, to send information from a Java PrintStream to a JTextPane?

In Java, I have a package that translates XML metadata from one standard to another. This package is ultimately accessed through a single function and sends all of its output through a PrintStream object. The output sent is just a status of each file and whether or not it was translated.
This is pretty fine and dandy if I'm just printing to System.out, but I'm actually wanting to print this to a JTextPane while it translates (kind of like a progress text box). It wouldn't be a big deal to just print the status after it was done translating the XML, but since there may be thousands of XML files, that's just not feasible.
One thing that I've tried is to use a thread that takes all of the information from the PrintStream (which is attached to a ByteArrayOutputStream) and let it send any new information to the text pane. Unfortunately, this still sends the information all at once at the end of the translation. This does work correctly for System.out.
Here's the code that does the translation and tries to show the output:
public class ConverterGUI extends javax.swing.JFrame {
boolean printToResultsBox = false;
PrintStream printStream = null;
ByteArrayOutputStream baos = null;
private class ResultsPrinter implements Runnable {
public ResultsPrinter() {
baos = new ByteArrayOutputStream();
printStream = new PrintStream(baos);
}
public void run() {
String tempString = "";
while (printToResultsBox) {
try {
if (!baos.toString().equals(tempString)) {
tempString = baos.toString();
resultsBox.setText(tempString);
}
} catch (Exception ex) {
}
}
}
}
...
ResultsPrinter rp = new ResultsPrinter();
Thread thread = new Thread(rp);
thread.start();
// Do the translation.
try {
printToResultsBox = true;
boolean success = false;
TranslationEngine te = new TranslationEngine();
// fileOrFolderToConvert is a text box in the GUI.
// linkNeeded and destinationFile are just parameters for the translation process.
success = te.translate(fileOrFolderToConvert.getText(), linkNeeded, destinationFile, printStream);
if (success) {
printStream.println("File/folder translation was a success.");
}
resultsBox.setText(baos.toString());
} catch (Exception ex) {
printStream.println("File translation failed.");
} finally {
printToResultsBox = false;
}
...
}
Ultimately, this code prints out to the JTextPane just fine after all the translation is done but not during. Any suggestions? Do I need to change the PrintStream to something else?

The problem with the way your thread works is you are not on the UI event thread when updating your results box. Take a look at the SwingWorker class. Or you could even use the SwingUtilities.invokeAndWait

Capturing large amounts of output from Apache Commons-Exec

I am writing a video application in Java by executing ffmpeg and capturing its output to standard output. I decided to use Apache Commons-Exec instead of Java's Runtime, because it seems better. However, I am have a difficult time capturing all of the output.
I thought using pipes would be the way to go, because it is a standard way of inter-process communication. However, my setup using PipedInputStream and PipedOutputStream is wrong. It seems to work, but only for the first 1042 bytes of the stream, which curiously happens to be the value of PipedInputStream.PIPE_SIZE.
I have no love affair with using pipes, but I want to avoid use disk I/O (if possible), because of speed and volume of data (a 1m 20s video at 512x384 resolution produces 690M of piped data).
Thoughts on the best solution to handle large amounts of data coming from a pipe? My code for my two classes are below. (yes, sleep is bad. Thoughts on that? wait() and notifyAll() ?)
WriteFrames.java
public class WriteFrames {
public static void main(String[] args) {
String commandName = "ffmpeg";
CommandLine commandLine = new CommandLine(commandName);
File filename = new File(args[0]);
String[] options = new String[] {
"-i",
filename.getAbsolutePath(),
"-an",
"-f",
"yuv4mpegpipe",
"-"};
for (String s : options) {
commandLine.addArgument(s);
}
PipedOutputStream output = new PipedOutputStream();
PumpStreamHandler streamHandler = new PumpStreamHandler(output, System.err);
DefaultExecutor executor = new DefaultExecutor();
try {
DataInputStream is = new DataInputStream(new PipedInputStream(output));
YUV4MPEGPipeParser p = new YUV4MPEGPipeParser(is);
p.start();
executor.setStreamHandler(streamHandler);
executor.execute(commandLine);
} catch (IOException e) {
e.printStackTrace();
}
}
}
YUV4MPEGPipeParser.java
public class YUV4MPEGPipeParser extends Thread {
private InputStream is;
int width, height;
public YUV4MPEGPipeParser(InputStream is) {
this.is = is;
}
public void run() {
try {
while (is.available() == 0) {
Thread.sleep(100);
}
while (is.available() != 0) {
// do stuff.... like write out YUV frames
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}

The problem is in the run method of YUV4MPEGPipeParser class. There are two successive loops. The second loop terminates immediately if there are no data currently available on the stream (e.g. all input so far was processed by parser, and ffmpeg or stream pump were not fast enough to serve some new data for it -> available() == 0 -> loop is terminated -> pump thread finishes).
Just get rid of these two loops and sleep and just perform a simple blocking read() instead of checking if any data are available for processing. There is also probably no need for wait()/notify() or even sleep() because the parser code is started on a separate thread.
You can rewrite the code of run() method like this:
public class YUV4MPEGPipeParser extends Thread {
...
// optimal size of buffer for reading from pipe stream :-)
private static final int BUFSIZE = PipedInputStream.PIPE_SIZE;
public void run() {
try {
byte buffer[] = new byte[BUFSIZE];
int len = 0;
while ((len = is.read(buffer, 0, BUFSIZE) != -1) {
// we have valid data available
// in first 'len' bytes of 'buffer' array.
// do stuff.... like write out YUV frames
}
} catch ...
}
}