I have some issues regarding ProcessBuilder.
The program is basically a simple wrapper invoking a command line script.
When running the script on its own via the terminal, the memory consumption stays below 2G.
When running the script via the java wrapper, the memory consumption explodes and even 8G is quickly filled up, resulting in out-of-memory errors.
The code to launch the process is simply:
public static int execute(String command) throws IOException
{
System.out.println("Executing: " + command);
ProcessBuilder pb = new ProcessBuilder(command.split(" +"));
Process p = pb.start();
// display any output in stderr or stdout
StreamConsumer stderr = new StreamConsumer(p.getErrorStream(), "stderr");
StreamConsumer stdout = new StreamConsumer(p.getInputStream(), "stdout");
new Thread(stderr).start();
new Thread(stdout).start();
try {
return p.waitFor();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
The StreamConsumer class is simply a class which consumes the stdout/stderr streams and display them on the console.
...the question is: why on earth does the memory consumption explode?
Regards,
Arnaud
Edit:
Whether I use ProcessBuilder or
Runtime.getRuntime.exec(...), the
result is the same.
The memory bursts tend to appear during unix 'sort' invoked by the
shell script called:
sort big-text-file > big-text-file.sorted
Edit 2 on request of Jim Garrison:
Ok, here is the StreamConsumer class which I omitted because it is rather simple:
class StreamConsumer implements Runnable
{
InputStream stream;
String descr;
StreamConsumer(InputStream stream, String descr) {
this.stream = stream;
this.descr = descr;
}
#Override
public void run()
{
String line;
BufferedReader brCleanUp =
new BufferedReader (new InputStreamReader (stream));
try {
while ((line = brCleanUp.readLine ()) != null)
System.out.println ("[" + descr + "] " + line);
brCleanUp.close();
} catch (IOException e) {
// TODO: handle exception
}
}
}
if you change your command like this :
sort -o big-text-file.sorted big-text-file
is it always the same ?
Maybe its because those StreamConsumer threads are not daemons so they don't die and get garbage collected when your processes return? You could try:
//...
final StreamConsumer stderr = new StreamConsumer(p.getErrorStream(), "stderr");
final StreamConsumer stdout = new StreamConsumer(p.getInputStream(), "stdout");
final Thread stderrThread = new Thread(stderr);
final Thread stdoutThread = new Thread(stdout);
stderrThread.setDaemon(true);
stdoutThread.setDaemon(true);
stderrThread.start();
stdoutThread.start();
//...
Is this behavior happening for single invocation or after doing this many times?
Related
I have a program where I use named pipes to share info with an external executable:
Process p = Runtime.getRuntime().exec("mkfifo /tmp/myfifo");
p.waitFor();
Process cat = Runtime.getRuntime().exec("cat /tmp/myfifo");
BufferedWriter fifo = new BufferedWriter(
new OutputStreamWriter(new FileOutputStream("/tmp/myfifo")));
fifo.write("Hello!\n");
fifo.close();
cat.waitFor();
When I execute this, the program hangs waiting for cat to finish. It seems that cat has not 'realized' that the fifo was closed.
I tried running $> touch /tmp/myfifo on the terminal, and it worked to 'unhang' the process and it finishing properly; but when I added code to run this within my program, it would remain hanging:
fifo.close();
Process touch = Runtime.getRuntime().exec("touch /tmp/myfifo");
touch.waitFor();
cat.waitFor();
The process will still hang waiting for cat to finish. I'm not sure what to do now.
NOTE - I have already added code to consume the output of the cat command, but the problem does not seem to be there.
Anyone know a workaround/fix for this?
some native platforms only provide limited buffer size for standard
input and output streams, failure to promptly write the input stream
or read the output stream of the subprocess may cause the subprocess
to block, and even deadlock.you need to consume the output like print it on stdout something or file
try something like this
Process cat = Runtime.getRuntime().exec("cat /tmp/myfifo");
new Thread(new Reader(cat.getErrorStream(), System.err)).start();
new Thread(new Reader(cat.getInputStream(), System.out)).start();
int returnCode = cat.waitFor();
System.out.println("Return code = " + returnCode);
class Reader implements Runnable
{
public Reader (InputStream istrm, OutputStream ostrm) {
this.istrm = istrm;
this.ostrm = ostrm;
}
public void run() {
try
{
final byte[] buffer = new byte[1024];
for (int length = 0; (length = istrm.read(buffer)) != -1; )
{
ostrm.write(buffer, 0, length);
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
private final OutputStream ostrm;
private final InputStream istrm;
}
Currently having troubles to redirect the output of a small windows batch console to a log file. My Java application needs to start the Runtime.exec() call without waiting it to finish and still log the output. Here is my logger class :
public class BatchThreadLogger extends Thread {
private Process process;
private String logFilePath;
private static final Logger logger = Logger.getLogger(BatchThreadLogger.class);
public BatchThreadLogger(Process process, String logFilePath) {
this.process = process;
this.logFilePath = logFilePath;
}
public void run() {
try {
// create logging file
File file = new File(logFilePath);
file.createNewFile();
// create a writer object
OutputStream os = new FileOutputStream(file);
PrintWriter pw = new PrintWriter(os);
// catch the process output in an InputStream
InputStreamReader isr = new InputStreamReader(process.getInputStream());
BufferedReader br = new BufferedReader(isr);
// wait for the process to complete
int processStatus = process.waitFor();
// redirect the output to the log file
String line = null;
while ((line = br.readLine()) != null) {
pw.println(line);
}
// add a small message with the return code to the log
pw.println("********************************************");
pw.println("********************************************");
pw.println("Batch call completed with return status " + processStatus);
pw.flush();
os.close();
}
catch (IOException e) {
logger.error("IOException raised during batch logging on file " + logFilePath, e);
}
catch (InterruptedException e) {
logger.error("InterruptedException raised during batch process execution", e);
}
}
}
My call is quite simple :
Process process = Runtime.getRuntime().exec(command);
BatchThreadLogger logger = new BatchThreadLogger(process, logFilePath);
logger.start();
My command is currently just calling my test.bat with two parameters. My test batch now just do :
echo "BATCH CALLED WITH PARAMETER %1 AND %2"
exit
My log file however does contains only :
********************************************
********************************************
Batch call completed with return status 0
I tried to place the waitFor()call before and after the code redirecting the output to the log file, without success. I always see the black screen of the command being launched, but nothing in logs...
Any help would be greatly appreciated, I'm missing something, but cannot understand what...
You're not reading from the standard error of the process you create.
I suspect that an error message is being written to the standard error, and because you're only reading from standard output, you're not picking up this error.
I would recommend replacing your use of Runtime.getRuntime().exec(...) with a ProcessBuilder, using something like the following:
ProcessBuilder pb = new ProcessBuilder("cmd.exe", "/c", "test.bat", "One", "Two");
pb.redirectErrorStream(true);
Process process = pb.start();
The line pb.redirectErrorStream(true); redirects the process' standard error into its standard output, so that you don't have to read from two streams (standard output and standard error) in two separate threads.
In other languages (like bash and Python), when we spawn a child process, this new process will inherit the stdout and stderr from the parent. This means that any output from the child process will be printed to the terminal as well as the output from the parent.
How can we achieve the same behavior in Java?
My first try was:
proc = Runtime.getRuntime().exec(cmd);
But it won't work. Based on this answer and this answer, I've replaced the code with:
ProcessBuilder pb = new ProcessBuilder(cmd);
pb.redirectOutput(System.out);
pb.redirectError(System.err);
But this doesn't even compile, as the arguments are incompatible with the expected method parameters.
You need to read the output and error streams of the process you've created and write the output to where you want it to go (in your case System.out and System.err).
Edit. The above answer is correct for java < 7. As of 7 from the javadoc it looks to be possible to instead call
processBuilder.redirectOutput(Redirect.INHERIT)
This isn't too bad. I initially said "easy", then I realized I'd had to code the InputStreamConsumer:
public static class InputStreamConsumer extends Thread {
private InputStream is;
public InputStreamConsumer(InputStream is) {
this.is = is;
}
#Override
public void run() {
try {
int value = -1;
while ((value = is.read()) != -1) {
System.out.print((char)value);
}
} catch (IOException exp) {
exp.printStackTrace();
}
}
}
private void captureOutput(Process p) {
InputStreamConsumer stdout;
InputStreamConsumer errout;
errout = new InputStreamConsumer(p.getErrorStream());
stdout = new InputStreamConsumer(p.getInputStream());
errout.start();
stdout.start();
}
....running inside of something like ...
ProcessBuilder pb = new ProcessBuilder("cmd.exe", "/D", "/C", myCommand, parm, parm...);
try {
System.out.println("Start "+myCommand);
Process myProcess = pb.start();
captureOutput(myProcess);
int returnCode = myProcess.waitFor();
System.out.println("myProcess: return code : "+returnCode);
}
catch (IOException e) {
e.printStackTrace();
}
It's certainly possible. The Ant Java task does this when fork is set true. I don't remember all the details, but on a quick look it appears that most of the magic is in PumpStreamHandler. You might try finding some inspiration from that.
my very first post here after long time of searching but yet get an answer regarding this issue, please help me in this issue.
I am using Netbean 6.9.1 to build a Java application which massive call to few different external program, therefore I used process and runtime function to call for external program.
The whole application process is separated into few stages and I wish to inform the user till which stage the application is currently running by updating the GUI textarea, the code is showed as below:
public void executeCommand (String cmd, File path)
{
try
{
****areaOutput.setText("Executing audio decoding, please wait till process is done\n");****
btnTranscribe.setEnabled(false);
areaOutput.setEditable(false);
areaOutput.setEnabled(false);
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd , null, path);
InputStream stderr = proc.getErrorStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
System.out.println("<ERROR>");
while ( (line = br.readLine()) != null)
System.out.println(line);
System.out.println("</ERROR>");
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
areaOutput.append("\n\nConversion is done, processing with features extraction....");
} catch (Throwable t)
{
t.printStackTrace();
}
}
As showed in the code above, I wish to set the Textarea and disable some button before executing the command, but when the application ran, all of these line seems unable to work and nothing is changed at the application itself till the command is finish executed, any solution to run the pre-command code execute first before the .exec() start to run?
I appreciate your great help and advice regarding this issue.
Best regards,
Striky
P/S:
hi there, I have make a Thread class for this CmdExec in order to execute cmd in different thread:
public class CmdExec extends Thread
{
private String cmd;
private File path;
public CmdExec() {
}
public CmdExec(String cmd, File path) {
this.cmd = cmd;
this.path = path;
}
public void run(){
try
{
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd , null, path);
InputStream stderr = proc.getErrorStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
System.out.println("<ERROR>");
while ( (line = br.readLine()) != null)
System.out.println(line);
System.out.println("</ERROR>");
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
} catch (Throwable t)
{
t.printStackTrace();
}
}
}
and in order to call to this class,
CmdExec tryDemo = new CmdExec();
tryDemo = new CmdExec(strSegment, fSegment);
tryDemo.run();
is used to start the thread, but I failed to put SwingUtilities.invokeLater in any part of these process, it simply won't run the tryDemo.run() because it is void...
Also, may I know so far am I doing right?? Very thank you for your kind help regarding this issue
P/S 2: I have just added another runnable code (so threads for process executing, runnable to GUI update) for GUI update command as below:
Runnable doWorkRunnable = new Runnable() {
public void run() {
System.out.println("hello world");
btnTranscribe.setEnabled(false);
areaOutput.setEditable(false);
areaOutput.setEnabled(false);
areaOutput.setText("Performing segmentation, please wait till process is done\n"); }
};
and I used SwingUtilies.invokeLater before the execution of process as below:
SwingUtilities.invokeLater(doWorkRunnable);
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd , null, path);
But all these failed, am I get the wrong sequence for the GUI and process thread coordination?
you are executing this work on the EDT (the thread which updates the gui). so, the gui cannot update until all this work finishes. what you want to do is run a separate thread which does all the work and periodically calls SwingUtilities.invokeLater with a status update.
Try to put sleep before execute method. To verify what is happening.
So I'm trying to execute a shell script which produces a lot of output(in 100s of MBs) from a Java file.
This hangs the process and never completes.
However, within the shell script, if I redirect the output of the script to some log file or /dev/null Java file executes and completes in a jiffy.
Is it because of amount of data that the Java program never completes?
If so, is there any documentation as such? or is there any limit on the amount of data(documented)?
Here's how you can simulate this scenario.
Java file will look like:
import java.io.InputStream;
public class LotOfOutput {
public static void main(String[] args) {
String cmd = "sh a-script-which-outputs-huuggee-data.sh";
try {
ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
pb.redirectErrorStream(true);
Process shell = pb.start();
InputStream shellIn = shell.getInputStream();
int shellExitStatus = shell.waitFor();
System.out.println(shellExitStatus);
shellIn.close();
} catch (Exception ignoreMe) {
}
}
}
The script 'a-script-which-outputs-huuggee-data.sh' may look like:
#!/bin/sh
# Toggle the line below
exec 3>&1 > /dev/null 2>&1
count=1
while [ $count -le 1000 ]
do
cat some-big-file
((count++))
done
echo
echo Yes I m done
Free beer for the right answer. :)
It's because you're not reading from the Process' output.
As per the class' Javadocs, if you don't do this then you may end up with a deadlock; the process fills its IO buffer and waits for the "shell" (or listening process) to read from it and empty it. Meanwhile your process, which should be doing this, is blocking waiting for the process to exit.
You'll want to call getInputStream() and read from that reliably (perhaps from another thread) to stop the process blocking.
Also take a look at Five Java Process Pitfalls and When Runtime.exec() Won't - both informative articles about common problems with Process.
You're never reading the input stream, so it's probably blocking because the input buffer is full.
The input/output buffer have a limited size (depending on the operating system). If I remember correctly this wasn't big or Windows XP at least. Try creating a thread that reads the InputStream as fast as possible.
Something along these lines:
class StdInWorker
implements Worker
{
private BufferedReader br;
private boolean run = true;
private int linesRead = 0;
private StdInWorker (Process prcs)
{
this.br = new BufferedReader(
new InputStreamReader(prcs.getInputStream()));
}
public synchronized void run ()
{
String in;
try {
while (this.run) {
while ((in = this.br.readLine()) != null) {
this.buffer.add(in);
linesRead++;
}
Thread.sleep(50);
}
}
catch (IOException ioe) {
ioe.printStackTrace();
}
catch (InterruptedException ie) {}
}
}
}