I have a java Spring REST API with a controller that runs a linux command with the ProcessBuilder class
. The command is a generated 'find' command
The problem is that I found a lot of unterminated processes in the hosting server after fiew days of use. I don't know why they still there and not ended or destroyed . (I checked with a ps -ef command)
Here is my runCmd function:
public static final BufferedReader runCmd(String cmd) throws IOException, InterruptedException {
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("bash", "-c", cmd);
processBuilder.redirectErrorStream(true);
Process process = processBuilder.start();
BufferedReader output = new BufferedReader(new InputStreamReader(process.getInputStream()));
int ret = process.waitFor();
return output;
}
Is there a way to make sur that there is no more process left behind ?
UPDATE
The problem comes only from commands with a very large output stream (std output) Thanks for the hint #DuncG
As this output is important, I can't ignore it. I have to find a way to consume it.
Any Idea on how to do it with Runnable Threads ?
Thanks
Are your commands generating a lot of output? The cause of the zombies may be simply that cmd has written a lot of output the STDOUT and the stream is blocking in the BufferedReader.
You can test if this the case by adding redirect to null - just append " > /dev/null" the end of cmd. This discards the sub-process output and means the BufferedReader is not full of unread data / blocking the sub-process.
processBuilder.command("bash", "-c", cmd + " > /dev/null");
If that fixes the zombie problem you can revert the redirect and make ProcessBuilder redirect output to files (before calling start()), or you'll need to add a thread to consume the IO as it is generated.
Path tmpdir = Path.of(System.getProperty("java.io.tmpdir"));
Path out = tmpdir.resolve("stdout.log");
processBuilder.redirectErrorStream(true);
processBuilder.redirectOutput(out.toFile());
At the end you should return the out file for caller to check, or could return Files.newBufferedReader(out).
If you don't use the redirect to file as above, this will store using thread to capture the output into memory buffer. Note you'd need to duplicate for STDERR too if not redirecting ERR->OUT:
Process p = pb.start();
ByteArrayOutputStream stdout = new ByteArrayOutputStream(8192);
new Thread(() -> copy(p.getInputStream(), stdout), "STDOUT").start();
int rc = p.waitFor();
byte[] sour = stdout.toByteArray()
Using method:
private static void copy(InputStream in, OutputStream buf)
{
try(var autoClose = in; var autoClose2 = buf)
{
in.transferTo(buf);
}
catch(IOException io)
{
throw new UncheckedIOException(io);
}
}
Related
I need to start a server using bash, so I had created an UNIX shell , but I am not able to execute it with Java from Eclipse.
I tried the following code which doesn't work :
Process proc = Runtime.getRuntime().exec(./startServer);
Here is content of the startServer file :
#!/bin/bash
cd /Users/sujitsoni/Documents/bet/client
npm start
You can try the following two options.
Option 1
Process proc = Runtime.getRuntime().exec("/bin/bash", "-c", "<Abosulte Path>/startServer");
Option 2
ProcessBuilder pb = new ProcessBuilder("/bin/bash", "-c", "<Absolute Path>/startServer");
pb.directory(new File("<Absolute Path>"));
Process proc = pb.start();
A couple Of things can go wrong:
The path to the file you have given might be wrong for eclipse it can take relative path but from the command line, it will take the absolute path.
error=13, Permission denied - If the script file doesn't have required permissions. In your scenario, that might not the case as you are not getting any error.
At last, you are executing the script by java program so the output of your script will not be printed out. In your scenario, this might be the case. You need to capture the output of script from BufferedReade and print it. ( In your case server might have started but you are not seeing the logs/output of the script.
See the code sample below for printing output.
public static void main(String[] args) throws IOException, InterruptedException {
Process proc = Runtime.getRuntime().exec("./startServer");
proc.waitFor();
StringBuffer output = new StringBuffer();
BufferedReader reader = new BufferedReader(new InputStreamReader(proc.getInputStream()));
String line = "";
while ((line = reader.readLine()) != null) {
output.append(line + "\n");
}
System.out.println(output);
}
I can run this command from the command line without any problem (the validation script executes):
c:/Python27/python ../feedvalidator/feedvalidator/src/demo.py https://das.dynalias.org:8080/das_core/das/2.16.840.1.113883.4.349/1012581676V377802/otherAdminData/careCoordinators
and from java if I leave off the URL parameter and just do:
String[] args1 = {"c:/Python27/python", "../feedvalidator/feedvalidator/src/demo.py" };
Runtime r = Runtime.getRuntime();
Process p = r.exec(args1);
it works fine. If I use certain URLs for a parameter such as:
String[] args1 = {"c:/Python27/python", "../feedvalidator/feedvalidator/src/demo.py" , "http://www.intertwingly.net/blog/index.atom"};
// or
String[] args1 = {"c:/Python27/python", "../feedvalidator/feedvalidator/src/demo.py" , "http://www.cnn.com"};
it also works fine.
But if I use this particular URL https://das.dynalias.org:8080/das_core/das/2.16.840.1.113883.4.349/1012581676V377802/otherAdminData/careCoordinators, then the script just hangs (java waits for the process to finish). I’m not sure why it works from the command line for that URL but not from a java program. I tried adding quotes to surround the URL parameter but that didn’t work either. I don’t see any character in the URL that I think need to be escaped.
Full Code:
String urlToValidate = "https://das.dynalias.org:8080/das_core/das/2.16.840.1.113883.4.349/1012581676V377802/otherAdminData/careCoordinators";
String[] args1 = {"c:/Python27/python", "C:/Documents and Settings/vhaiswcaldej/DAS_Workspace/feedvalidator/feedvalidator/src/demo.py", urlToValidate };
System.out.println(args1[0] + " " + args1[1] + " " + args1[2]);
Runtime r = Runtime.getRuntime();
Process p = r.exec(args1);
BufferedReader br = new BufferedReader(new InputStreamReader(
p.getInputStream()));
int returnCode = p.waitFor();
System.out.println("Python Script or OS Return Code: " + Integer.toString(returnCode));
if (returnCode >= 2) {
.out.println("OS Error: Unable to Find File or other OS error.");
}
String line = "";
while (br.ready()) {
String str = br.readLine();
System.out.println(str);
if (str.startsWith("line")) {
//TODO: Report this error back to test tool.
//System.out.println("Error!");
}
}
You need to drain the output and error streams of the process, or else it will block when the executed program produces output.
From the Process documentation:
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
People usually got caught by exec routine hangs in Java. I was cought by that once too. The problem is that the process you are trying to execute may (depending on lot of things) either first write to stdOut or stdErr. If you handle them in wrong order exec will hang. To handle this properly always you must create 2 threads to read stdErr and stdOut simulteneously. Sth like:
Process proc = Runtime.getRuntime().exec( cmd );
// handle process' stdout stream
Thread out = new StreamHandlerThread( stdOut, proc.getInputStream() );
out.start();
// handle process' stderr stream
Thread err = new StreamHandlerThread( stdErr, proc.getErrorStream() );
err.start();
exitVal = proc.waitFor(); // InterruptedException
...
out.join();
err.join();
Read (and close) p.getInputStream() and p.getErrorStream().
For example:
// com.google.common.io.CharStreams
CharStreams.toString(new InputStreamReader(p.getInputStream()));
CharStreams.toString(new InputStreamReader(p.getErrorStream()));
I want to execute this command within my java program and check if it was successfully executed.
sudo ntpdate -u someserver.com
I create a bash with the command
#!/bin/sh
sudo ntpdate -u omeserver.com
and execute it with java
ProcessBuilder pb = new ProcessBuilder("/updateTime");
Process p = pb.start();
p.waitFor();
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
System.out.println(stdInput.readLine());
But I get no output, there are no lines in stdInput, how can I check if the command was correctly executed?
If I add for example Echo updated in the end of the bash file I get it in the stdInput, but it still don't mean that the time were updated
You'd probably get by easier when just calling sudo directly with ProcessBuilder instead of an external script. That's just redundant complexity for the task at hand.
You can feed ProcessBuilder with the whole command line, for example, like this:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
public class q39836547 {
private static String[] cmdl = { "/usr/bin/sudo",
"ntpdate",
"-u",
"some.ntp.server" };
public static void main(String[] as) throws IOException {
ProcessBuilder pb = new ProcessBuilder(cmdl);
Process p = pb.start();
BufferedReader stdin = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedReader stderr = new BufferedReader(new InputStreamReader(p.getErrorStream()));
try { p.waitFor(); }
catch(InterruptedException e) { }
if(p.exitValue() != 0)
System.err.println("The process was not executed successfully.");
else
System.err.println("The process ran and exited cleanly.");
stdin.lines().forEach(s -> System.out.println("STDOUT: " + s));
stderr.lines().forEach(s -> System.out.println("STDERR: " + s));
}
}
You also have to waitFor() (as you properly did) the ntpdate to finish. Otherwise you might end up reading its standard input or standard error with getInputStream() or getErrorStream() before there is any output produced into either stream.
If you comment out the try-catch-block, you'll occasionally see how the process is still running while you're trying to read its input. That is likely to happen almost every time, actually.
I can run this command from the command line without any problem (the validation script executes):
c:/Python27/python ../feedvalidator/feedvalidator/src/demo.py https://das.dynalias.org:8080/das_core/das/2.16.840.1.113883.4.349/1012581676V377802/otherAdminData/careCoordinators
and from java if I leave off the URL parameter and just do:
String[] args1 = {"c:/Python27/python", "../feedvalidator/feedvalidator/src/demo.py" };
Runtime r = Runtime.getRuntime();
Process p = r.exec(args1);
it works fine. If I use certain URLs for a parameter such as:
String[] args1 = {"c:/Python27/python", "../feedvalidator/feedvalidator/src/demo.py" , "http://www.intertwingly.net/blog/index.atom"};
// or
String[] args1 = {"c:/Python27/python", "../feedvalidator/feedvalidator/src/demo.py" , "http://www.cnn.com"};
it also works fine.
But if I use this particular URL https://das.dynalias.org:8080/das_core/das/2.16.840.1.113883.4.349/1012581676V377802/otherAdminData/careCoordinators, then the script just hangs (java waits for the process to finish). I’m not sure why it works from the command line for that URL but not from a java program. I tried adding quotes to surround the URL parameter but that didn’t work either. I don’t see any character in the URL that I think need to be escaped.
Full Code:
String urlToValidate = "https://das.dynalias.org:8080/das_core/das/2.16.840.1.113883.4.349/1012581676V377802/otherAdminData/careCoordinators";
String[] args1 = {"c:/Python27/python", "C:/Documents and Settings/vhaiswcaldej/DAS_Workspace/feedvalidator/feedvalidator/src/demo.py", urlToValidate };
System.out.println(args1[0] + " " + args1[1] + " " + args1[2]);
Runtime r = Runtime.getRuntime();
Process p = r.exec(args1);
BufferedReader br = new BufferedReader(new InputStreamReader(
p.getInputStream()));
int returnCode = p.waitFor();
System.out.println("Python Script or OS Return Code: " + Integer.toString(returnCode));
if (returnCode >= 2) {
.out.println("OS Error: Unable to Find File or other OS error.");
}
String line = "";
while (br.ready()) {
String str = br.readLine();
System.out.println(str);
if (str.startsWith("line")) {
//TODO: Report this error back to test tool.
//System.out.println("Error!");
}
}
You need to drain the output and error streams of the process, or else it will block when the executed program produces output.
From the Process documentation:
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
People usually got caught by exec routine hangs in Java. I was cought by that once too. The problem is that the process you are trying to execute may (depending on lot of things) either first write to stdOut or stdErr. If you handle them in wrong order exec will hang. To handle this properly always you must create 2 threads to read stdErr and stdOut simulteneously. Sth like:
Process proc = Runtime.getRuntime().exec( cmd );
// handle process' stdout stream
Thread out = new StreamHandlerThread( stdOut, proc.getInputStream() );
out.start();
// handle process' stderr stream
Thread err = new StreamHandlerThread( stdErr, proc.getErrorStream() );
err.start();
exitVal = proc.waitFor(); // InterruptedException
...
out.join();
err.join();
Read (and close) p.getInputStream() and p.getErrorStream().
For example:
// com.google.common.io.CharStreams
CharStreams.toString(new InputStreamReader(p.getInputStream()));
CharStreams.toString(new InputStreamReader(p.getErrorStream()));
I need to start external executable in such way that user can interact with program that was just started.
For example in OpenSuse Linux there is a package manager - Zypper. You can start zypper in command mode and give commands like install, update, remove, etc. to it.
I would like to run it from Java code in a way user could interact with it: input commands and see output and errors of the program he started.
Here is a Java code I tried to use:
public static void main(String[] args) throws IOException, InterruptedException {
Process proc = java.lang.Runtime.getRuntime().exec("zypper shell");
InputStream stderr = proc.getInputStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
char ch;
while ( (ch = (char)br.read()) != -1)
System.out.print(ch);
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
}
But unfortunately I can only see it's output:
zypper>
but no matter what I write, my input doesn't affect program that was started.
How can I do what want to?
You need to get an output stream in order to write to the process:
OutputStream out = proc.getOuptutStream();
This output stream is piped into the standard input stream of the process, so you can just write to it (perhaps you want to wrap it in a PrintWriter first) and the data will be sent to the process' stdin.
Note that it might also be convenient to get the error stream (proc.getErrorStream) in order to read any error output that the process writes to its stderr.
API reference:
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html
Seems like the converting inside the while condition fails in your example, this seems to work better (I don't run Suse so I haven't tried with Zypper):
public static void main(String[] args) throws IOException, InterruptedException
{
//Process proc = java.lang.Runtime.getRuntime().exec("zypper shell");
Process proc = java.lang.Runtime.getRuntime().exec("ping -t localhost");
InputStream stderr = proc.getInputStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
int i;
while ( (i = br.read()) != -1)
{
System.out.print((char) i);
}
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
}
I recently wrapped Google Closure Compiler into a .jar-file which is extracted and used in a Process. This compiler only talks via System.in/out/err. There's a big "gotcha" in connecting pipes together, which is just briefly mentioned in the Process javadoc.
"...failure to promptly write the
input stream or read the output stream
of the subprocess may cause the
subprocess to block, and even
deadlock."
On Mac OS X the buffer is 16k, and if you don't read it promptly as suggested, the process deadlocks. My only solution to this problem ATM, is a rather nasty busy wait.
https://github.com/algesten/googccwrap/blob/master/src/main/java/googccwrap/GoogleClosureCompilerWrapper.java