Develop Reference

Linux zombies processes left unterminated with Java ProcessBuilder

I have a java Spring REST API with a controller that runs a linux command with the ProcessBuilder class
. The command is a generated 'find' command
The problem is that I found a lot of unterminated processes in the hosting server after fiew days of use. I don't know why they still there and not ended or destroyed . (I checked with a ps -ef command)
Here is my runCmd function:
public static final BufferedReader runCmd(String cmd) throws IOException, InterruptedException {
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("bash", "-c", cmd);
processBuilder.redirectErrorStream(true);
Process process = processBuilder.start();
BufferedReader output = new BufferedReader(new InputStreamReader(process.getInputStream()));
int ret = process.waitFor();
return output;
}
Is there a way to make sur that there is no more process left behind ?
UPDATE
The problem comes only from commands with a very large output stream (std output) Thanks for the hint #DuncG
As this output is important, I can't ignore it. I have to find a way to consume it.
Any Idea on how to do it with Runnable Threads ?
Thanks

Are your commands generating a lot of output? The cause of the zombies may be simply that cmd has written a lot of output the STDOUT and the stream is blocking in the BufferedReader.
You can test if this the case by adding redirect to null - just append " > /dev/null" the end of cmd. This discards the sub-process output and means the BufferedReader is not full of unread data / blocking the sub-process.
processBuilder.command("bash", "-c", cmd + " > /dev/null");
If that fixes the zombie problem you can revert the redirect and make ProcessBuilder redirect output to files (before calling start()), or you'll need to add a thread to consume the IO as it is generated.
Path tmpdir = Path.of(System.getProperty("java.io.tmpdir"));
Path out = tmpdir.resolve("stdout.log");
processBuilder.redirectErrorStream(true);
processBuilder.redirectOutput(out.toFile());
At the end you should return the out file for caller to check, or could return Files.newBufferedReader(out).
If you don't use the redirect to file as above, this will store using thread to capture the output into memory buffer. Note you'd need to duplicate for STDERR too if not redirecting ERR->OUT:
Process p = pb.start();
ByteArrayOutputStream stdout = new ByteArrayOutputStream(8192);
new Thread(() -> copy(p.getInputStream(), stdout), "STDOUT").start();
int rc = p.waitFor();
byte[] sour = stdout.toByteArray()
Using method:
private static void copy(InputStream in, OutputStream buf)
{
try(var autoClose = in; var autoClose2 = buf)
{
in.transferTo(buf);
}
catch(IOException io)
{
throw new UncheckedIOException(io);
}
}

cmd netstat via java cant run [duplicate]

In the following program am giving name as "don" so the command will search activedirectory
with all the names starting with don (like donald etc). But the line2 variable becomes null after the assignment from reader object and it never goes into the loop. What am i doing wrong? FYI: the command works when i give it on the command line.
try {
Process p = Runtime.getRuntime().exec(
"dsquery user -name " + name + "* -limit 200|dsget user -samid -display");
p.waitFor();
BufferedReader reader = new BufferedReader(
new InputStreamReader(p.getInputStream()));
String line2 = reader.readLine();
HashMap<String,String> hmap = new HashMap<String,String>();
while (line2 != null) {
line2 = line2.trim();
if (line2.startsWith("dsget")||line2.startsWith("samid")) {
continue;
}
String[] arr = line2.split(" ",1);
hmap.put(arr[0].toLowerCase(),arr[1].toLowerCase());
line2 = reader.readLine();
}
reader.close();
line2 = reader.readLine();
}

If I am not mistaken, the pipe (or redirection) requires to launch the programs with cmd.exe.
Something like:
Process p = Runtime.getRuntime().exec("cmd /c dsquery user -name " + name + "* -limit 200|dsget user -samid -display");

I can see at least some possible problems:
1) as PhiLho wrote: pipe and redirection is done by the shell (sh, bash,... or cmd.exe on Windows). You must handle it in the Java code or run your commands in a shell.
2) after calling waitFor() the Thread is blocked until the process terminates, the process only terminates if you "consume" it's InputStream. This is not happening since waitFor() is still waiting... Better to read and process the InputStream in an additional Thread (or call waitFor after reading the InputStream).
3) reading after closing (2 last lines) should throw an Exception.
Reading the ErrorStream could help find some errors, and checking the return of waitFor is also indicated.
EDIT:
actually there should be some Exceptions being throw by that code.
Are the Exceptions being reported (printStackTrace) or just ignored?

Java Capture all output from a process and log it to file

I'm trying log all output from an Application in java and for some reason it only capturing the first 2 lines i know the application outputs a lot more than this this is my code
logOut = new BufferedWriter(new FileWriter("WebAdmin.log"));
ProcessBuilder procBuild = new ProcessBuilder(
"java", "-Xmx1G", "-Xms512M", "-jar", "TekkitServer\\Tekkit.jar", "nogui", "-nojline"
);
server = procBuild.start();
inputStream = new BufferedReader(new InputStreamReader(server.getInputStream()));
errorStream = new BufferedReader(new InputStreamReader(server.getErrorStream()));
outputStream = new BufferedWriter(new OutputStreamWriter(server.getOutputStream()));
String line = "";
while(!shutdown){
while((line = inputStream.readLine()) != null){
logOut.write(line+"\r\n");
logOut.flush();
System.out.println(line);
}
System.out.println("checking error stream");
while((line = errorStream.readLine()) != null){
logOut.write(line+"\r\n");
logOut.flush();
System.out.println(line);
}
}
System.out.println("Stoped Reading");
logOut.close();
server.destroy();
I'm not even seeing "checking error stream" in my console.

I don't think there is a need for outer while, reading from the output stream blocks until there is some data available, and if the sub process ends you'll get a null and the inner while will break. Also you could use ProcessBuilder's redirectErrorStream(true) to redirect stderr to stdout so you will catch them both in only one loop.

You must read the stdout and stderr streams in its own thread. Otherwise you will get all kinds of blocking situations (depending on operating system and output patterns).
If you merge the error stream with the output stream (redirectErrorStream(true)), you can get along with a single (additional) background thread. You can avoid this, when you use redirectOutput(File) (or inheritIO).

The first two lines in a minecraft server go to STDOUT (tested on bukkit, tekkit, and vanilla), but the rest go to STDERR for what ever reason. Your application isn't doing anything with errorStream. You can try using redirection functionality by doing this:.
ProcessBuilder procBuild = new ProcessBuilder(
"java", "-Xmx1G", "-Xms512M", "-jar", "TekkitServer\\Tekkit.jar",
"nogui", "-nojline", "2>&1");
Let me know if this works, as this may be a bash-only trick.

Java Runtime.exec()

I can run this command from the command line without any problem (the validation script executes):
c:/Python27/python ../feedvalidator/feedvalidator/src/demo.py https://das.dynalias.org:8080/das_core/das/2.16.840.1.113883.4.349/1012581676V377802/otherAdminData/careCoordinators
and from java if I leave off the URL parameter and just do:
String[] args1 = {"c:/Python27/python", "../feedvalidator/feedvalidator/src/demo.py" };
Runtime r = Runtime.getRuntime();
Process p = r.exec(args1);
it works fine. If I use certain URLs for a parameter such as:
String[] args1 = {"c:/Python27/python", "../feedvalidator/feedvalidator/src/demo.py" , "http://www.intertwingly.net/blog/index.atom"};
// or
String[] args1 = {"c:/Python27/python", "../feedvalidator/feedvalidator/src/demo.py" , "http://www.cnn.com"};
it also works fine.
But if I use this particular URL https://das.dynalias.org:8080/das_core/das/2.16.840.1.113883.4.349/1012581676V377802/otherAdminData/careCoordinators, then the script just hangs (java waits for the process to finish). I’m not sure why it works from the command line for that URL but not from a java program. I tried adding quotes to surround the URL parameter but that didn’t work either. I don’t see any character in the URL that I think need to be escaped.
Full Code:
String urlToValidate = "https://das.dynalias.org:8080/das_core/das/2.16.840.1.113883.4.349/1012581676V377802/otherAdminData/careCoordinators";
String[] args1 = {"c:/Python27/python", "C:/Documents and Settings/vhaiswcaldej/DAS_Workspace/feedvalidator/feedvalidator/src/demo.py", urlToValidate };
System.out.println(args1[0] + " " + args1[1] + " " + args1[2]);
Runtime r = Runtime.getRuntime();
Process p = r.exec(args1);
BufferedReader br = new BufferedReader(new InputStreamReader(
p.getInputStream()));
int returnCode = p.waitFor();
System.out.println("Python Script or OS Return Code: " + Integer.toString(returnCode));
if (returnCode >= 2) {
.out.println("OS Error: Unable to Find File or other OS error.");
}
String line = "";
while (br.ready()) {
String str = br.readLine();
System.out.println(str);
if (str.startsWith("line")) {
//TODO: Report this error back to test tool.
//System.out.println("Error!");
}
}

You need to drain the output and error streams of the process, or else it will block when the executed program produces output.
From the Process documentation:
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

People usually got caught by exec routine hangs in Java. I was cought by that once too. The problem is that the process you are trying to execute may (depending on lot of things) either first write to stdOut or stdErr. If you handle them in wrong order exec will hang. To handle this properly always you must create 2 threads to read stdErr and stdOut simulteneously. Sth like:
Process proc = Runtime.getRuntime().exec( cmd );
// handle process' stdout stream
Thread out = new StreamHandlerThread( stdOut, proc.getInputStream() );
out.start();
// handle process' stderr stream
Thread err = new StreamHandlerThread( stdErr, proc.getErrorStream() );
err.start();
exitVal = proc.waitFor(); // InterruptedException
...
out.join();
err.join();

Read (and close) p.getInputStream() and p.getErrorStream().
For example:
// com.google.common.io.CharStreams
CharStreams.toString(new InputStreamReader(p.getInputStream()));
CharStreams.toString(new InputStreamReader(p.getErrorStream()));

Start external executable from Java code with streams redirection

I need to start external executable in such way that user can interact with program that was just started.
For example in OpenSuse Linux there is a package manager - Zypper. You can start zypper in command mode and give commands like install, update, remove, etc. to it.
I would like to run it from Java code in a way user could interact with it: input commands and see output and errors of the program he started.
Here is a Java code I tried to use:
public static void main(String[] args) throws IOException, InterruptedException {
Process proc = java.lang.Runtime.getRuntime().exec("zypper shell");
InputStream stderr = proc.getInputStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
char ch;
while ( (ch = (char)br.read()) != -1)
System.out.print(ch);
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
}
But unfortunately I can only see it's output:
zypper>
but no matter what I write, my input doesn't affect program that was started.
How can I do what want to?

You need to get an output stream in order to write to the process:
OutputStream out = proc.getOuptutStream();
This output stream is piped into the standard input stream of the process, so you can just write to it (perhaps you want to wrap it in a PrintWriter first) and the data will be sent to the process' stdin.
Note that it might also be convenient to get the error stream (proc.getErrorStream) in order to read any error output that the process writes to its stderr.
API reference:
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html

Seems like the converting inside the while condition fails in your example, this seems to work better (I don't run Suse so I haven't tried with Zypper):
public static void main(String[] args) throws IOException, InterruptedException
{
//Process proc = java.lang.Runtime.getRuntime().exec("zypper shell");
Process proc = java.lang.Runtime.getRuntime().exec("ping -t localhost");
InputStream stderr = proc.getInputStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
int i;
while ( (i = br.read()) != -1)
{
System.out.print((char) i);
}
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
}

I recently wrapped Google Closure Compiler into a .jar-file which is extracted and used in a Process. This compiler only talks via System.in/out/err. There's a big "gotcha" in connecting pipes together, which is just briefly mentioned in the Process javadoc.
"...failure to promptly write the
input stream or read the output stream
of the subprocess may cause the
subprocess to block, and even
deadlock."
On Mac OS X the buffer is 16k, and if you don't read it promptly as suggested, the process deadlocks. My only solution to this problem ATM, is a rather nasty busy wait.
https://github.com/algesten/googccwrap/blob/master/src/main/java/googccwrap/GoogleClosureCompilerWrapper.java