Develop Reference

Linux zombies processes left unterminated with Java ProcessBuilder

I have a java Spring REST API with a controller that runs a linux command with the ProcessBuilder class
. The command is a generated 'find' command
The problem is that I found a lot of unterminated processes in the hosting server after fiew days of use. I don't know why they still there and not ended or destroyed . (I checked with a ps -ef command)
Here is my runCmd function:
public static final BufferedReader runCmd(String cmd) throws IOException, InterruptedException {
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("bash", "-c", cmd);
processBuilder.redirectErrorStream(true);
Process process = processBuilder.start();
BufferedReader output = new BufferedReader(new InputStreamReader(process.getInputStream()));
int ret = process.waitFor();
return output;
}
Is there a way to make sur that there is no more process left behind ?
UPDATE
The problem comes only from commands with a very large output stream (std output) Thanks for the hint #DuncG
As this output is important, I can't ignore it. I have to find a way to consume it.
Any Idea on how to do it with Runnable Threads ?
Thanks

Are your commands generating a lot of output? The cause of the zombies may be simply that cmd has written a lot of output the STDOUT and the stream is blocking in the BufferedReader.
You can test if this the case by adding redirect to null - just append " > /dev/null" the end of cmd. This discards the sub-process output and means the BufferedReader is not full of unread data / blocking the sub-process.
processBuilder.command("bash", "-c", cmd + " > /dev/null");
If that fixes the zombie problem you can revert the redirect and make ProcessBuilder redirect output to files (before calling start()), or you'll need to add a thread to consume the IO as it is generated.
Path tmpdir = Path.of(System.getProperty("java.io.tmpdir"));
Path out = tmpdir.resolve("stdout.log");
processBuilder.redirectErrorStream(true);
processBuilder.redirectOutput(out.toFile());
At the end you should return the out file for caller to check, or could return Files.newBufferedReader(out).
If you don't use the redirect to file as above, this will store using thread to capture the output into memory buffer. Note you'd need to duplicate for STDERR too if not redirecting ERR->OUT:
Process p = pb.start();
ByteArrayOutputStream stdout = new ByteArrayOutputStream(8192);
new Thread(() -> copy(p.getInputStream(), stdout), "STDOUT").start();
int rc = p.waitFor();
byte[] sour = stdout.toByteArray()
Using method:
private static void copy(InputStream in, OutputStream buf)
{
try(var autoClose = in; var autoClose2 = buf)
{
in.transferTo(buf);
}
catch(IOException io)
{
throw new UncheckedIOException(io);
}
}

cmd netstat via java cant run [duplicate]

In the following program am giving name as "don" so the command will search activedirectory
with all the names starting with don (like donald etc). But the line2 variable becomes null after the assignment from reader object and it never goes into the loop. What am i doing wrong? FYI: the command works when i give it on the command line.
try {
Process p = Runtime.getRuntime().exec(
"dsquery user -name " + name + "* -limit 200|dsget user -samid -display");
p.waitFor();
BufferedReader reader = new BufferedReader(
new InputStreamReader(p.getInputStream()));
String line2 = reader.readLine();
HashMap<String,String> hmap = new HashMap<String,String>();
while (line2 != null) {
line2 = line2.trim();
if (line2.startsWith("dsget")||line2.startsWith("samid")) {
continue;
}
String[] arr = line2.split(" ",1);
hmap.put(arr[0].toLowerCase(),arr[1].toLowerCase());
line2 = reader.readLine();
}
reader.close();
line2 = reader.readLine();
}

If I am not mistaken, the pipe (or redirection) requires to launch the programs with cmd.exe.
Something like:
Process p = Runtime.getRuntime().exec("cmd /c dsquery user -name " + name + "* -limit 200|dsget user -samid -display");

I can see at least some possible problems:
1) as PhiLho wrote: pipe and redirection is done by the shell (sh, bash,... or cmd.exe on Windows). You must handle it in the Java code or run your commands in a shell.
2) after calling waitFor() the Thread is blocked until the process terminates, the process only terminates if you "consume" it's InputStream. This is not happening since waitFor() is still waiting... Better to read and process the InputStream in an additional Thread (or call waitFor after reading the InputStream).
3) reading after closing (2 last lines) should throw an Exception.
Reading the ErrorStream could help find some errors, and checking the return of waitFor is also indicated.
EDIT:
actually there should be some Exceptions being throw by that code.
Are the Exceptions being reported (printStackTrace) or just ignored?

Why the command line utility hangs when invoked through java program?

I need your suggestions and guidance in following task.
I am using libdmtx which comes with a command line utility which reads the image files for ECC200 Data Matrix barcodes, reads their contents, and writes the decoded messages to standard output.
I want to use this command line utility in my java program on linux platform. I amd using ubuntu linux. I have installed the libdmtx on my linux machine. and when I invoke the command
dmtxread -n /home/admin/ab.tif
on linux terminal it gives the decoded value of barcode in image immediately.
when I am going to invoke this command using my java program the code stuks in execution of the command and dotn gives output.
it looks like the program is processing or got hang.
Following is my java code which invokes the following command
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
public class Classtest {
public static void getCodes(){
try
{
Process p;
String command[]=new String[3];
command[0]="dmtxread";
command[1]="-n";
command[2]="/home/admin/ab.tif";
System.out.println("Command : "+command[0]+command[1]+command[2]);
p=Runtime.getRuntime().exec(command); //I think hangs over here.
BufferedReader reader=new BufferedReader(new InputStreamReader(p.getErrorStream()));
String line=reader.readLine();
if(line==null){
reader=new BufferedReader(new InputStreamReader(p.getInputStream()));
line=reader.readLine();
System.out.print("Decoded :- "+line);
}else{
System.out.print("Error :- "+line);
}
System.out.println(p.waitFor());
}catch(IOException e1) {
e1.getMessage();
e1.printStackTrace();
}catch(InterruptedException e2) {
e2.getMessage();
e2.printStackTrace();
}
}
public static void main(String args[]){
getCodes();
}
}
Please tell me friends where my code is going wrong.
I refered to following article but dint get any help
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=1
Please guide me friends!
Thank you!
Here is the new code in which I used the ProcessBuilder Class this code also giving the same output as above code that is it hangs at the line
Process process = pb.start();
public class Test {
public static void main(final String[] args) throws IOException, InterruptedException {
//Build command
List<String> commands = new ArrayList<String>();
commands.add("dmtxread");
commands.add("-n");
commands.add("/home/admin/ab.tif");
System.out.println(commands);
//Run macro on target
ProcessBuilder pb = new ProcessBuilder(commands);
pb.redirectErrorStream(true);
Process process = pb.start();
//Read output
StringBuilder out = new StringBuilder();
BufferedReader br = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line = null, previous = null;
while ((line = br.readLine()) != null){
System.out.println(line);
}
//Check result
if (process.waitFor() == 0)
System.out.println("Success!");
System.exit(0);
//Abnormal termination: Log command parameters and output and throw ExecutionException
System.err.println(commands);
System.err.println(out.toString());
System.exit(1);
}
}
Please guide me to solve this problem.
Thanks You!

The readLine blocks until it receives a new line from the error stream. So, if there is no output, your program won't get past the first readLine.
For simplicity I would recommend you use a ProcessBuilder instead of Runtime.exec(), which lets you merge the two InputStreams as follows:
ProcessBuilder builder = new ProcessBuilder(cmd,arg0,arg1);
builder.redirectErrorStream(true);
Process process = builder.start();
So, now you can just read from one.
Alternatively you can use separate threads to consume the two InputStreams.
Hope that helps

Your stream-consumption code is very confused. You try to read a single line from the stderr, then abandon that reader, then try to read a single line from the stdout.
If the program doesn't print anything to stderr, you'll hang at line 2.
If the program sends too much stuff to stderr so it fills its buffer, then the program itself will block and your Java will block at waitFor.
Both of these apply to stdout.
The proper way to consume the process's output streams is covered in detail in that article you have linked. Take that advice, nobody can give you better advice than that.

I am not sure what exactly happens with your program and where does it hang (you could use a debugger or trace output to check that), but here is the possible scenario:
Imagine that the program wants to output 2 lines of text. Or only one line but into stderr. Your code reads only 1 line fro stdout and than waits for the process to exit. This means that the child program may wait for the reader to read the next line, so it waits in write until someone unblocks the pipe -- forever.
When you run dmtxread from command line, there is no blocking on output pipe, so the program runs just finely.

How can I execute unix sort command (/usr/bin/sort) from Java?

I am hoping to leverage the unix sort command to sort a large text file in Java. I've tried executing sort with the process builder, but with no luck. However when I print the exact command it is going to execute and copy and paste it into the terminal, it works fine.
So far I've tried executing with /bin/sh -c "", making sure the directory the input file is and where the output file will be is fully permissioned (chmod 777) but with no luck.
Here is the code (if it looks funny, note is using some functions found in Guava)
File inputFile = new File(inputFileName);
//build the command (optional number of sort columns)
List<String> command = new LinkedList<String>();
command.addAll(ImmutableList.<String>of("sort","-t"+delimiter));
for (int i : sortFieldPositions) {
command.add("-k"+i+","+i);
}
command.addAll(ImmutableList.<String>of(inputFileName,">",outputFileName));
//for debugging: output the command that will be executed
System.out.println("Executing: "+Joiner.on(" ").join(command));
//construct and start the process
Process process = new ProcessBuilder(command).redirectErrorStream(true).directory(inputFile.getParentFile()).start();
//for debugging: save process output
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder outputStringBuilder = new StringBuilder();
for (String line; (line = bufferedReader.readLine()) != null; /*reading taking place in check */) {
System.out.println("FROM PROCESS: "+line);
outputStringBuilder.append(line);
}
bufferedReader.close();
if (process.exitValue() != 0) {
//something went wrong
throw new RuntimeException("Error code "+process.exitValue()+" executing command: "+Joiner.on(" ").join(command)+"\n"+outputStringBuilder.toString());
}
Unfortunately this does not work, with the following output:
Executing: sort -t, -k2,2 -k1,1 /tmp/java/TestDataSorterImporterInput.txt /tmp/java/TestDataSorterImporterOutput.txt
FROM PROCESS: sort: stat failed: >: No such file or directory
Edit: It may be helpful to note that if I remove saving the output (> outputfile) from the command, then the command executes without complaint and the sorted version appears in the output from the Processes' input stream)

It is the shell that knows how to perform output redirection. The sort program cannot do it on its own. So if you want redirection, you need to do /bin/sh -c ... to let she shell into the loop.
(You write that you have tried this, but something else must have gone wrong with that).

Try this:
String whatever = "filename";
Process p = Runtime.getRuntime().exec("sort -t -k2 2 -k1 1 " + whatever);
See this site.

Process process = new ProcessBuilder("/bin/bash", "-c", "sort -t'|' -k2").start();

Java Process with Input/Output Stream

I have the following code example below. Whereby you can enter a command to the bash shell i.e. echo test and have the result echo'd back. However, after the first read. Other output streams don't work?
Why is this or am I doing something wrong? My end goal is to created a Threaded scheduled task that executes a command periodically to /bash so the OutputStream and InputStream would have to work in tandem and not stop working. I have also been experiencing the error java.io.IOException: Broken pipe any ideas?
Thanks.
String line;
Scanner scan = new Scanner(System.in);
Process process = Runtime.getRuntime ().exec ("/bin/bash");
OutputStream stdin = process.getOutputStream ();
InputStream stderr = process.getErrorStream ();
InputStream stdout = process.getInputStream ();
BufferedReader reader = new BufferedReader (new InputStreamReader(stdout));
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stdin));
String input = scan.nextLine();
input += "\n";
writer.write(input);
writer.flush();
input = scan.nextLine();
input += "\n";
writer.write(input);
writer.flush();
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
input = scan.nextLine();
input += "\n";
writer.write(input);
writer.close();
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}

Firstly, I would recommend replacing the line
Process process = Runtime.getRuntime ().exec ("/bin/bash");
with the lines
ProcessBuilder builder = new ProcessBuilder("/bin/bash");
builder.redirectErrorStream(true);
Process process = builder.start();
ProcessBuilder is new in Java 5 and makes running external processes easier. In my opinion, its most significant improvement over Runtime.getRuntime().exec() is that it allows you to redirect the standard error of the child process into its standard output. This means you only have one InputStream to read from. Before this, you needed to have two separate Threads, one reading from stdout and one reading from stderr, to avoid the standard error buffer filling while the standard output buffer was empty (causing the child process to hang), or vice versa.
Next, the loops (of which you have two)
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
only exit when the reader, which reads from the process's standard output, returns end-of-file. This only happens when the bash process exits. It will not return end-of-file if there happens at present to be no more output from the process. Instead, it will wait for the next line of output from the process and not return until it has this next line.
Since you're sending two lines of input to the process before reaching this loop, the first of these two loops will hang if the process hasn't exited after these two lines of input. It will sit there waiting for another line to be read, but there will never be another line for it to read.
I compiled your source code (I'm on Windows at the moment, so I replaced /bin/bash with cmd.exe, but the principles should be the same), and I found that:
after typing in two lines, the output from the first two commands appears, but then the program hangs,
if I type in, say, echo test, and then exit, the program makes it out of the first loop since the cmd.exe process has exited. The program then asks for another line of input (which gets ignored), skips straight over the second loop since the child process has already exited, and then exits itself.
if I type in exit and then echo test, I get an IOException complaining about a pipe being closed. This is to be expected - the first line of input caused the process to exit, and there's nowhere to send the second line.
I have seen a trick that does something similar to what you seem to want, in a program I used to work on. This program kept around a number of shells, ran commands in them and read the output from these commands. The trick used was to always write out a 'magic' line that marks the end of the shell command's output, and use that to determine when the output from the command sent to the shell had finished.
I took your code and I replaced everything after the line that assigns to writer with the following loop:
while (scan.hasNext()) {
String input = scan.nextLine();
if (input.trim().equals("exit")) {
// Putting 'exit' amongst the echo --EOF--s below doesn't work.
writer.write("exit\n");
} else {
writer.write("((" + input + ") && echo --EOF--) || echo --EOF--\n");
}
writer.flush();
line = reader.readLine();
while (line != null && ! line.trim().equals("--EOF--")) {
System.out.println ("Stdout: " + line);
line = reader.readLine();
}
if (line == null) {
break;
}
}
After doing this, I could reliably run a few commands and have the output from each come back to me individually.
The two echo --EOF-- commands in the line sent to the shell are there to ensure that output from the command is terminated with --EOF-- even in the result of an error from the command.
Of course, this approach has its limitations. These limitations include:
if I enter a command that waits for user input (e.g. another shell), the program appears to hang,
it assumes that each process run by the shell ends its output with a newline,
it gets a bit confused if the command being run by the shell happens to write out a line --EOF--.
bash reports a syntax error and exits if you enter some text with an unmatched ).
These points might not matter to you if whatever it is you're thinking of running as a scheduled task is going to be restricted to a command or a small set of commands which will never behave in such pathological ways.
EDIT: improve exit handling and other minor changes following running this on Linux.

I think you can use thread like demon-thread for reading your input and your output reader will already be in while loop in main thread so you can read and write at same time.You can modify your program like this:
Thread T=new Thread(new Runnable() {
#Override
public void run() {
while(true)
{
String input = scan.nextLine();
input += "\n";
try {
writer.write(input);
writer.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
} );
T.start();
and you can reader will be same as above i.e.
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
make your writer as final otherwise it wont be able to accessible by inner class.

You have writer.close(); in your code. So bash receives EOF on its stdin and exits. Then you get Broken pipe when trying to read from the stdoutof the defunct bash.