Develop Reference

ProcessBuilder failing when calling a system command where Runtime exec works

I'm having trouble with ProcessBuilder not running a command on the server.
Early in my project I use Runtime.exec() just to retrieve output from a program which works fine:
private List<SatelliteCode> getSatelliteCodes() {
List<SatelliteCode> codes = new ArrayList<>();
Runtime runtime = Runtime.getRuntime();
String[] commands = { "w_scan", "-s?" };
Process process;
try {
process = runtime.exec(commands);
BufferedReader error = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String s = error.readLine(); // discard first line
while ((s = error.readLine()) != null) {
s = s.trim();
int i = s.indexOf('\t'); // separated by a tab!?!?
codes.add(new SatelliteCode(s.substring(0, i), s.substring(i)));
}
} catch (IOException e) {
e.printStackTrace();
}
return codes;
}
Running this in the terminal works fine and I get all the output I need:
w_scan -fs -cGB -sS19E2 > channels.conf
However, the server needs to grab the ongoing output from the 'process.getErrorStream()' to display in the web interface. What is actually happening is the ProcessBuilder is failing and returning an exit code of 1.
The function that initialises the ProcessBuilder and to start the scan running is [EDIT 1]:
private static StringBuilder scan_error_output = null;
#Override
public boolean startSatelliteScan(String user, String country_code, String satellite_code) {
UserAccountPermissions perm = validateUserEdit(user);
if (perm == null) return false;
Shared.writeUserLog(user, Shared.getTimeStamp() +
": DVB satellite scan started " +
country_code + " - " + satellite_code +
System.lineSeparator() + System.lineSeparator());
scan_error_output = new StringBuilder();
new ScanThread(country_code, satellite_code).start();
// write out country code and satellite code to prefs file
BufferedWriter bw = null;
try {
bw = new BufferedWriter(new FileWriter(satellite_last_scan_codes));
bw.write(country_code); bw.newLine();
bw.write(satellite_code); bw.newLine();
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
return true;
}
That will then run two other threads on the server, one that will run the scan itself and wait for it to finish so that it can get the final scan data. And the other which constantly updates the output from the std error stream which is then polled at intervals from the clients browser. This is much like showing the ongoing output from the terminal.
The scan thread (which fails to start the process) [EDIT 1]:
private static class ScanThread extends Thread {
private String cc, sc;
public ScanThread(String country_code, String satellite_code) {
cc = country_code;
sc = satellite_code;
}
public void run() {
ProcessBuilder pb = new ProcessBuilder("/usr/bin/w_scan",
"-fs", "-c" + cc, "-s" + sc);
pb.redirectOutput(new File(satellite_scan_file));
Process process;
try {
System.out.println("Scan thread started");
process = pb.start();
IOScanErrorOutputHandler error_output_handler = new IOScanErrorOutputHandler(process.getErrorStream());
error_output_handler.start();
int result = process.waitFor();
System.out.println(cc + " - " + sc + " - " +
"Process.waitFor() result " + result);
} catch (IOException e) {
System.out.println(e.getMessage());
e.printStackTrace();
} catch (InterruptedException e) {
System.out.println(e.getMessage());
e.printStackTrace();
}
System.out.println("Scan thread finished");
}
}
The error output stream thread which captures the output which obviously doesn't start due to the scan thread failing:
private static class IOScanErrorOutputHandler extends Thread {
private InputStream inputStream;
IOScanErrorOutputHandler(InputStream inputStream) {
this.inputStream = inputStream;
}
public void run() {
Scanner br = null;
try {
System.out.println("Scan thread Error IO capture running");
br = new Scanner(new InputStreamReader(inputStream));
String line = null;
while (br.hasNextLine()) {
line = br.nextLine();
scan_error_output.append(line + System.getProperty("line.separator"));
}
} finally {
br.close();
}
System.out.println("Scan thread Error IO capture finished");
scan_error_output = null;
}
}
And the server function which returns the std error output progress:
#Override
public String pollScanResult(String user) {
if (validateUserEdit(user) == null) return null;
StringBuilder sb = scan_error_output; // grab instance
if (sb == null) return null;
return sb.toString();
}
As mentioned above, Runtime.exec() works fine, but the ProcessBuilder is failing.
NB: I'm on Linux Mint 18.1, using Apache Tomcat 8 as the server, linux default JDK 8 and GWT 2.7 [Correction from 2.8] in Eclipse Neon.
Can anyone see what I am doing wrong?
Many thanks in advance...
[EDIT 1]
Whilst developing this on another machine, Linux Mint 17.2, JDK 8 and Apache Tomcat 7, for DVB-T, this method worked fine and polling for the scan output showed up in the client's browser.
The ProcessBuilder.start still returns 1 and an empty file is created for the output scan file.
[EDIT 2]
It appears that the reason the ProcessBuilder is failing is because the user 'tomcat8' doesn't have permissions to run 'w_scan'. 'w_scan' works from the terminal, but not from the tomcat server. Somehow I've got to fix that now.
[SOLUTIONS]
After being put in the right direction by VGR for getting the error stream from the ProcessBuilder, I started digging further and found I was getting:
main:3909: FATAL: failed to open '/dev/dvb/adapter0/frontend0': 13 Permission denied
Apache tomcat 8 didn't have permission to access the DVB-S frontend to run a scan. This was fixed in two ways:
1 - 03catalina.policy I added the extra permissions (whether they made a difference I do not know).
grant codeBase "file:/dev/dvb/-" {
permission java.io.FilePermission "file:/dev/dvb/-", "read, write";
permission java.security.AllPermission;
};
2 - The dvb frontends belong to the 'video' group. So I needed to add the user tomcat8 to that group.
usermod -a -G video tomcat8
All works for now...

You are not doing the same thing with ProcessBuilder that you’re doing with Runtime.exec, so I don't know why you think ProcessBuilder is the problem.
You have a few problems with how you’re writing the command’s output to a file.
First, the presence of ">", satellite_scan_temp_file in your ProcessBuilder command is incorrect. Output redirection is not part of any command; it is handled by a shell. But when you run with Runtime.exec or ProcessBuilder, you are not running in a shell, you are executing the process directly. Neither w_scan nor any other command considers > a special character.
The correct way to redirect to a file is with the redirectOutput method:
ProcessBuilder pb = new ProcessBuilder(
"/usr/bin/w_scan", "-fs", "-s" + satellite_code, "-c" + country_code);
pb.redirectOutput(new File(satellite_scan_temp_file));
Second, your ScanThread code is ignoring the (currently incorrect) redirect, and is attempting to read the command’s output. But there is no output, because you are redirecting it all to a file.
Once you are properly redirecting output to a file, you can remove your BufferedReader and BufferedWriter loops completely.
Finally, it is worth noting that the error output you captured probably told you that > is not a valid argument to the w_scan process.

Not receiving output from ProcessBuilder in YARN job

I have a Hadoop YARN cluster set up on some machines at my university (all machines running Linux Fedora 25). When running a mapreduce job in YARN, I am unable to receive the output from a call I make to a separate program. Interestingly, if I run my job locally (configured in mapred-site.xml), my method for calling the program and receiving its output works just fine. Below is my executeShellCommand class, which is instantiated and used in my first map task.
public class ExecuteShellCommand {
public String executeCommand(String command) {
StringBuffer output = new StringBuffer();
Process p;
try {
String [] args = command.split(" ");
String cmd = args[0];
ProcessBuilder pb = new ProcessBuilder().command(cmd, args[1], args[2], args[3], args[4], args[5], args[6], args[7]).directory(new File("path to executable"));
p = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
p.waitFor();
String line = "";
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
e.printStackTrace();
return e.toString();
}
return output.toString();
}
}
Things I have made sure to check:
1) Permissions are appropriately set for all files/directories needed
2) Map tasks are run as current user (me), so no issues with illegal access
3) I am not receiving a file not found exception, so the path to the program I'm calling is correct
4) Checked the input/output stream for Process p (input stream set as java.lang.UNIXProcess$ProcessPipeInputStream#1000e80, output stream is null)
5) Instead of calling the program I need to use, I have tried a simple "echo" command, and could not receive that output either.
6) I have also tried using
p = Runtime.getRuntime().exec("myCommand")
but the results are the same (no output received)
As I already mentioned, when I run a job locally, my executeCommand method functions perfectly, and returns the output from the program I call. Only in YARN do the issues occur. I have a feeling that the problem caused by either not reading from the correct buffer, or the command issued to ProcessBuilder is never actually executed. I am quite stumped as to how to debug what is going on here, and any tips would be greatly appreciated!

After hours of trying all sorts of solutions, I figured out how to get the error stream from the process spawned with ProcessBuilder. Turns out when I set the working directory for the process, I forgot to update the path to one of the arguments I was passing it. D'oh!!!

Java execute process on linux

I've been struggling for a while now with this problem and i can't seem to fix it.
i already have tried different approaches (Runtime.exec(), ProcessBuiler) but none seem to work.
This is my issue.
I have a laptop which is always on. This laptop runs a java tool connected to an arduino via usb to turn on and off the lights in the house. i have created this program myself, therefore i'm also doing some regular maintenance work on it. Recently i have added a button to restart the program from my html interface (in case i have an update, or if for some other reason i might need to restart the program or i decide to implement auto updating in the near future).
This idea behind this is to start a second instance of the application from the first instance and then System.exit(0) the first instance.
For some reason i'm not able to start a second instance of the application.
Here's some code.
public void shutdown(boolean restart) {
if (this.serial != null) {
this.serial.disconnect();
}
if (restart) {
System.out.println(this.getClass().getProtectionDomain().getCodeSource().getLocation().getPath());
String startupCommand = "java -jar \"" + this.getClass().getProtectionDomain().getCodeSource().getLocation().getPath().replace("%20", " ") + "\"";
ProcessBuilder builder = new ProcessBuilder();
// String[] command = new String[1];
// command[0] = "-jar \"" + (System.getProperty("user.dir") + "/Home_Automation_Executor.jar") + "\"";
try {
// //System.out.println("Restarting Home Automation with command: " + command[0]);
// System.out.println("Restarting Home Automation with command: " + startupCommand);
// Runtime.getRuntime().exec("bash");
// Process proc = Runtime.getRuntime().exec(startupCommand);
Process proc = builder.command(startupCommand).start();
InputStream stderr = proc.getErrorStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
System.out.println("<ERROR>");
while ((line = br.readLine()) != null) {
System.out.println(line);
}
System.out.println("</ERROR>");
int exitVal = 0;
try {
exitVal = proc.waitFor();
} catch (InterruptedException ex) {
Logger.getLogger(Engine.class.getName()).log(Level.SEVERE, null, ex);
}
System.out.println("Process exitValue: " + exitVal);
} catch (IOException ex) {
ex.printStackTrace();
}
}
System.out.println("Terminating Home Automation");
System.exit(0);
}
java.io.IOException: Cannot run program "java -jar "/Users/NightWalker/Dropbox/Development/Source Code/Java/NightWare Tools/Home Automation/Home Automation Executor/dist/Home_Automation_Executor.jar"": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at home.automation.executor.Engine.shutdown(Engine.java:186)
at home.automation.executor.webserver.HTTPGenerator._handleActionCommand(HTTPGenerator.java:190)
at home.automation.executor.webserver.HTTPGenerator._generateHTTPPage(HTTPGenerator.java:165)
at home.automation.executor.webserver.HTTPGenerator.getHTTPPage(HTTPGenerator.java:58)
at home.automation.executor.webserver.HTTPRequestHandler.run(HTTPRequestHandler.java:160)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:53)
at java.lang.ProcessImpl.start(ProcessImpl.java:91)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 5 more

The problem is this:
String startupCommand = "java -jar \"" + this.getClass().getProtectionDomain().getCodeSource().getLocation().getPath().replace("%20", " ") + "\"";
/* more stuff */ builder.command(startupCommand);
This means Jav will look for a command named java -jar ...stuff with spaces.... But what you want is, that Java looks for a command named java and give that command several parameters.
You should use
/*...*/ builder.command("java", "-jar", jarLocation) /*...*/

Since it is another Java program you might want to consider running it in the same process because it's much easier to communicate between the two programs if they live in the same process. Have you tried running the command outside your program? Does it work? What does the meta-inf.mf file in the jar hold? It might be that the classpath in the meta-inf.mf file isn't relative so any dependent jars can't be found.

Tomcat fails to execute external java program

I am a total newbie in JSPs/Tomcat and to a large extent in Java as well. Here's what I have to do -- when a user clicks a button/visits a URL, I want to launch a Java program (which takes some command line arguments).
I can very easily do
Runtime.exec("C:\\Python27\\python.exe test.py")
OR
Runtime.exec("java -cp %CLASSPATH%;C:\\devprojects HelloWorld"
and this works fine. Where HelloWorld.class just prints "HelloWorld".
However, when I attempt a java program which takes command line arguments, the GET request just hangs doing nothing. I don't know what logs to look for or what could be wrong here. After having spent TWO days on trying various things, I am just about to give up now.
Runtime.exec("java -cp %CLASSPATH%;C:\\devprojects Run --username Blah --password Foo");
What user does Tomcat end up running this java program as? Can I make it to be Administrator? This is on Windows 2008, does UAC interfere with things?
I cannot modify the Run.class here, I HAVE to run it as is and with command line parameters.
Please advise.

One idea: you are relying on the default tokenization of your command line as one complete String, and it is not parsing the last one as you expect. Instead you should use the form of this method that takes a String[], after you have chopped up the command line yourself:
http://docs.oracle.com/javase/6/docs/api/java/lang/Runtime.html#exec(java.lang.String[])
Or, it is waiting for input from you, or waiting for you to read its output. This could explain the hang. Search the internet for the dangers of streams and Runtime.exec().
Consider ProcessBuilder instead.

Remember also that you have to be sure that the executed file dont run "forever", and
if you need to pass some arguments, you could use this:
static String startProcess(String command, String dir) throws IOException {
StringBuffer ret = new StringBuffer();
String[] comm = new String[3];
comm[0] = COMMAND_INTERPRETER[0];
comm[1] = COMMAND_INTERPRETER[1];
comm[2] = command;
long start = System.currentTimeMillis();
try {
//Start process
Process ls_proc = Runtime.getRuntime().exec(comm, null, new File(dir));
//Get input and error streams
BufferedInputStream ls_in = new BufferedInputStream(ls_proc.getInputStream());
BufferedInputStream ls_err = new BufferedInputStream(ls_proc.getErrorStream());
boolean end = false;
while (!end) {
int c = 0;
while ((ls_err.available() > 0) && (++c <= 1000)) {
ret.append(conv2Html(ls_err.read()));
}
c = 0;
while ((ls_in.available() > 0) && (++c <= 1000)) {
ret.append(conv2Html(ls_in.read()));
}
try {
ls_proc.exitValue();
//if the process has not finished, an exception is thrown
//else
while (ls_err.available() > 0)
ret.append(conv2Html(ls_err.read()));
while (ls_in.available() > 0)
ret.append(conv2Html(ls_in.read()));
end = true;
}
catch (IllegalThreadStateException ex) {
//Process is running
}
//The process is not allowed to run longer than given time.
if (System.currentTimeMillis() - start > MAX_PROCESS_RUNNING_TIME)
//this is very important
{
ls_proc.destroy();
end = true;
ret.append("!!!! Process has timed out, destroyed !!!!!");
}
try {
Thread.sleep(50);
}
catch (InterruptedException ie) {}
}
}
catch (IOException e) {
ret.append("Error: " + e);
}
return ret.toString();
}

Log4j hanging my application

Im trying to get console output from an external application in my application. When i call the externall app from my code it hangs with the message:
Configuring logging...
Configuring log4j from: C:\GPAT\log4j.cfg
and nothing happens. I searched through internet and it seems that it might be thread issue. But i cant modify this external application and i must go through log4j. I read the external app like this:
StringBuffer output = new StringBuffer();
try {
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(GSATCommand);
BufferedReader input = new BufferedReader(new InputStreamReader(proc.getInputStream()));
System.out.println("Test running...");
String line = null;
while ((line = input.readLine()) != null) {
System.out.println(line); // Writes the test output to console
output.append(line); output.append("\n");
}
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
System.out.println("Test successfully executed");
} catch (Throwable t) {
t.printStackTrace();
}
Thanks for reading.

You need to consume both stdout and stderr from a spawned process in separate threads, to prevent blocking behaviour (your spawned process will write to buffers, and block if those buffers aren't being emptied by your consuming process).
See the 2nd paragraph of this answer for more details and a link to a suitable fix.