Develop Reference

ProcessBuilder failing when calling a system command where Runtime exec works

I'm having trouble with ProcessBuilder not running a command on the server.
Early in my project I use Runtime.exec() just to retrieve output from a program which works fine:
private List<SatelliteCode> getSatelliteCodes() {
List<SatelliteCode> codes = new ArrayList<>();
Runtime runtime = Runtime.getRuntime();
String[] commands = { "w_scan", "-s?" };
Process process;
try {
process = runtime.exec(commands);
BufferedReader error = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String s = error.readLine(); // discard first line
while ((s = error.readLine()) != null) {
s = s.trim();
int i = s.indexOf('\t'); // separated by a tab!?!?
codes.add(new SatelliteCode(s.substring(0, i), s.substring(i)));
}
} catch (IOException e) {
e.printStackTrace();
}
return codes;
}
Running this in the terminal works fine and I get all the output I need:
w_scan -fs -cGB -sS19E2 > channels.conf
However, the server needs to grab the ongoing output from the 'process.getErrorStream()' to display in the web interface. What is actually happening is the ProcessBuilder is failing and returning an exit code of 1.
The function that initialises the ProcessBuilder and to start the scan running is [EDIT 1]:
private static StringBuilder scan_error_output = null;
#Override
public boolean startSatelliteScan(String user, String country_code, String satellite_code) {
UserAccountPermissions perm = validateUserEdit(user);
if (perm == null) return false;
Shared.writeUserLog(user, Shared.getTimeStamp() +
": DVB satellite scan started " +
country_code + " - " + satellite_code +
System.lineSeparator() + System.lineSeparator());
scan_error_output = new StringBuilder();
new ScanThread(country_code, satellite_code).start();
// write out country code and satellite code to prefs file
BufferedWriter bw = null;
try {
bw = new BufferedWriter(new FileWriter(satellite_last_scan_codes));
bw.write(country_code); bw.newLine();
bw.write(satellite_code); bw.newLine();
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
return true;
}
That will then run two other threads on the server, one that will run the scan itself and wait for it to finish so that it can get the final scan data. And the other which constantly updates the output from the std error stream which is then polled at intervals from the clients browser. This is much like showing the ongoing output from the terminal.
The scan thread (which fails to start the process) [EDIT 1]:
private static class ScanThread extends Thread {
private String cc, sc;
public ScanThread(String country_code, String satellite_code) {
cc = country_code;
sc = satellite_code;
}
public void run() {
ProcessBuilder pb = new ProcessBuilder("/usr/bin/w_scan",
"-fs", "-c" + cc, "-s" + sc);
pb.redirectOutput(new File(satellite_scan_file));
Process process;
try {
System.out.println("Scan thread started");
process = pb.start();
IOScanErrorOutputHandler error_output_handler = new IOScanErrorOutputHandler(process.getErrorStream());
error_output_handler.start();
int result = process.waitFor();
System.out.println(cc + " - " + sc + " - " +
"Process.waitFor() result " + result);
} catch (IOException e) {
System.out.println(e.getMessage());
e.printStackTrace();
} catch (InterruptedException e) {
System.out.println(e.getMessage());
e.printStackTrace();
}
System.out.println("Scan thread finished");
}
}
The error output stream thread which captures the output which obviously doesn't start due to the scan thread failing:
private static class IOScanErrorOutputHandler extends Thread {
private InputStream inputStream;
IOScanErrorOutputHandler(InputStream inputStream) {
this.inputStream = inputStream;
}
public void run() {
Scanner br = null;
try {
System.out.println("Scan thread Error IO capture running");
br = new Scanner(new InputStreamReader(inputStream));
String line = null;
while (br.hasNextLine()) {
line = br.nextLine();
scan_error_output.append(line + System.getProperty("line.separator"));
}
} finally {
br.close();
}
System.out.println("Scan thread Error IO capture finished");
scan_error_output = null;
}
}
And the server function which returns the std error output progress:
#Override
public String pollScanResult(String user) {
if (validateUserEdit(user) == null) return null;
StringBuilder sb = scan_error_output; // grab instance
if (sb == null) return null;
return sb.toString();
}
As mentioned above, Runtime.exec() works fine, but the ProcessBuilder is failing.
NB: I'm on Linux Mint 18.1, using Apache Tomcat 8 as the server, linux default JDK 8 and GWT 2.7 [Correction from 2.8] in Eclipse Neon.
Can anyone see what I am doing wrong?
Many thanks in advance...
[EDIT 1]
Whilst developing this on another machine, Linux Mint 17.2, JDK 8 and Apache Tomcat 7, for DVB-T, this method worked fine and polling for the scan output showed up in the client's browser.
The ProcessBuilder.start still returns 1 and an empty file is created for the output scan file.
[EDIT 2]
It appears that the reason the ProcessBuilder is failing is because the user 'tomcat8' doesn't have permissions to run 'w_scan'. 'w_scan' works from the terminal, but not from the tomcat server. Somehow I've got to fix that now.
[SOLUTIONS]
After being put in the right direction by VGR for getting the error stream from the ProcessBuilder, I started digging further and found I was getting:
main:3909: FATAL: failed to open '/dev/dvb/adapter0/frontend0': 13 Permission denied
Apache tomcat 8 didn't have permission to access the DVB-S frontend to run a scan. This was fixed in two ways:
1 - 03catalina.policy I added the extra permissions (whether they made a difference I do not know).
grant codeBase "file:/dev/dvb/-" {
permission java.io.FilePermission "file:/dev/dvb/-", "read, write";
permission java.security.AllPermission;
};
2 - The dvb frontends belong to the 'video' group. So I needed to add the user tomcat8 to that group.
usermod -a -G video tomcat8
All works for now...

You are not doing the same thing with ProcessBuilder that you’re doing with Runtime.exec, so I don't know why you think ProcessBuilder is the problem.
You have a few problems with how you’re writing the command’s output to a file.
First, the presence of ">", satellite_scan_temp_file in your ProcessBuilder command is incorrect. Output redirection is not part of any command; it is handled by a shell. But when you run with Runtime.exec or ProcessBuilder, you are not running in a shell, you are executing the process directly. Neither w_scan nor any other command considers > a special character.
The correct way to redirect to a file is with the redirectOutput method:
ProcessBuilder pb = new ProcessBuilder(
"/usr/bin/w_scan", "-fs", "-s" + satellite_code, "-c" + country_code);
pb.redirectOutput(new File(satellite_scan_temp_file));
Second, your ScanThread code is ignoring the (currently incorrect) redirect, and is attempting to read the command’s output. But there is no output, because you are redirecting it all to a file.
Once you are properly redirecting output to a file, you can remove your BufferedReader and BufferedWriter loops completely.
Finally, it is worth noting that the error output you captured probably told you that > is not a valid argument to the w_scan process.

Error Could not find or load main class -jar

I am working on a java UNO project, OS : Ubuntu 14. I am calling exec via passing command to run via a jar file with some set of sub commands of that jar file.
String finalOutputMSG = "";
String[] cmd = {JAVA_LOCATION, " -jar ", JAR_LOCATION, " " + inputFile, " -dir ", ".isc", " -out xml"};//java location provides java location, jar location provides jar location, inputfile contains input file's location -dir provides output directory with name .isc, -out is output file with file format for output is xml
Similar command ran properly without showing any errors but in a case where I am trying to import a file and convert it into another format eg .xlsx to .xml, is giving error. In commands it worked, I have already generated outputs from an input file.
finalOutputMSG = exec(cmd);
/**
* exec() is executed and outputs are displayed
*
* #param String[] command passed to jar
* #return output message containing outputs or output message
*/
private static String exec(String[] cmd) {
String outputMSG = "";
Process proc = null;
try {
ProcessBuilder pb = new ProcessBuilder(cmd);
pb.redirectErrorStream(true);//any error output generated by subprocesses merged with the standard output,
//read using the Process.getInputStream()
///* Start the process */
proc = pb.start();
if (debug) {
System.out.println("Process started !");
}
outputMSG = getOutput(proc);
if (debug) {
System.out.println("outputMSG " + outputMSG);
}
} catch (IOException e) {
if (debug) {
System.out.println("Exception in exec " + e.getMessage());
JOptionPane.showMessageDialog(null, "Exception in exec ");
}
// StringBuilder append = appendToFile.append("Exception in exec ").append(e.getMessage());
} catch (Exception e) {
if (debug) {
System.out.println("Exception in exec " + e.getMessage());
JOptionPane.showMessageDialog(null, "Exception in exec ");
}
} finally {
///* Clean-up */
proc.destroy();
if (debug) {
System.out.println("Process ended !");
}
}
return outputMSG;
}
/**
* Reads output from current process
*
* #param current process
* #return output read in current process
*/
private static String getOutput(Process p) {
StringBuilder outStream = new StringBuilder();
if (debug) {
System.out.println("StringBuilder initialized in getOutput");
}
try {
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
if (debug) {
System.out.println("BufferedReader initialized in getOutput");
}
String line = null;
if (debug) {
System.out.println("in.readLine() in getOutput abt to be read");
}
while ((line = in.readLine()) != null) {
outStream.append(line);
if (debug) {
System.out.println("line in getOutput " + line);
System.out.println("outStream in getOutput " + outStream);
}
outStream.append("\n");
}
} catch (IOException e) {
if (debug1) {
System.out.println("IOException in getOutputs " + e.getMessage());
}
} catch (Exception ex) {
if (debug1) {
System.out.println("Exception in getOutputs" + ex.getMessage());
}
}
return outStream.toString();
}
Error Message depicted by Netbeans
Error: Could not find or load main class -jar
I have searched on the issue, but could not find any help that is useful, I could not understand, what is missing.

Solution:
String[] cmd = {JAVA_LOCATION, " -jar ", JAR_LOCATION, " " + inputFile, " -dir ", ".isc", " -out xml"};
I replaced the values that printed in console, and ran the command so got on terminal, it worked fine.
Solution: the command to be used must be without any spaces in the ends. Because terminal in linux interprets the commands like for "ls", but in java/ any programming language, it doesn't interprets for ls, so in case of the following parameter cmdarray
public Process exec(String[] cmdarray)
throws IOException
takes the command as it is.
String[] cmd = {JAVA_LOCATION, "-jar", JAR_LOCATION, inputFile, "-dir", ".isc", "-out", "xml"};

trying to execute mysqldump from java app, multiple problems

I am writing a java app that needs to perform mysql dump, and I am using the runtime.exec, based in the when runtime.exec won't article. The code is below:
public int exectuteCommand(){
Runtime rt = Runtime.getRuntime();
logger.debug("exexuting cmd: " + showCommand());
int exit = -1;
try {
Process proc = rt.exec(cmd);
ExtProcessStreamHandler errorHandler = new ExtProcessStreamHandler(proc.getErrorStream(), "ERROR");
ExtProcessStreamHandler outHandler = new ExtProcessStreamHandler(proc.getInputStream(), "OUTPUT");
// kick it off
errorHandler.start();
outHandler.start();
exit = proc.waitFor();
} catch (IOException e) {
logger.error("ERROR!! ~~ executing command " + showCommand(), e);
e.printStackTrace();
} catch (InterruptedException e) {
logger.error("ERROR!! ~~ unexpected return for " + showCommand() + " , returned " + exit, e);
e.printStackTrace();
}
return exit;
}
1) The command that the process returns works in the shell (I'm running this on a mac). The first error I had was an inability to find the mysqldump command. That results in this error:
java.io.IOException: Cannot run program "mysqldump": error=2, No such file or directory
I resolved that by adding the complete path of the file to the command. The $PATH var shows
/usr/local/mysql/bin/mysqldump
as the complete path. How can I make sure my java app has that info?
2) when adding the complete path to the command, I get this error msg:
INFO [Thread-1] (ExtProcessStreamHandler.java:28) - external process ERROR : mysqldump: Couldn't find table: ">"
Here is the code that builds the command array:
return new String[] {MYSQLDUMP_CMD, "-u", USER_DEFAULT, "-p"+ PW_DEFAULT, TEST_DB_NAME,
">", DUMP_LOC};
again, when I copy the command passed to the java app into the shell on my mac, it works. Not sure what I'm doing wrong.
thanks in advance!

It thinks ">" is an argument intended for mysqldump. You are invoking an executable, not evaluating a shell expression. If you want to pipe your output, do it with the outHandler and errorHandler in your code.
An alternative is to invoke a shell and pass the expression you want to evaluate as an argument:
expr = new StringBuilder()
.append(MYSQLDUMP_CMD).append(' ')
.append("-u").append(USER_DEFAULT).append(' ')
.append("-p").append(PW_DEFAULT).append(' ')
.append(TEST_DB_NAME).append(' ')
.append(">").append(' ')
.append(DUMP_LOC)
.toString();
return new String[] {"/bin/bash", "-c", expr};
If your code to build the command array doesn't wrap spaced arguments in single quotes (or if the JDK doesn't do this for you), then modify the StringBuilder statement to create the wrapped quotes for you.

Below Code is worked for me
public static void backup() {
String currentDate = LocalDate.now().format(DateTimeFormatter.ofPattern("yyyy_MM_dd"));
String backupPath = String.format("%s/%s.%s", Helper.BACKUP_PATH, currentDate, "sql");
File backupFile = new File(backupPath);
if (!backupFile.exists()) {
try {
backupFile.createNewFile();
String mysqlCom=String.format("mysqldump -u%s -p%s %s",USER_NAME,PASSWORD,DB);
String[] command = new String[] { "/bin/bash", "-c",mysqlCom};
ProcessBuilder processBuilder = new ProcessBuilder(Arrays.asList(command));
processBuilder.redirectError(Redirect.INHERIT);
processBuilder.redirectOutput(Redirect.to(backupFile));
Process process = processBuilder.start();
process.waitFor();
LOGGER.info("Backup done");
} catch (IOException e1) {
e1.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
} else {
LOGGER.info("Database already backuped today");
}
}

Java's System.getRuntime().exec not behaving as if the shell user called it

I am running a java application from the console on an HP-UX machine. In it, I generate some reports, zip them, and then email them. Everything is working, except the email.
I am using the mail binary to send mail from the command line. Since it's HP-UX, it's a bit different than the standard GNU sendmail.
This is the code I'm using to send the mail:
public static void EmailReports(String[] recipients, String reportArchive, String subject){
SimpleDateFormat dateFormat = new SimpleDateFormat("MM-dd-yyyy");
String today = dateFormat.format(new Date());
File tempEmailFile;
BufferedWriter emailWriter;
try {
tempEmailFile = File.createTempFile("report_email_" + today, "msg");
emailWriter = new BufferedWriter(new FileWriter(tempEmailFile));
} catch (IOException e) {
e.printStackTrace();
System.out.println("Failed to send email. Could not create temporary file.");
return;
}
try {
emailWriter.write("SUBJECT: " + subject + "\n");
emailWriter.write("FROM: " + FROM + "\n");
emailWriter.write(BODY + "\n");
emailWriter.close();
} catch (IOException e) {
e.printStackTrace();
System.out.println("Failed to send email. Could not write to temporary file.");
}
//read the archive in
try {
FileInputStream archiveIS = new FileInputStream(new File(reportArchive));
OutputStream archiveEncoder = MimeUtility.encode(new FileOutputStream(tempEmailFile, true), "uuencode", Zipper.getArchiveName(reportArchive));
//read archive
byte[] buffer = new byte[archiveIS.available()]; //these should never be more than a megabyte or two, so storing it in memory is no big deal.
archiveIS.read(buffer);
//encode archive
archiveEncoder.write(buffer);
//close both
archiveIS.close();
archiveEncoder.close();
} catch (FileNotFoundException e) {
System.out.println("Failed to send email. Could not find archive to email.");
e.printStackTrace();
} catch (MessagingException e) {
System.out.println("Failed to send email. Could not encode archive.");
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
System.out.println("Failed to send email. Could not encode archive.");
}
System.out.println("Sending '" + subject + "' email.");
try {
Process p = Runtime.getRuntime().exec("mail me#example.com < " + tempEmailFile.getAbsolutePath());
System.out.println("mail me#example.com < " + tempEmailFile.getAbsolutePath());
StringBuffer buffer = new StringBuffer();
while(p.getErrorStream().available() > 0){
buffer.append((char) p.getErrorStream().read());
}
System.out.println("STDERR: " + buffer.toString());
buffer = new StringBuffer();
while(p.getInputStream().available() > 0){
buffer.append((char) p.getInputStream().read());
}
System.out.println("STDOUT: " + buffer.toString());
} catch (IOException e) {
e.printStackTrace();
System.out.println("Failed to send email. Could not get access to the shell.");
}
}
When I run the program, and it sends the email, I get a blank email, no subject, no body, no attachment, and it's from the user#hostname from the HP-UX box instead of from the email specified in FROM.
However, when I run the same line that it runs (see the command printed out after I call exec), I get the correct email, from the correct user, with a subject, body, and attachment.
STDOUT and STDERR are both empty. It's almost as if I'm sending mail a blank file, but when I print the file before I call the exec, it's there.
What's going on here?
Edit: Attempts made:
Using Ksh:
try {
String cmd = "mail me#example.com.com < " + tempEmailFile.getAbsolutePath();
Runtime.getRuntime().exec(new String[] {"/usr/bin/ksh", cmd});
} catch (IOException e) {
e.printStackTrace();
System.out.println("Failed to send email. Could not get access to the shell.");
}
Using STDIN:
try {
System.out.println("mail me#example.com < " + tempEmailFile.getAbsolutePath());
Process p = Runtime.getRuntime().exec("mail me#example.com ");
FileInputStream inFile = new FileInputStream(tempEmailFile);
byte[] byteBuffer = new byte[inFile.available()];
inFile.read(byteBuffer);
p.getOutputStream().write(byteBuffer);
inFile.close();
p.getOutputStream().close();
StringBuffer buffer = new StringBuffer();
while(p.getErrorStream().available() > 0){
buffer.append((char) p.getErrorStream().read());
}
System.out.println("STDERR: " + buffer.toString());
buffer = new StringBuffer();
while(p.getInputStream().available() > 0){
buffer.append((char) p.getInputStream().read());
}
System.out.println("STDOUT: " + buffer.toString());
} catch (IOException e) {
e.printStackTrace();
System.out.println("Failed to send email. Could not get access to the shell.");
}

I strongly suspect the problem is the redirection. That's normally handled by the shell - and there's no shell here.
Either you need to execute the process normally and then get the process's standard input stream and write to it from Java, or (probably simpler) run /bin/sh (or whatever) to get the shell to do the redirection.

Try exec'ing { "ksh", "-c", "mail me#example.com < " + etc }. The -c option tells the shell specifically to parse the next argument as a shell command with possible redirection and so on. Without the -c, ksh follows a heuristic to decide what to do with its command line, and it may not be running the command in the way you want it to.

Split into two lines, just to get better readability:
String cmd = "mail me#example.com < " + tempEmailFile.getAbsolutePath () ;
Process p = Runtime.getRuntime().exec (cmd);
This will look for a program named "mail me#example.com < " + tempEmailFile.getAbsolutePath (). It will not do redirection - for that to do you have to read the output of that process yourself.
Furtermore it will not lookup the path, so you might have to specify the whole path /usr/bin/mail or whatever it is.
And you have to split command and parameters; use an Array of String instead: ("/path/to/prg", "param1", "param2", "foo=bar");
You can use redirection, if you call as program a script, like
String cmd = "/usr/bin/mail me#example.com < " + tempEmailFile.getAbsolutePath () ;
String cmdarr = new String [] {"/bin/bash", "-c", cmd};
Process p = Runtime.getRuntime().exec (cmdarr);
It is shorter than invoking file redirection from Java yourself, more simple but you lose the ability to react sensible on different errors.

Running shell script from Java

I am trying to run some shell scripts for Java by using commons exec package and clear the STDOUT & STDERR buffers by using PumpStreamHandler. Most of the scripts run fine without any problems but some of them hangs.
Particularly those scripts that takes some time to return. My guess is that the PumpStramHandle might be reading end of stream as there is nothing put on the stream for a while and after that the buffers fill up.
Is there any better way to get across this problem?

Extract the script/command being executed and run it yourself in a shell. When running things that are 'exec'd through some other language(c,c++, python java etc) and things start going 'wrong' this should be the first step.
You find all sorts of things going on. Scripts that stop and prompt for input(big source of hangups) errors that don't parse correctly, seg faults, files not found.

To expand on the first answer about running the commands directly to test, you can test your hypothesis with a simple script that sleeps for a while before returning output. If you
can't test your command, test your idea.
#!/bin/bash
sleep 60;
echo "if you are patient, here is your response"

Not the best solution. But does what I need. :)
class OSCommandLogger extends Thread {
private static final Logger logger = Logger.getLogger(OSCommandLogger.class);
private volatile boolean done = false;
private final String name;
// Each process is associated with an error and output stream
private final BufferedReader outputReader;
private final BufferedReader errorReader;
private final Logger log;
/**
* Reads the output & error streams of the processes and writes them to
* specified log
*
* #param p
* #param name
* #param log
*/
OSCommandLogger(Process p, String name, Logger log) {
// Create readers
outputReader = new BufferedReader(new InputStreamReader(p.getInputStream()));
errorReader = new BufferedReader(new InputStreamReader(p.getErrorStream()));
this.log = log;
if (name != null)
this.name = name;
else
this.name = "OSCommandStreamsLogger";
}
private void logLine(BufferedReader reader, boolean isError) {
try {
String line = null;
while ((line = reader.readLine()) != null) {
if (log != null && log.isDebugEnabled()) {
if (!isError)
log.debug("[OuputStream] " + line);
else
log.warn("[ErrorStream] " + line);
} else
logger.debug(line);
}
} catch (Exception ex) {
if (log != null)
log.error(name + ":" + "Error while reading command process stream", ex);
}
}
public void run() {
while (!done) {
logLine(outputReader, false);
logLine(errorReader, true);
try {
// Sleep for a while before reading the next lines
Thread.sleep(100);
} catch (InterruptedException e) {
log.debug("Done with command");
}
}
// Process is done. Close all the streams
try {
logLine(outputReader, false);
outputReader.close();
logLine(errorReader, true);
errorReader.close();
if (log != null && log.isDebugEnabled())
log.debug(name + ": Closed output/ error Streams.");
} catch (IOException ie) {
if (log != null)
log.error(name + ":" + "Error while reading command process stream", ie);
}
}
public void stopLoggers() {
if (log != null && log.isDebugEnabled())
log.debug(name + ":Stop loggers");
this.done = true;
}
}
Usage:
Process p = Runtime.getRuntime().exec("Command");
OSCommandLogger logger = new OSCommandLogger(p, "Command", log);
// Start the thread using thread pool
threadExec.executeRunnable(logger);
int exitValue = p.waitFor(); // Wait till the process is finished
// Required to stop the logger threads
logger.stopLoggers();
logger.interrupt();