Develop Reference

I've already created a Process Builder. How do I run all of the programs in the Process builder?

Heere is the code I have so far. How do I have miktex-pdftex run?
List<String> processes = new ArrayList<String>();
processes.add("miktex-pdftex --output-directory=[Directory] [file_name].tex");
ProcessBuilder processbuild = new ProcessBuilder(processes);

First, you need to make sure the command you are using actually works at the command. If it does not, then it's not going to work in Java.
Next, one of the main reasons for using ProcessBuilder is to deals with spaces in the command/parameters better then Runtime#exec.
String command = "/Applications/MiKTeX Console.app/Contents/bin/miktex-pdftex";
String outputDir = System.getProperty("user.dir");
String sourceFile = "Sample.tex";
List<String> commands = new ArrayList<>();
commands.add(command);
commands.add("--interaction=nonstopmode");
commands.add("--output-directory=" + outputDir);
commands.add(sourceFile);
So the above is very simple...
The command I want to run is /Applications/MiKTeX Console.app/Contents/bin/miktex-pdftex (I'm running on MacOS and I couldn't get the command installed outside the application bundle)
I want the output-directory to be the same as the current working directory (System.getProperty("user.dir")), but you could supply what every you need
I'm running in "nonstopmode" (--interaction=nonstopmode) because otherwise I would be required to provide input, which is just more complex
And my input file (Sample.tex) which is also in the working directory.
Next, we build the ProcessBuilder and redirect the error stream into the InputStream, this just reduces the next to read these two streams separately...
ProcessBuilder pb = new ProcessBuilder(commands);
pb.redirectErrorStream(true);
Next, we run the command, read the contents of the InputStream (otherwise you can stall the process), you can do what ever you want with this, I've just echoed it to the screen
try {
Process process = pb.start();
InputStream is = process.getInputStream();
int in = -1;
while ((in = is.read()) != -1) {
System.out.print((char)in);
}
int exitValue = process.waitFor();
System.out.println("");
System.out.println("Did exit with " + exitValue);
} catch (IOException | InterruptedException ex) {
ex.printStackTrace();
}
The use int exitValue = process.waitFor(); here is just to ensure that command has completed and get the exit value it generated. Normally, 0 is success, but you'd need to read the documentation of the command to be sure

Not receiving output from ProcessBuilder in YARN job

I have a Hadoop YARN cluster set up on some machines at my university (all machines running Linux Fedora 25). When running a mapreduce job in YARN, I am unable to receive the output from a call I make to a separate program. Interestingly, if I run my job locally (configured in mapred-site.xml), my method for calling the program and receiving its output works just fine. Below is my executeShellCommand class, which is instantiated and used in my first map task.
public class ExecuteShellCommand {
public String executeCommand(String command) {
StringBuffer output = new StringBuffer();
Process p;
try {
String [] args = command.split(" ");
String cmd = args[0];
ProcessBuilder pb = new ProcessBuilder().command(cmd, args[1], args[2], args[3], args[4], args[5], args[6], args[7]).directory(new File("path to executable"));
p = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
p.waitFor();
String line = "";
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
e.printStackTrace();
return e.toString();
}
return output.toString();
}
}
Things I have made sure to check:
1) Permissions are appropriately set for all files/directories needed
2) Map tasks are run as current user (me), so no issues with illegal access
3) I am not receiving a file not found exception, so the path to the program I'm calling is correct
4) Checked the input/output stream for Process p (input stream set as java.lang.UNIXProcess$ProcessPipeInputStream#1000e80, output stream is null)
5) Instead of calling the program I need to use, I have tried a simple "echo" command, and could not receive that output either.
6) I have also tried using
p = Runtime.getRuntime().exec("myCommand")
but the results are the same (no output received)
As I already mentioned, when I run a job locally, my executeCommand method functions perfectly, and returns the output from the program I call. Only in YARN do the issues occur. I have a feeling that the problem caused by either not reading from the correct buffer, or the command issued to ProcessBuilder is never actually executed. I am quite stumped as to how to debug what is going on here, and any tips would be greatly appreciated!

After hours of trying all sorts of solutions, I figured out how to get the error stream from the process spawned with ProcessBuilder. Turns out when I set the working directory for the process, I forgot to update the path to one of the arguments I was passing it. D'oh!!!

How to make Runtime.getRuntime().exec execute command one by one and get output rather than executing all commands

I am trying to run a *.bat file (which is capable of running several commands and retrieve the output one by one) from my java application. My intention is to send one command, read output use this output for second command and again retrieve the output.
To achieve this, through Runtime.getRuntime().exec I am passing more than one command as an input to PrintWriter. Issue is that after completing all the steps only I can read the output from *.bat through buffer ,but my intention is to run one command get the output and manipulate this output to send second command.
Unfortunately is not working. Any resolution for this?..
I got the idea to send more than one command to Runtime.getRuntime().exec from this link (How to execute cmd commands via Java)
The following is the same code which I got from above link
String[] command =
{
"cmd",
};
Process p = Runtime.getRuntime().exec(command);
new Thread(new SyncPipe(p.getErrorStream(), System.err)).start();
new Thread(new SyncPipe(p.getInputStream(), System.out)).start();
PrintWriter stdin = new PrintWriter(p.getOutputStream());
stdin.println("dir c:\\ /A /Q");
// write any other commands you want here
stdin.close();
int returnCode = p.waitFor();
System.out.println("Return code = " + returnCode);
class SyncPipe implements Runnable
{
public SyncPipe(InputStream istrm, OutputStream ostrm) {
istrm_ = istrm;
ostrm_ = ostrm;
}
public void run() {
try
{
final byte[] buffer = new byte[1024];
for (int length = 0; (length = istrm_.read(buffer)) != -1; )
{
ostrm_.write(buffer, 0, length);
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
private final OutputStream ostrm_;
private final InputStream istrm_;
}

In your case I would not use Threads, you want a sequential execution path.

Actually, instead of trying to reinvent the wheel, I strongly suggest you to use an expect-like java library to do that kind of thing.
Just because there are several things that you'll have to deal with, such as timeout between requests, waiting for the output to return, etc.
Take a look at these libraries
http://expectj.sourceforge.net/
https://code.google.com/p/expect4j/
https://github.com/ronniedong/Expect-for-Java
http://code.google.com/p/enchanter/
In particular, I use expectj in my project and it works pretty well (although I think expect4j is more popular)
With expectj, your code will look like this (from http://expectj.sourceforge.net/)
// Create a new ExpectJ object with a timeout of 5s
ExpectJ expectinator = new ExpectJ(5);
// Fork the process
Spawn shell = expectinator.spawn("/bin/sh");
// Talk to it
shell.send("echo Chunder\n");
shell.expect("Chunder");
shell.send("exit\n");
shell.expectClose();

You can do the redirection of output of one command to other in the bat file itself using pipe.
I am sorry, i hadn't noticed that you want to manipulate the output first.
So instead of using bat file, you can run the commands that are in bat file from java using exec , get the out put, and use the out put to execute the next command.

How to execute a interactive shell script using java Runtime?

I am wondering is there any way to execute following shell script, which waits for user input using java's Runtime class?
#!/bin/bash
echo "Please enter your name:"
read name
echo "Welcome $name"
I am using following java code to do this task but it just shows blank console.
public class TestShellScript {
public static void main(String[] args) {
File wd = new File("/mnt/client/");
System.out.println("Working Directory: " +wd);
Process proc = null;
try {
proc = Runtime.getRuntime().exec("sudo ./test.sh", null, wd);
} catch (Exception e) {
e.printStackTrace();
}
}
}
Thing is when I execute above program, I believed it will execute a shell script and that shell script will wait for user input, but it just prints current directory and then exits. Is there any way to do this or it is not possible at all in java?
Thanks in advance

The reason it prints the current dir and exits is because your java app exits. You need to add a (threaded) listener to the input and error streams of your created process, and you'll probably want to add a printStream to the process's output stream
example:
proc = Runtime.getRuntime().exec(cmds);
PrintStream pw = new PrintStream(proc.getOutputStream());
FetcherListener fl = new FetcherListener() {
#Override
public void fetchedMore(byte[] buf, int start, int end) {
textOut.println(new String(buf, start, end - start));
}
#Override
public void fetchedAll(byte[] buf) {
}
};
IOUtils.loadDataASync(proc.getInputStream(), fl);
IOUtils.loadDataASync(proc.getErrorStream(), fl);
String home = System.getProperty("user.home");
//System.out.println("home: " + home);
String profile = IOUtils.loadTextFile(new File(home + "/.profile"));
pw.println(profile);
pw.flush();
To run this, you will need to download my sourceforge project: http://tus.sourceforge.net/ but hopefully the code snippet is instructive enough that you can just adapt to J2SE and whatever else you are using.

If you use a Java ProcessBuilder you should be able to get the Input, Error and Output streams of the Process you create.
These streams can be used to get information coming out of the process (like prompts for input) but they can also be written to to put information into the process directly too. For instance:
InputStream stdout = process.getInputStream ();
BufferedReader reader = new BufferedReader (new InputStreamReader(stdout));
String line;
while(true){
line = reader.readLine();
//...
That'll get you the output from the process directly. I've not done it myself, but I'm pretty sure that process.getOutputStream() gives you something that can be written to directly to send input to the process.

The problem with running interactive programs, such as sudo, from Runtime.exec is that it attaches their stdin and stdout to pipes rather than the console device they need. You can make it work by redirecting the input and output to /dev/tty.
You can achieve the same behaviour using the new ProcessBuilder class, setting up the redirection using ProcessBuilder.Redirect.INHERIT.

Note sure at all you can send input to your script from Java. However I very strongly recommend to have a look at Commons Exec if you are to execute external scripts from Java:
Commons Exec homepage
Commons Exec API

Difference - executing a unix command through java and through prompt i.e normal execution

Could you please help me to resolve this issue.
I have a Java code which runs the rsync command using Runtime object.
I am running the below code at source machine, If there is any rsync connectivity problem during sync at target machine, the code should receive exit value, but that is not happening now.
String rsyncCommand = "rsync –abv <source> <remoteAddr:dest>"
Runtime rt = Runtime.getRuntime ();
rt.exec(rsyncCommand);
To give you more details:
When I run the rsync command directly(not through java code) in source machine and if I kill the rsync process at target machine using kill -9 option during sync, the rsync process at source will exit with exit message.
But if I run the rsync through my java code and if I kill the process during the sync at target, it is not receiving any exit message. The java and rsync process are still in running mode. But not doing any tasks.
What is the difference in running the command through java and directly through command prompt?
Any one has similar kind of problem with rsync, do we have any other options to run the rsync through java, I tried with “ProcessBuilder” as well.
Please provide me some pointers to solve this issue.
Thanks for the response, i gave only sample code, below is the complete code which i am using in my java.
Runtime rt = Runtime.getRuntime();
Process proc = null;
try {
proc = rt.exec(rsyncCommand);
InputStream stderr = proc.getErrorStream();
InputStreamReader isrErr = new InputStreamReader(stderr);
BufferedReader brErr = new BufferedReader(isrErr);
InputStream stdout = proc.getInputStream();
InputStreamReader isrStd = new InputStreamReader(stdout);
BufferedReader brStd = new BufferedReader(isrStd);
String val = null;
while ((val = brStd.readLine()) != null) {
System.out.println(val);
}
while ((val = brErr.readLine()) != null) {
System.out.println(val);
}
int exitVal = proc.waitFor();
} catch (Exception e) {
e.printStackTrace();
}

if you do this and the process is not finished yet you will not receive exit value
Process process = rt.exec(rsyncCommand);
int exitValue = process.exitValue();
instead you should use
int exitValue = process.waitFor()
then the thread will wait until the process returns exit value

Your invocation of exec() is incorrect, it should specify the parameters directly, something like:
Runtime rt = Runtime.getRuntime ();
rt.exec(new String[]{"rsync", "-abv", "<source>", "<remoteAddr:dest>"});
exec doesn't do any parsing of the command line, so it's trying to exec a command called "rsync –abv " (as a single string)