Develop Reference

Not able to execute sort command using Runtime or ProcessBuilder

I am trying to execute this command sort --field-separator="," --key=2 /home/dummy/Desktop/sample.csv" -o /home/dummy/Desktop/sample_temp.csv using Java Runtime and ProcessBuilder.
Manually I am able to execute this command in linux, but using Runtime or ProcessBuilder, this command does not execute. It returns with an error code = 2.
Edit:
If I am trying to execute 'ls' command in linux through Java, I get the list of files in the current directory. But, If I try to execute the command 'ls | grep a', an IOException is thrown with error code=2.
Here is the snippet:
public static void main(String[] args) throws IOException {
InputStream is = null;
ByteArrayOutputStream baos = null;
ProcessBuilder pb = new ProcessBuilder("ls | grep a");
try {
Process prs = pb.start();
is = prs.getInputStream();
byte[] b = new byte[1024];
int size = 0;
baos = new ByteArrayOutputStream();
while((size = is.read(b)) != -1){
baos.write(b, 0, size);
}
System.out.println(new String(baos.toByteArray()));
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
try {
if(is != null) is.close();
if(baos != null) baos.close();
} catch (Exception ex){}
}
}

There could be a range of issue with your code. Hence you did not supply your code I can only guess.
The output file needs to be already created
The ',' field separator does not need the quotes around it (see code below)
So after these 2 issues (both making the program exit with '2'), this code actually works:
import java.io.IOException;
import java.util.Arrays;
public class Test {
public static void main(String[] args) throws IOException, InterruptedException {
ProcessBuilder pb = new ProcessBuilder(Arrays.asList("sort", "--field-separator=,", "--key=2", "/tmp/sample.csv", "-o",
"/tmp/sample_temp.csv"));
Process p = pb.start();
int returnCode = p.waitFor();
System.out.println(returnCode);
}
}
Will print '0' and will sort the file correctly.
For the 'ls | grep' issue, read this great article: http://www.javaworld.com/article/2071275/core-javahen-runtime-exec---won-t/core-java/when-runtime-exec---won-t.html
The article basically explains that the Runtime.exec (and the ProcessBuilder wrapper) is for running processes and not a Shell (the ls | grep you are trying are actually 2 processes in Linux communicating with each other thru stdout/in).

I am able to execute that manually. And error code 2 means Misuse of Shell BuiltIns
I see in your example you are only invoking "ls", not "/usr/bin/ls" (or something like that).
When you execute manually you have the luxury of PATH environment variable which is not availabled to the process you create.
Use "which ls" to discover the location of 'ls' on your target system. For your code to be portable you will have to make it a configurable option.

this is the way to execute any bash commands like sort, ls, cat (with sub-options). Please find the snippet:
private String executeCommand(String command) {
StringBuffer output = new StringBuffer();
Process p;
try {
p = Runtime.getRuntime().exec("script.sh");
p.waitFor();
BufferedReader reader =
new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = "";
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
e.printStackTrace();
}
return output.toString();
}
In the exec() method, I passed a shell script which contains the bash command within it. That linux command will be executed and you can carry on with the next task. Hope this was helpful.

Running a terminal program from java and saves it to a file called output1.txt

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.FileOutputStream;
import java.io.InputStream;
public class RunInput {
public static void main(String[] args) {
try {
Process p = Runtime.getRuntime().exec("java Island < island1.in");
p.waitFor();
InputStream in = p.getInputStream();
int b;
while ((b = in.read()) != -1) {
outputStream.write(b);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
I could not figure out why my program doesn't seem to run the command line and save it to output1.txt. Can someone tell me what is the proper way to do it?

There are I think 2 things missing:
you need to set the classpath for your Java command
you need toseparate the arguments of the command line you want to execute
For instance, assuming the classpath is the current directory:
String[] commands = { "java", "-cp", ".", "Island", "< island1.in" };
Process p = Runtime.getRuntime().exec(commands);

You can do this without invoking a seperate Java VM with code like this:
System.setIn(new FileInputStream("island1.in"));
System.setOut(new FileOutputStream("output1.txt"));
new Island().main(new String[0]);
This replaces the "stdin" and "stdout" with your files and then invokes the main of the Island class.

Runtime.exec does not execute the given command in a shell, so redirection via < and > will not work as you expect. Instead, they will be passed as additional parameters to the executed command. Use the getOutputStream and getInputStream methods of the Process object returned by exec instead.
Don't be confused by the names: getOutputStream returns the standard input stream of the subprocess (which is of type java.io.OutputStream) while getInputStream returns the standard output stream of the subprocess (which is of type java.io.InputStream). In my humble opinion these methods were better named getStandardInput, getStandardOutput and getStandardError.

Outputting result of "dir" to console in Java

I want to output the result of the "dir" command to the java console. I have already looked on Google and here, but none of the examples work for me, thus making me rite this post.
My code is as follows:
try
{
System.out.println("Thread started..");
String line = "";
String cmd = "dir";
Process child = Runtime.getRuntime().exec(cmd);
//Read output
BufferedReader dis = new BufferedReader( new InputStreamReader(child.getInputStream() ));
while ((line = dis.readLine()) != null)
{
System.out.println("Line: " + line);
}
dis.close();
}catch (IOException e){
}
What am I doing wrong?
Any help would be very nice.
Thanks in advance,
Darryl

You cannot run "dir" as a process, you need to change it to String cmd = "cmd dir";.
You don't handle the exception at all. adding the line e.printStackTrace()); in the catch block would tell you what I wrote in (1). Never ignore exceptions!
You don't handle error stream. This might cause your program to hang, handle error stream and read from both streams (error and input) in parallel using different streams.

The best way is to use commons exec http://commons.apache.org/exec/
This has things that catch quirks that can really drive you up the wall, such as the whole process blocking if you didn't clear its output stream, escaping quotes in commands, etc.
Here is a method that will run the del command in windows successfully. Note the use of cmd since del is not a standalone executable:
private void deleteWithCmd(File toDelete) throws IOException {
CommandLine cmdLine = new CommandLine("cmd.exe");
cmdLine.addArgument("/C");
cmdLine.addArgument("del");
cmdLine.addArgument(toDelete.getAbsolutePath());
DefaultExecutor executor = new DefaultExecutor();
int exitValue = executor.execute(cmdLine);
}

a) What is the java console?
b) You should use javas File.listFiles () instead of Runtime.exec, which isn't portable, and makes Name splitting neccessary - a hard thing, for filenames which contain spaces, blanks, newlines and so on.
c) Whats wrong with your code?
d) Why don't you do anything in the case of Exception?

Here is a more thorough example that accounts for OS versions and error conditions ( as stated by MByD above)
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4
Remember that using "exec" means that your application is no longer cross-platform and loses one of the main advantages of Java.
A better approach is to use java.io package.
http://download.oracle.com/javase/tutorial/essential/io/dirs.html

You can also do something like that in java 6 :
import java.io.File;
import java.io.FilenameFilter;
import java.io.IOException;
public class ListFilesFromRegExp {
public static void main(String[] args) throws IOException {
File dir = new File("files/");
File[] files = dir.listFiles(new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.matches("File[0-9].c");
}
});
for (int i = 0; i < files.length; i++) {
System.out.println(files[i].getAbsolutePath());
}
}
}

Why executing interactive process with redirected input/output streams causes application being stopped?

I have a console Java program that executes sh -i in a separate process and copies the data between the processes' input/output stream and corresponding System streams:
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
class StreamCopier implements Runnable {
private InputStream in;
private OutputStream out;
public StreamCopier(InputStream in, OutputStream out) {
this.in = in;
this.out = out;
}
public void run() {
try {
int n = 0;
byte[] buffer = new byte[4096];
while (-1 != (n = in.read(buffer))) {
out.write(buffer, 0, n);
out.flush();
}
} catch (IOException e) {
System.out.println(e);
}
}
}
public class Test {
public static void main(String[] args)
throws IOException, InterruptedException {
Process process = Runtime.getRuntime().exec("sh -i");
new Thread(new StreamCopier(
process.getInputStream(), System.out)).start();
new Thread(new StreamCopier(
process.getErrorStream(), System.err)).start();
new Thread(new StreamCopier(
System.in, process.getOutputStream())).start();
process.waitFor();
}
}
Running it under Linux results in the following:
$
[1]+ Stopped java -cp . Test
Could anyone clarify why is the application stopped and how to avoid it?
This is related to my question on copying streams, but I think this particular issue deserves separate attention.

You can turn off job control by invoking the shell like sh -i +m, which will stop it taking over the tty. This means that the fg and bg commands will not work and a Ctrl+Z will suspend your Java application, the shell and all programs started from it.
If you still want job control, you should use a pseudo terminal to communicate with the shell, which creates a new tty for the shell to use, but I don't think Java supports that.

You are being stopped by SIGTTIN or SIGTTOU. These signals are sent to a background process when they attempt to do IO to a TTY. In this case "background" means "not the controlling process group of the terminal". I suspect the subshell you're forking off is creating a new pgrp and taking over your tty. Then the parent program (java) does IO (in your case probably reading from the TTY) and gets SIGTTIN.
An easy way to confirm this theory would be to replace sh with something simpler (not a shell) which will not try to take over the tty.

Calling a shell script from java hangs

So I'm trying to execute a shell script which produces a lot of output(in 100s of MBs) from a Java file.
This hangs the process and never completes.
However, within the shell script, if I redirect the output of the script to some log file or /dev/null Java file executes and completes in a jiffy.
Is it because of amount of data that the Java program never completes?
If so, is there any documentation as such? or is there any limit on the amount of data(documented)?
Here's how you can simulate this scenario.
Java file will look like:
import java.io.InputStream;
public class LotOfOutput {
public static void main(String[] args) {
String cmd = "sh a-script-which-outputs-huuggee-data.sh";
try {
ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
pb.redirectErrorStream(true);
Process shell = pb.start();
InputStream shellIn = shell.getInputStream();
int shellExitStatus = shell.waitFor();
System.out.println(shellExitStatus);
shellIn.close();
} catch (Exception ignoreMe) {
}
}
}
The script 'a-script-which-outputs-huuggee-data.sh' may look like:
#!/bin/sh
# Toggle the line below
exec 3>&1 > /dev/null 2>&1
count=1
while [ $count -le 1000 ]
do
cat some-big-file
((count++))
done
echo
echo Yes I m done
Free beer for the right answer. :)

It's because you're not reading from the Process' output.
As per the class' Javadocs, if you don't do this then you may end up with a deadlock; the process fills its IO buffer and waits for the "shell" (or listening process) to read from it and empty it. Meanwhile your process, which should be doing this, is blocking waiting for the process to exit.
You'll want to call getInputStream() and read from that reliably (perhaps from another thread) to stop the process blocking.
Also take a look at Five Java Process Pitfalls and When Runtime.exec() Won't - both informative articles about common problems with Process.

You're never reading the input stream, so it's probably blocking because the input buffer is full.

The input/output buffer have a limited size (depending on the operating system). If I remember correctly this wasn't big or Windows XP at least. Try creating a thread that reads the InputStream as fast as possible.
Something along these lines:
class StdInWorker
implements Worker
{
private BufferedReader br;
private boolean run = true;
private int linesRead = 0;
private StdInWorker (Process prcs)
{
this.br = new BufferedReader(
new InputStreamReader(prcs.getInputStream()));
}
public synchronized void run ()
{
String in;
try {
while (this.run) {
while ((in = this.br.readLine()) != null) {
this.buffer.add(in);
linesRead++;
}
Thread.sleep(50);
}
}
catch (IOException ioe) {
ioe.printStackTrace();
}
catch (InterruptedException ie) {}
}
}
}