Develop Reference

PowerShell script does not return

I write a PDF generator that is implemented in Java and uses PowerShell scripts to compile TEX files with latexmk.
An example for a generated command is powershell.exe -NoProfile -NoLogo "&'C:\path\to\scripts\compileTex.ps1' 'C:\path\to\workingDir' 'filename.tex'".
If I this command manually in the cmd the command compiles the TEX file and returns properly.
When I call this script via Java it does not return.
I read many other questions that suggest adding < NUL at the end of the command. I do this by explicitly closing the input stream of the process. Resulting from that all PowerShell scripts return properly except for the one with the latexmk command.
I use the following PowerShell script for compilation:
param(
[Parameter(Mandatory = $true)][string] $workingDir,
[Parameter(Mandatory = $true)][string] $texFileName
)
if (Test-Path -LiteralPath "$workingDir") {
cd "$workingDir"
if (Test-Path -LiteralPath "$texFileName") {
$texFileBaseName = $texFileName.Trim(".tex")
latexmk -nobibtex -norc -pdf -jobname="$texFileBaseName" "$texFileName" # FIXME This call leads to a timeout
} else {
Write-Error "Could not find \"$texFileName\""
}
} else {
Write-Error "The working directory \"$workingDir\" does not exist"
}
I use the following Java snippet for calling this script:
StringJoiner innerCommand = new StringJoiner(" ", "\"&", "\"")
.add(String.format("'%s'", script.normalize().toString()));
for (String param : params) {
if (param.startsWith("-")) { // Its an option or switch
innerCommand.add(param);
} else {
innerCommand.add(String.format("'%s'", param));
}
}
StringJoiner command = new StringJoiner(" ")
.add("powershell.exe")
.add("-NoProfile")
.add("-NoLogo")
.add(innerCommand.toString());
try {
Process process = Runtime.getRuntime()
.exec(command);
process.getOutputStream().close(); // Make sure CMD does not wait for further commands
boolean exited = process.waitFor(COMMAND_TIMEOUT_VALUE, COMMAND_TIMEOUT_UNIT);
String stdOutResult = new String(process.getInputStream().readAllBytes());
T commandResult;
if (exited) {
if (process.exitValue() == 0) {
commandResult = onSuccess.apply(stdOutResult);
} else {
String stdErrResult = new String(process.getErrorStream().readAllBytes());
commandResult = onFailure.apply(stdOutResult, stdErrResult);
}
} else {
commandResult = onTimeout.get();
}
return commandResult;
} catch (IOException | InterruptedException ex) {
throw new SystemCommandFailedException(String.format("Processing %d errored", commandID), ex);
}
}
EDIT:
Appending a [Console]::Out.Flush() does not do the trick.

ProcessBuilder failing when calling a system command where Runtime exec works

I'm having trouble with ProcessBuilder not running a command on the server.
Early in my project I use Runtime.exec() just to retrieve output from a program which works fine:
private List<SatelliteCode> getSatelliteCodes() {
List<SatelliteCode> codes = new ArrayList<>();
Runtime runtime = Runtime.getRuntime();
String[] commands = { "w_scan", "-s?" };
Process process;
try {
process = runtime.exec(commands);
BufferedReader error = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String s = error.readLine(); // discard first line
while ((s = error.readLine()) != null) {
s = s.trim();
int i = s.indexOf('\t'); // separated by a tab!?!?
codes.add(new SatelliteCode(s.substring(0, i), s.substring(i)));
}
} catch (IOException e) {
e.printStackTrace();
}
return codes;
}
Running this in the terminal works fine and I get all the output I need:
w_scan -fs -cGB -sS19E2 > channels.conf
However, the server needs to grab the ongoing output from the 'process.getErrorStream()' to display in the web interface. What is actually happening is the ProcessBuilder is failing and returning an exit code of 1.
The function that initialises the ProcessBuilder and to start the scan running is [EDIT 1]:
private static StringBuilder scan_error_output = null;
#Override
public boolean startSatelliteScan(String user, String country_code, String satellite_code) {
UserAccountPermissions perm = validateUserEdit(user);
if (perm == null) return false;
Shared.writeUserLog(user, Shared.getTimeStamp() +
": DVB satellite scan started " +
country_code + " - " + satellite_code +
System.lineSeparator() + System.lineSeparator());
scan_error_output = new StringBuilder();
new ScanThread(country_code, satellite_code).start();
// write out country code and satellite code to prefs file
BufferedWriter bw = null;
try {
bw = new BufferedWriter(new FileWriter(satellite_last_scan_codes));
bw.write(country_code); bw.newLine();
bw.write(satellite_code); bw.newLine();
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
return true;
}
That will then run two other threads on the server, one that will run the scan itself and wait for it to finish so that it can get the final scan data. And the other which constantly updates the output from the std error stream which is then polled at intervals from the clients browser. This is much like showing the ongoing output from the terminal.
The scan thread (which fails to start the process) [EDIT 1]:
private static class ScanThread extends Thread {
private String cc, sc;
public ScanThread(String country_code, String satellite_code) {
cc = country_code;
sc = satellite_code;
}
public void run() {
ProcessBuilder pb = new ProcessBuilder("/usr/bin/w_scan",
"-fs", "-c" + cc, "-s" + sc);
pb.redirectOutput(new File(satellite_scan_file));
Process process;
try {
System.out.println("Scan thread started");
process = pb.start();
IOScanErrorOutputHandler error_output_handler = new IOScanErrorOutputHandler(process.getErrorStream());
error_output_handler.start();
int result = process.waitFor();
System.out.println(cc + " - " + sc + " - " +
"Process.waitFor() result " + result);
} catch (IOException e) {
System.out.println(e.getMessage());
e.printStackTrace();
} catch (InterruptedException e) {
System.out.println(e.getMessage());
e.printStackTrace();
}
System.out.println("Scan thread finished");
}
}
The error output stream thread which captures the output which obviously doesn't start due to the scan thread failing:
private static class IOScanErrorOutputHandler extends Thread {
private InputStream inputStream;
IOScanErrorOutputHandler(InputStream inputStream) {
this.inputStream = inputStream;
}
public void run() {
Scanner br = null;
try {
System.out.println("Scan thread Error IO capture running");
br = new Scanner(new InputStreamReader(inputStream));
String line = null;
while (br.hasNextLine()) {
line = br.nextLine();
scan_error_output.append(line + System.getProperty("line.separator"));
}
} finally {
br.close();
}
System.out.println("Scan thread Error IO capture finished");
scan_error_output = null;
}
}
And the server function which returns the std error output progress:
#Override
public String pollScanResult(String user) {
if (validateUserEdit(user) == null) return null;
StringBuilder sb = scan_error_output; // grab instance
if (sb == null) return null;
return sb.toString();
}
As mentioned above, Runtime.exec() works fine, but the ProcessBuilder is failing.
NB: I'm on Linux Mint 18.1, using Apache Tomcat 8 as the server, linux default JDK 8 and GWT 2.7 [Correction from 2.8] in Eclipse Neon.
Can anyone see what I am doing wrong?
Many thanks in advance...
[EDIT 1]
Whilst developing this on another machine, Linux Mint 17.2, JDK 8 and Apache Tomcat 7, for DVB-T, this method worked fine and polling for the scan output showed up in the client's browser.
The ProcessBuilder.start still returns 1 and an empty file is created for the output scan file.
[EDIT 2]
It appears that the reason the ProcessBuilder is failing is because the user 'tomcat8' doesn't have permissions to run 'w_scan'. 'w_scan' works from the terminal, but not from the tomcat server. Somehow I've got to fix that now.
[SOLUTIONS]
After being put in the right direction by VGR for getting the error stream from the ProcessBuilder, I started digging further and found I was getting:
main:3909: FATAL: failed to open '/dev/dvb/adapter0/frontend0': 13 Permission denied
Apache tomcat 8 didn't have permission to access the DVB-S frontend to run a scan. This was fixed in two ways:
1 - 03catalina.policy I added the extra permissions (whether they made a difference I do not know).
grant codeBase "file:/dev/dvb/-" {
permission java.io.FilePermission "file:/dev/dvb/-", "read, write";
permission java.security.AllPermission;
};
2 - The dvb frontends belong to the 'video' group. So I needed to add the user tomcat8 to that group.
usermod -a -G video tomcat8
All works for now...

You are not doing the same thing with ProcessBuilder that you’re doing with Runtime.exec, so I don't know why you think ProcessBuilder is the problem.
You have a few problems with how you’re writing the command’s output to a file.
First, the presence of ">", satellite_scan_temp_file in your ProcessBuilder command is incorrect. Output redirection is not part of any command; it is handled by a shell. But when you run with Runtime.exec or ProcessBuilder, you are not running in a shell, you are executing the process directly. Neither w_scan nor any other command considers > a special character.
The correct way to redirect to a file is with the redirectOutput method:
ProcessBuilder pb = new ProcessBuilder(
"/usr/bin/w_scan", "-fs", "-s" + satellite_code, "-c" + country_code);
pb.redirectOutput(new File(satellite_scan_temp_file));
Second, your ScanThread code is ignoring the (currently incorrect) redirect, and is attempting to read the command’s output. But there is no output, because you are redirecting it all to a file.
Once you are properly redirecting output to a file, you can remove your BufferedReader and BufferedWriter loops completely.
Finally, it is worth noting that the error output you captured probably told you that > is not a valid argument to the w_scan process.

execute julia scripts from Java

I'm coding Julia script with ZeroMQ.
My goal is to communicate with ZMQ between two scripts. Here is an example:
# script1
using ZMQ
ctx = ZMQ.Context()
sockDealer = ZMQ.Socket(ctx, DEALER)
ZMQ.set_identity(sockDealer, "idA")
ZMQ.connect(sockDealer, "tcp://localhost:5555")
ZMQ.send(sockDealer, "hello world!")
ZMQ.close(sockDealer)
ZMQ.close(ctx)
#script2
using ZMQ
function pollrecv(socket::ZMQ.Socket,zmsg::Message)
rc = -1
while true
rc = ccall((:zmq_msg_recv, ZMQ.zmq), Cint, (Ptr{Message}, Ptr{Void}, Cint),
&zmsg, socket.data, ZMQ.ZMQ_DONTWAIT)
if rc == -1
# Base.Libc.EAGAIN = 11
# Problem unsolved: Failure to find Base.Libc.EAGAIN
if !(ZMQ.zmq_errno() == 11)
throw(ZMQ.StateError(ZMQ.jl_zmq_error_str()))
end
return false
else
ZMQ.get_events(socket) != 0 && notify(socket)
break
end
end
return true
end
ctx = ZMQ.Context()
sockRouter = ZMQ.Socket(ctx, ROUTER)
ZMQ.bind(sockRouter, "tcp://*:5555")
fini = false
while !fini
println("listening...")
idSock = Message()
while pollrecv(sockRouter, idSock)
msg = ZMQ.recv(sockRouter)
println("msg recv: " * bytestring(msg))
fini = true
end
sleep(1)
end
ZMQ.close(sockRouter)
ZMQ.close(ctx)
I can execute them with Julia on the command prompt. Everything goes fine. Script 2 can receive the message of Script 1.
Now, I need to execute them from Java. Meaning that I need to create a java project which is just like a controller. Here is my Java project:
public class Container {
private Vector<String[]> commands;
public Container() {
this.commands = new Vector<String[]>();
}
public void addCommand(String[] strs) {
this.commands.addElement(strs);
}
public void execute() {
for(int i = 0; i < this.commands.size(); i++) {
try {
Process p = Runtime.getRuntime().exec(this.commands.get(i));
if(p.waitFor() != 0){
System.err.println("exit value = " + p.exitValue());
}
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
StringBuffer stringBuffer = new StringBuffer();
String line = null;
while((line = in.readLine()) != null){
stringBuffer.append(line + "-");
}
System.out.println(stringBuffer.toString());
} catch (IOException ex) {
// TODO Auto-generated catch block
ex.printStackTrace();
} catch(InterruptedException e){
System.err.println(e);
}
}
}
}
//main
public class Main {
public static void main(String[] args) {
Container c = new Container();
String[] script1 = {"/usr/bin/julia", "/home/thomas/Julia/script1.jl"};
String[] script2 = {"/usr/bin/julia", "/home/thomas/Julia/script2.jl"};
c.addCommand(script1);
c.addCommand(script2);
c.execute();
}
}
However, when I run my java project, I can see that it keeps running but I can't see anything on the console: no result, no message, no error.
I think there is something wrong in my java project.

You'll want to run the two scripts concurrently: script2 is the server script, so it should be running when you run script1. As it is now, Process.waitFor() will wait for script1, the client script, to complete, before executing the server script script2 in the next for iteration.
You could start them as such:
String[] clientScript = { "/usr/bin/julia", "/home/thomas/Julia/script1.jl" };
String[] serverScript = { "/usr/bin/julia", "/home/thomas/Julia/script2.jl" };
Process server = Runtime.getRuntime().exec(serverScript);
Process client = Runtime.getRuntime().exec(clientScript);
and instantiate two threads to read their outputs:
(new ProcessReader(server)).start();
(new ProcessReader(client)).start();
using
public class ProcessReader extends Thread {
private Process p;
public ProcessReader(Process p) {
this.p = p;
}
#Override
public void run() {
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
try {
String line;
while ((line = in.readLine()) != null) {
System.out.println("Read: " + line);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Although, since the clientScript doesn't produce any output, you could just
start the scripts, and only read the output from the server script - no thread needed.
There's one more thing to consider: the serverScript must be listening... before the clientScript attempts to connect. So you may want to do this:
Process server = Runtime.getRuntime().exec(serverScript);
BufferedReader in = new BufferedReader(new InputStreamReader(server.getInputStream()));
if ( in.readLine().equals("listening...") ) {
Process client = Runtime.getRuntime().exec(clientScript);
String line;
while ( (line=in.readLine()) != null )
System.out.println("Read: " + line );
}

Since this is not a specific answer to your question, this may help to you or other users that attempt to similar jobs.
JuliaCaller is an early stage library for calling Julia from Java. It executes the Julia executable as a Java process and runs a script in the Julia side. This script opens a TCP server that listens on a given port number. Every command, statement or expression sent from Java is then executed and results are sent back to Java in JSON format.
This library also implements the standard javax.script interface, that means, Julia libraries, functions, and programs can run like a scripting language that is implemented in Java (mimics).
Here is the example:
Constants.setProperties(Constants.JULIA_PATH, "/usr/local/bin/julia");
Constants.setProperties(Constants.JULIA_PORT, "8001");
// Creating a scripting interface for Julia
manager = new ScriptEngineManager();
engine = manager.getEngineByName("Julia");
// Sending command 'a = 3' to Julia from Java
engine.eval("a = 3");
// Handling the result in Java
Object a = engine.get("a");
More examples are given in the GitHub page.
Source code with Apache License

How to access file path with spaces from command prompt in Java

I am trying to run a batch file from a Java program.
For instance: I have a batch "abc.bat" in a folder in "Program Files".
I want to execute this batch from my Java Program. I am using CommandLine class, Commons-exec jar.
CommandLine cmdLine = CommandLine.parse("cmd");
cmdLine.addArgument("/c start \"\" \"C:\\Program Files\\abc.bat\"");
DefaultExecutor exec = new DefaultExecutor();
Process p = Runtime.getRuntime().exec(cmdLine.toString());
exec.execute(cmdLine);
The above code throws an error saying "Windows cant find the file. Make sure you typed the name correctly, and try again". And, that is because of the spaces in the path.
So, I tried the answer provided here by #brso05 and that works. But I want it to be in a Future Class. Please find my code below and help me fix it.
final CommandLine cmdLine = CommandLine.parse("cmd.exe");
cmdLine.addArgument("/c");
cmdLine.addArgument("start");
cmdLine.addArgument("\""+ batchFileExecute.getParent().toString() + "\"");
ExecutorService es = Executors.newFixedThreadPool(1);
Future<?> future = es.submit(new Runnable() {
public void run() {
DefaultExecutor exec = new DefaultExecutor();
try {
Process p = Runtime.getRuntime().exec(cmdLine.toString());
exec.execute(cmdLine);
System.out.println(p.waitFor());
}
catch (IOException e)
{
e.printStackTrace();
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
});
String thread_status = null;
try
{
thread_status = future.get().toString();
System.out.println(thread_status+" completed the execution");
}
catch (NullPointerException e)
{
System.out.println("The execution of the received project is complete.");
// In here I want to do some processing again.
}
The code I mentioned works but it doesnt work if my batch file has a spaces in the path. Can you help me fix this?
Becuase the snippet you've given works but then I cant put it into Future. It doesnt work in the desired manner.
Thanks in advance!

This is an alternative way:
Runtime rt = Runtime.getRuntime();
rt.exec("cmd.exe /c start \"\" \"C:\\Program Files\\abc.bat\"");

Have you tried with single quotes? According to this, it should work.

I had the same filenames with spaces issue while using ImageMagick. Here is the code to solve the issue:
String imageOutput = null;
ByteArrayOutputStream identifyStdout = new ByteArrayOutputStream();
ByteArrayOutputStream identifyStderr = new ByteArrayOutputStream();
try
{
DefaultExecutor identifyExecutor = new DefaultExecutor();
// End the process if it exceeds 60 seconds
ExecuteWatchdog identifyWatchdog = new ExecuteWatchdog(60000);
identifyExecutor.setWatchdog(identifyWatchdog);
PumpStreamHandler identifyPsh = new PumpStreamHandler(identifyStdout, identifyStderr);
identifyExecutor.setStreamHandler(identifyPsh);
identifyExecutor.setExitValue(0); //0 is success
CommandLine identifyCommandline = new CommandLine("identify");
identifyCommandline.addArgument(destFile.getAbsolutePath(), false);
DefaultExecuteResultHandler identifyResultHandler = new DefaultExecuteResultHandler();
identifyExecutor.execute(identifyCommandline, identifyResultHandler);
identifyResultHandler.waitFor();
if (identifyResultHandler.getExitValue() != 0)
{
String output = identifyStdout.toString();
_logger.debug("Standard Out = " + output);
_logger.debug("Standard Err = " + identifyStderr.toString());
String msg = "ImageMagick overlay encountered an error. ImageMagick returned a value of " + identifyResultHandler.getExitValue();
throw new Exception(msg);
}
else
{
imageOutput = identifyStdout.toString();
_logger.debug("Standard Out = " + imageOutput);
identifyStdout.close();
identifyStderr.close();
}
}
catch(Exception e)
{
_logger.debug("Error: " + e.getLocalizedMessage(), e);
}
finally
{
identifyStdout.close();
identifyStderr.close();
}
The important part here is:
identifyCommandline.addArgument(destFile.getAbsolutePath(), false);
This line allows a filepath with spaces to process correctly.

When using the CommandLine class addArgument method without defining the boolean handleQuoting, it will set handleQuoting to true for you, basically adding quotes to the argument. This is the method invoking:
public CommandLine addArgument(String argument) {
return this.addArgument(argument, true);
}
public CommandLine addArgument(String argument, boolean handleQuoting) {
if (argument == null) {
return this;
} else {
if (handleQuoting) {
StringUtils.quoteArgument(argument);
}
this.arguments.add(new Argument(argument, handleQuoting));
return this;
}
}
Changing my method from:
CommandLine cmd = new CommandLine("pdfinfo");
cmd.addArgument("-box");
cmd.addArgument(pdfFile.getAbsolutePath());
To:
CommandLine cmd = new CommandLine("pdfinfo");
cmd.addArgument("-box");
cmd.addArgument(pdfFile.getAbsolutePath(), false); <-- change here
Solved the issue for me. No quotes were added and the CommandLine was able to find the file.

Tomcat fails to execute external java program

I am a total newbie in JSPs/Tomcat and to a large extent in Java as well. Here's what I have to do -- when a user clicks a button/visits a URL, I want to launch a Java program (which takes some command line arguments).
I can very easily do
Runtime.exec("C:\\Python27\\python.exe test.py")
OR
Runtime.exec("java -cp %CLASSPATH%;C:\\devprojects HelloWorld"
and this works fine. Where HelloWorld.class just prints "HelloWorld".
However, when I attempt a java program which takes command line arguments, the GET request just hangs doing nothing. I don't know what logs to look for or what could be wrong here. After having spent TWO days on trying various things, I am just about to give up now.
Runtime.exec("java -cp %CLASSPATH%;C:\\devprojects Run --username Blah --password Foo");
What user does Tomcat end up running this java program as? Can I make it to be Administrator? This is on Windows 2008, does UAC interfere with things?
I cannot modify the Run.class here, I HAVE to run it as is and with command line parameters.
Please advise.

One idea: you are relying on the default tokenization of your command line as one complete String, and it is not parsing the last one as you expect. Instead you should use the form of this method that takes a String[], after you have chopped up the command line yourself:
http://docs.oracle.com/javase/6/docs/api/java/lang/Runtime.html#exec(java.lang.String[])
Or, it is waiting for input from you, or waiting for you to read its output. This could explain the hang. Search the internet for the dangers of streams and Runtime.exec().
Consider ProcessBuilder instead.

Remember also that you have to be sure that the executed file dont run "forever", and
if you need to pass some arguments, you could use this:
static String startProcess(String command, String dir) throws IOException {
StringBuffer ret = new StringBuffer();
String[] comm = new String[3];
comm[0] = COMMAND_INTERPRETER[0];
comm[1] = COMMAND_INTERPRETER[1];
comm[2] = command;
long start = System.currentTimeMillis();
try {
//Start process
Process ls_proc = Runtime.getRuntime().exec(comm, null, new File(dir));
//Get input and error streams
BufferedInputStream ls_in = new BufferedInputStream(ls_proc.getInputStream());
BufferedInputStream ls_err = new BufferedInputStream(ls_proc.getErrorStream());
boolean end = false;
while (!end) {
int c = 0;
while ((ls_err.available() > 0) && (++c <= 1000)) {
ret.append(conv2Html(ls_err.read()));
}
c = 0;
while ((ls_in.available() > 0) && (++c <= 1000)) {
ret.append(conv2Html(ls_in.read()));
}
try {
ls_proc.exitValue();
//if the process has not finished, an exception is thrown
//else
while (ls_err.available() > 0)
ret.append(conv2Html(ls_err.read()));
while (ls_in.available() > 0)
ret.append(conv2Html(ls_in.read()));
end = true;
}
catch (IllegalThreadStateException ex) {
//Process is running
}
//The process is not allowed to run longer than given time.
if (System.currentTimeMillis() - start > MAX_PROCESS_RUNNING_TIME)
//this is very important
{
ls_proc.destroy();
end = true;
ret.append("!!!! Process has timed out, destroyed !!!!!");
}
try {
Thread.sleep(50);
}
catch (InterruptedException ie) {}
}
}
catch (IOException e) {
ret.append("Error: " + e);
}
return ret.toString();
}