Develop Reference

Run python script through Java with files arguments [duplicate]

This question already has answers here:
Java Runtime.getRuntime(): getting output from executing a command line program
(12 answers)
Closed 2 years ago.
I have a standard Maven project and I want to run the meTypeset script. This script takes 3 args where the second one is a file and the third one is a folder created as output.
This is how the script runs in a cmd:
meTypeset.py docx <input> <output_folder> [options]
This is how I try to run it in Java:
public static void main(String args[]) {
String[] cmd = {
"python",
"resources\\pyscripts\\meTypeset.py",
"docx",
"resources\\exampledocs\\example_journal.docx",
"resources\\output"
};
try {
Runtime.getRuntime().exec(cmd);
} catch (IOException e) {
e.printStackTrace();
}
}
Nothing happens, no errors but no result also

Unlike python Java may need some help. Do I guess correctly you are running on Windows?
You invoke the Runtime.exec() method. The method returns a Process instance, and in it's documentation you can read
By default, the created process does not have its own terminal or
console. All its standard I/O (i.e. stdin, stdout, stderr) operations
will be redirected to the parent process, where they can be accessed
via the streams obtained using the methods getOutputStream(),
getInputStream(), and getErrorStream(). The parent process uses these
streams to feed input to and get output from the process. Because some
native platforms only provide limited buffer size for standard input
and output streams, failure to promptly write the input stream or read
the output stream of the process may cause the process to block, or
even deadlock.
So it is likely your process is started by the OS but gets blocked due to I/O restrictions. Get around that by reading the STDOUT and STDERR streams until your process finishes. One good programming model is visible at https://www.baeldung.com/run-shell-command-in-java

#Hiran Chaudhuri explained the error correctly. I am just posting how I solved it, thanks to # Sonnenhut comment.
Runtime rt = Runtime.getRuntime();
String[] commands = {
"python",
"src\\main\\resources/pyscripts/meTypeset.py",
"docx",
"src\\main\\resources/exampledocs/example_journal.docx",
"src\\main\\resources/output"
};
Process proc = rt.exec(commands);
BufferedReader stdInput = new BufferedReader(new
InputStreamReader(proc.getInputStream()));
BufferedReader stdError = new BufferedReader(new
InputStreamReader(proc.getErrorStream()));
// Read the output from the command
System.out.println("Here is the standard output of the command:\n");
String s = null;
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
// Read any errors from the attempted command
System.out.println("Here is the standard error of the command (if any):\n");
while ((s = stdError.readLine()) != null) {
System.out.println(s);
}
}

Java Process causing python script JSON error

I have a Java program that manages data. When it wants to create a report from said data it does so via saving a JSON file with the relevant data for the report and then setting off the python script via the use of a ProcessBuilder object. I'm however having a weird error should I try and extract data from the output of the python script.
ProcessBuilder pythonProcess = new
ProcessBuilder("python","ReportingTool.py");
pythonProcess.directory(new File("invoice_python_files\\"));
Process pythonRunnable =pythonProcess.start();
/*
BufferedReader outputReader = new BufferedReader(new
InputStreamReader(pythonRunnable.getInputStream()));
BufferedReader errorReader = new BufferedReader(new
InputStreamReader(pythonRunnable.getErrorStream()));
String line =null;
System.out.println("<ERROR>");
while ( (line = errorReader.readLine()) != null)
System.out.println(line);
System.out.println("</ERROR>");
System.out.println("<Output">)
while ( (line = outputReader.readLine()) != null)
System.out.println(line);
System.out.println("</OUTPUT>");
*/
This works fine and produces the report as expected (without the input stream code). If I then uncomment the code I get an error from the python script.
File "C:\Users\o.cohen\AppData\Local\Programs\Python\Python36-
32\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I don't understand how the java process is causing the error and more importantly how to fix it. Below is the code that causes the error (specifically the last line:
with open("InvoiceMakerDoc.json") as json_file:
json_data=json_file.read()
decoded_data =json.loads(json_data)

Your java application need to wait till completion of python job first. Hence you need to add pythonRunnable.waitFor() before reading the input stream.
Process pythonRunnable =pythonProcess.start();
pythonRunnable.waitFor();

Managed to fix it.
The program was writing to the JSON file before it set the python program off and the FileWriter object hadn't been closed causing the issue.

Java: Redirecting inner process output when running from command line

I use the following code, for redirecting the output of a process I launch from my Java app:
ProcessBuilder builder = new ProcessBuilder("MyProcess.exe");
builder.redirectOutput(Redirect.INHERIT);
builder.redirectErrorStream(true);
Now, this works fine when I run the code from eclipse - I can see the output in Eclipse's console.
Yet when I create a jar file and run it from a cmd window, e.g. java -jar MyJar.jar, it doesn't print the output of the process. What could be the reason for this?

I know I'm late in answering, but I came across this question before coming across the answer, and wanted to save anybody else in the same boat some searching.
This is actually a known bug for Windows: https://bugs.openjdk.java.net/browse/JDK-8023130
You can get around it by redirecting the streams yourself:
Process p = pb.start();
BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = null;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
p.waitFor();
br.close();

It may be, that process is printing an error and exiting for some reason. So, the actual output goes into Err stream and not into the Out stream. Your code redirects Out stream only, so important process error information may be lost. I would suggest to inherit both Out and Err streams using this code:
ProcessBuilder builder = new ProcessBuilder("MyProcess.exe");
builder.inheritIO();
One more reason to redirect both streams is related to the output buffering for child process. If parent process (your java application) is not reading or redirecting standard streams (Out and Err) of the child process, then the latter may be blocked after a while, unable to make any further progress.
It definitely wouldn't hurt to have possible errors in the output anyway.

Imagemagick can't open file when executed from Java Servlet

I'm trying to convert files from png's to pdf using imagemagick and Java. I've got everything working to a place when I'm executing imagemagick command to actually merge multiple png's into one pdf. The command itself looks properly, and it works fine when executed in the terminal but my application gives me error showing that imgck can't open the file (even though it exists and I've set permissions to the folder to 777 :
line: convert: unable to open image `"/Users/mk/Documents/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/sch-java/print-1357784001005.png"': No such file or directory # error/blob.c/OpenBlob/2642.
This is my command :
/opt/local/bin/convert "/Users/mk/Documents/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/sch-java/print-1357784001005.png" "/Users/mk/Documents/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/sch-java/print-1357784001219.png" "/Users/mk/Documents/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/sch-java/complete-exportedPanel2013-01-1003:13:17.212.pdf"
And my Java code :
String filesString = "";
for (String s : pdfs){
filesString += "\""+ s + "\" ";
}
Process imgkProcess = null;
BufferedReader br = null;
File f1 = new File(pdfs[0]);
//returns true
System.out.println("OE: "+f1.exists());
String cmd = imgkPath+"convert "+ filesString+ " \""+outputPath+outName+"\"";
try {
imgkProcess = Runtime.getRuntime().exec(cmd);
InputStream stderr = imgkProcess.getErrorStream();
InputStreamReader isr = new InputStreamReader(stderr);
br = new BufferedReader(isr);
} catch (IOException e1) {
msg = e1.getMessage();
}
imgkProcess.waitFor();
while( (line=br.readLine() ) != null){
System.out.println("line: "+line);
}
The whole code is executed from a java servlet controller after getting request from a form. Any ideas what can cause this ? I'm using latest imgck, jdk, and osx 10.7 .

A few things:
When spawning anything but really trivial processes, it's usually better to use ProcessBuilder than Runtime.exec() - it gives you much better control
Even with ProcessBuilder, it often works better to write a shell script that does what you need. Then spawn a process to run the script. You get a lot more control in shell script than you do in ProcessBuilder
Remember that a spawned process is not a shell. It can't, for instance, evaluate expressions, or expand shell variables. If you need that, then you must execute a shell (like sh or bash). Better yet, write a shell script as described above
If all you need to do is to execute some ImageMagick commands, it would probably be easier to jmagick, a Java interface to ImageMagick - see http://www.jmagick.org/
Actually, since the you're assembling images into a PDF, the iText library - http://itextpdf.com is probably the best tool for the job, as it is native Java code, does not require spawning a native process, and will therefore be much more portable.

Solved it by adding all arguments to an arrayList and then casting it to String array.
ArrayList<String> cmd = new ArrayList<String>();
cmd.add(imgkPath+"convert");
for (int i=0, l=pdfs.length; i<l; i++){
cmd.add(pdfs[i]);
}
cmd.add(outputPath+outName);
imgkProcess = Runtime.getRuntime().exec(cmd.toArray(new String[cmd.size()]));

Launching wkhtmltopdf from Runtime.getRuntime().exec(): never terminates?

I'm launching wkhtmltopdf from within my Java app (part of a Tomcat server, running in debug mode within Eclipse Helios on Win7 64-bit): I'd like to wait for it to complete, then Do More Stuff.
String cmd[] = {"wkhtmltopdf", htmlPathIn, pdfPathOut};
Process proc = Runtime.getRuntime().exec( cmd, null );
proc.waitFor();
But waitFor() never returns. I can still see the process in the Windows Task Manager (with the command line I passed to exec(): looks fine). AND IT WORKS. wkhtmltopdf produces the PDF I'd expect, right where I'd expect it. I can open it, rename it, whatever, even while the process is still running (before I manually terminate it).
From the command line, everything is fine:
c:\wrk>wkhtmltopdf C:\Temp\foo.html c:\wrk\foo.pdf
Loading pages (1/6)
Counting pages (2/6)
Resolving links (4/6)
Loading headers and footers (5/6)
Printing pages (6/6)
Done
The process exits just fine, and life goes on.
So what is it about runtime.exec() that's causing wkhtmltopdf to never terminate?
I could grab proc.getInputStream() and look for "Done", but that's... vile. I want something that is more general.
I've calling exec() with and without a working directory. I've tried with and without an empty "env" array. No joy.
Why is my process hanging, and what can I do to fix it?
PS: I've tried this with a couple other command line apps, and they both exhibit the same behavior.
Further exec woes.
I'm trying to read standard out & error, without success. From the command line, I know there's supposed to be something remarkably like my command line experience, but when I read the input stream returned by proc.getInputStream(), I immediately get an EOL (-1, I'm using inputStream.read()).
I checked the JavaDoc for Process, and found this
The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the [b]subprocess to block, and even deadlock[/b].
Emphasis added. So I tried that. The first 'read()' on the Standard Out inputStream blocked until I killed the process...
WITH WKHTMLTOPDF
With the generic command line ap & no params so it should "dump usage and terminate", it sucks out the appropriate std::out, then terminates.
Interesting!
JVM version issue? I'm using 1.6.0_23. The latest is... v24. I just checked the change log and don't see anything promising, but I'll try updating anyway.
Okay. Don't let the Input Streams fill or they'll block. Check. .close() can also prevent this, but isn't terribly bright.
That works in general (including the generic command line apps I've tested).
In specific however, it falls down. It appears that wkhtmltopdf is using some terminal manipulation/cursor stuff to do an ASCII-graphic progress bar. I believe this is causing the inputStream to immediately return EOF rather than giving me the correct values.
Any ideas? Hardly a deal-breaker, but it would definitely be Nice To Have.

I had the same exact issue as you and I solved it. Here are my findings:
For some reason, the output from wkhtmltopdf goes to STDERR of the process and NOT STDOUT. I have verified this by calling wkhtmltopdf from Java as well as perl
So, for example in java, you would have to do:
//ProcessBuilder is the recommended way of creating processes since Java 1.5
//Runtime.getRuntime().exec() is deprecated. Do not use.
ProcessBuilder pb = new ProcessBuilder("wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
Process process = pb.start();
BufferedReader errStreamReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//not "process.getInputStream()"
String line = errStreamReader.readLine();
while(line != null)
{
System.out.println(line); //or whatever else
line = reader.readLine();
}
On a side note, if you spawn a process from java, you MUST read from the stdout and stderr streams (even if you do nothing with it) because otherwise the stream buffer will fill and the process will hang and never return.
To futureproof your code, just in case the devs of wkhtmltopdf decide to write to stdout, you can redirect stderr of the child process to stdout and read only one stream like this:
ProcessBuilder pb = new ProcessBuilder("wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader inStreamReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
Actually, I do this in all the cases where I have to spawn an external process from java. That way I don't have to read two streams.
You should also read the streams of the spawned process in different threads if you dont want your main thread to block, since reading from streams is blocking.
Hope this helps.
UPDATE: I raised this issue in the project page and was replied that this is by design because wkhtmltopdf supports giving the actual pdf output in STDOUT. Please see the link for more details and java code.

A process has 3 streams: input, output and error. you can read both output and error stream at the same time using separate processes. see this question and its accepted answer and also this one for example.

You should read from the streams in a different thread.

final Semaphore semaphore = new Semaphore(numOfThreads);
final String whktmlExe = tmpwhktmlExePath;
int doccount = 0;
try{
File fileObject = new File(inputDir);
for(final File f : fileObject.listFiles()) {
if(f.getAbsolutePath().endsWith(".html")) {
doccount ++;
if(doccount >500 ) {
LOG.info(" done with conversion of 1000 docs exiting ");
break;
}
System.out.println(" inside for before "+semaphore.availablePermits());
semaphore.acquire();
System.out.println(" inside for after "+semaphore.availablePermits() + " ---" +f.getName());
new java.lang.Thread() {
public void run() {
try {
String F_ = f.getName().replaceAll(".html", ".pdf") ;
ProcessBuilder pb = new ProcessBuilder(whktmlExe , f.getAbsolutePath(), outPutDir + F_ .replaceAll(" ", "_") );//"wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader errStreamReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line = errStreamReader.readLine();
while(line != null)
{
System.err.println(line); //or whatever else
line = errStreamReader.readLine();
}
System.out.println("after completion for ");
} catch (Exception e) {
e.printStackTrace();
}finally {
System.out.println(" in finally releasing ");
semaphore.release();
}
}
}.start();
}
}
}catch (Exception ex) {
LOG.error(" *** Error in pdf generation *** ", ex);
}
while (semaphore.availablePermits() < numOfThreads) {//till all threads finish
LOG.info( " Waiting for all threads to exit "+ semaphore.availablePermits() + " --- " +( numOfThreads - semaphore.availablePermits()));
java.lang.Thread.sleep(10000);
}