Develop Reference

Calling python script from java web application with Processbuilder

I am working on a simple meteo station - I want to use raspberry pi 3b+ as a host, dht22 sensor and write a web application in Java (with spring boot, then deploy it to tomcat 8) and Python for retrieving sensor's data.
What I've done so far:
Python application for retrieving and displaying data. Works as expected, it just prints something like "22.5;37.4":
import Adafruit_DHT
DHT_SENSOR = Adafruit_DHT.DHT22
DHT_PIN = 4
humidity, temperature = Adafruit_DHT.read_retry(DHT_SENSOR, DHT_PIN)
if humidity is not None and temperature is not None:
print("{0:0.1f};{1:0.1f}".format(temperature, humidity))
else:
print("FAIL")
Then I've wrote a java application, put it into .jar and checked if I am able to get sensor's data. Not a rocket science, also works as expected when I use java -jar InputTest.jar on my raspberry pi:
public static void main(String[] args) {
try {
ProcessBuilder pb = new ProcessBuilder("python", "/home/pi/Desktop/input/dht_once.py");
Process process = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null) {
System.out.println("measured: " + line);
}
process.waitFor();
} catch (IOException ) {
System.out.println(" exception " + e.getLocalizedMessage());
}
}
Then I've created a spring boot application, put my java code inside (logic same as above), packed as a war, deployed to tomcat 8 and run it. It turned out nothing is being printed (of course I've changed code to log output to logfile, it works fine, I can see other logs inside). No issues in logs, it looks like reader never returns a line.
I believe application does not wait for a process to produce output, but I have no idea why. Important thing: it takes up to few seconds to produce sensor's output. I've also changed python script just for test purposes to return value immediately:
print("22.4;33.0")
and it results in successful read by java web application. But when it has to wait few seconds for the output it kills process (process.isAlive() is false right after while loop).
I've also tried to play with sleep() on current thread to force it to wait for python process but no success.
Do you guys have any idea what can be the reason for this behavior? Is there anything more I should check?
TLDR;
Java application which creates python process works fine until I run it as a web application - then it looks like it does not wait for a process' output

I haven't found a solution yet, however I've implemented workaround/cleaner solution.
I've decided to separate totally java and python code and created microservice for data retrieving. I use flask for rest webservice (followed this tutorial https://docs.dataplicity.com/docs/control-gpios-using-rest-api) and call it directly from java.
Since this does not resolve my initial question I do not mark it as an answer, however it might help someone.

Run bat file from java code and wait for it to execute TestComplete script - no can do :(

I am writing this question which is related to my previous topic:
Run bat file from java code to get desired result in txt file - no can do :(
In a shortcut: i wrote a program in java that runs a bat file. This bat file runs TestComplete8 script that performs desktop application test. After test is finished, bat file generates file called result.txt and prints information about test to it.
I'm stuck with another issue right now: Now from my java code i would like to wait until the bat run is finished. I do that by looping until the file called result.txt exists. Not the nicesest solution i guess but I thought it could work, also tried different solutions. What happens is that it will loop fine and wait until file exists, but testcomplete doesn't perform the test. It is very strange, because testcomplete runs, i can see that test starts, my AUT starts as well, but than nothing happens. Testcomplete is waiting for any object and doesn't click anywhere just waits until predefined time for action runs out. When i run the test without any waiting done in code, everything is fine. I just don't understand why nothing happens during the test when waiting is enabled and why it works fine when i just remove any do - while or waitFor(), or even i tried running it in seperate threads. :(
I have a feeling that it may be somehow related to the OS and have something to do with processes as it runs something like a bat as process and than bat runs it's child process as testcomplete or sth like that.
Thanks for any answers
Source code as asked:
Right now i was trying a solution with modified bat file:
#ECHO OFF
"C:\Program Files (x86)\Automated QA\TestComplete 8\Bin\TestComplete.exe" "C:..." /r /p:projname PathToApp="C:\...p" Login=... Password=B1 /t:"KeywordTests|..." /exit
and the code to run and wait in latest version is:
new Thread(new Runnable() {
public void run() {
File file = new File("D:\\");
int exitValue = -1;
try {
Process process = Runtime.getRuntime().exec(batch, null, file);
while (true) {
try {
exitValue = process.exitValue();
System.out.println(exitValue);
break;
} catch (IllegalThreadStateException e) {
// e.printStackTrace();
try {
Thread.sleep(1000);
} catch (InterruptedException e1) {
e1.printStackTrace();
}
System.out.println("Waiting for process...");
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
}).start();

The most likely cause, without knowing more about the problem, is a common one faced when launching external processes from Java. When launching an external process three streams are created between the parent and child process, input, output, error.
You can liken these to System.in, System.out and System.err. If the parent process (Java) does not actively consume the data on the out and error streams the child process may block as the OS will reach a buffer limit on the stream and prevent any more being written until it is consumed. This is quite likely if your script writes to standard out or standard error.
I would recommend using apache commons-exec to handle Java process launching.
Here's a code sample that I know works.
CommandLine commandLine = new CommandLine( "TestComplete8.bat" );
commandLine.addArgument( ... );
commandLine.addArgument( ... );
DefaultExecutor executor = new DefaultExecutor();
executor.setExitValue( 0 );
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ByteArrayOutputStream errorStream = new ByteArrayOutputStream();
executor.setStreamHandler( new PumpStreamHandler( outputStream, errorStream ) );
try
{
executor.execute( commandLine );
}
catch ( ExecuteException e )
{
// TODO: ...
}
catch ( IOException e )
{
// TODO: ...
}
Then you can examine the output/error streams if you wish when execute returns.

Need guidance to automate a multiple .bat file processes

I want to create a utility in Java to automate our building process. The current process involve
Opening 2 consoles for servers. ( I want to open these consoles from java program )
Running mulitple bat files in consoles and based on one batch file output, running other commands.
I need head start, what libraries should i use. Can i open 2 consoles from Java (independently). Like even if my program closes those consoles keep running. (consoles are bea server, startWebLogic.cmd).

Alee, yes you can do that with Runtime.getRuntime().exec("file.bat"); and then you have 2 options, you can capture the output of the execution
for example:
public static void main(String[] args) {
System.out.println("hello");
try {
Process p = Runtime.getRuntime().exec("./prog.sh");
InputStream in = p.getInputStream();
System.out.println("OUTPUT");
StringBuilder sb = new StringBuilder();
int c;
while( (c = in.read() ) > 0 ) {
sb.append((char) c);
}
//here the script finished
String output = sb.toString();
if( output.contains("Exception")) {
System.out.println("script failed");
}
if( p.exitValue() == 0) {
System.out.println("The script run without errors");
} else {
System.out.println("The script failed");
}
} catch (IOException e) {
e.printStackTrace();
}
}
this captures both scenarios, where you need to capture the output and then decide whether the script run successfully, or if you can use the exit status from the script.
The exitValue code is 0 for success and any other number for failure.

Surely you can open as many consoles as you want. If you wish to do it simultaneously creaete separate threads and then create process using Runtime.exec() or ProcessBuilder.
But why do you want to do this? There is a good old ant that is a build tool dedicated for such tasks. It supports everything and it is extendable using custom tasks.
Moreover if you suddenly remember that it is 2011 now, use newer tools like Apache Buildr.

Java's Runtime class has methods to launch Processes which have programmatic interfaces to their input/output streams and other management. However, I'm not sure how Windows handles processes whose parents have died.
You should also consider using the Windows Script Host via VB or JScript as you will probably have finer control.

Help running Java runtime.exec() on multiple threads

In my program, I need to run a external command in a Ubuntu environment (ntpdate) using java. Currently my code looks like this:
Runtime rt = Runtime.getRuntime();
byte[] readBuffer = new byte[131072];
// Exec a process to do the query
Process p = null;
try {
p = rt.exec("ntpdate -q " + ip);
} catch (Exception ex) {
ex.printStackTrace();
}
if(p!= null){
try {
Thread.sleep(1000);
} catch (Exception e) {
}
// Read the input stream, copy it to the file
InputStream in = p.getInputStream();
try {
int count = 0, rc;
while ((rc = in.read(readBuffer, count, readBuffer.length - count)) != -1) {
count += rc;
if (count >= readBuffer.length) {
p.destroy();
break;
}
}
p.destroy();
result = processOutput(readBuffer, count);
} catch (IOException ex) {
ex.printStackTrace();
}
p.destroy();
This code need to be ran simultaneously on multiple threads in order to maximize performance (I need to test a list of 1.000.000 addresses using ntpdate). However, it runs very slowly, barely consuming machine processing. What am I doing wrong? How could I make this more efficient?
The same problem arises when trying to execute "dig" using .exec(), so I doubt it is because of the specific program being called. Is there some restriction in using Runtime.exec() in a multi Threaded environment?

Is Java the most appropriate approach here? Perhaps this would be better in a shell script, which calls ntpdate in the background multiple times? I'm not sure what benefit you're getting from this code snippet by doing this in Java.
What are you doing with the InputStream from the process?
A bash script could do this like:
for ip in #...IP list
do
ntpdate -q $ip > $ip.txt &
done

Why are you waiting for 1 second at each time ?
try {
Thread.sleep(1000);
} catch (Exception e) {
}
This will do nothing but slowing the execution of your application.

Not sure why it's slow but you need to do a lot more to close your resources. Runtime.exec() needs quite a bit of care and attention to avoid hang-ups and leaking of file descriptors.
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

Are you sure the issue isn't ntpdate? If ntpdate is just sitting there waiting for a server response and has a large timeout value, then your application is going to sit there too.
Try calling ntpdate with a timeout of 0.2 and see if it makes a difference.
Also, as you're opening streams in your code, you definitely want to explicitly .close() them when you're done. Otherwise it might not happen until a GC which could be a very long time away.

I think I found the solution, and that is that there is no solution using java's Runtime.exec(). The problem seems to be that all calls to start a process are synchronized. Indeed, if you start each process alone (via synchronization) you get the exact same result of starting all processes together.
Are there any alternatives to exec? Otherwise, I will need to get some solution without linux's ntpdate...

I notice that both of the commands you tried involve network round-trips. How is the speed if you call something like echo or cat instead?

unable to run ksh script from java: java.io.IOException: .: not found

I tried to run a shell script from java code, but I am facing problem. The script is in batchstart.sh file -
#!/bin/ksh
export DISPLAY=:0.0
Now the script is run with a dot on the command line -- . batchstart.sh
How do I run it from java? My java code is below. It throws the following exception -
java.io.IOException: .: not found
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:102)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
at java.lang.Runtime.exec(Runtime.java:591)
at java.lang.Runtime.exec(Runtime.java:429)
at SetDisplay.main(SetDisplay.java:12)
import java.io.*;
public class SetDisplay {
public static void main(String[] args) {
File wd = new File("/myhomedir/");
System.out.println("Working Directory: " +wd);
Process proc = null;
try {
proc = Runtime.getRuntime().exec(". batchstart.sh", null, wd);
} catch (Exception e) {
e.printStackTrace();
}
}
}
How do I make the shell script run ?
I tried the following code as well, but that too doesn't work.
File wd = new File("/bin");
System.out.println(wd);
Process proc = null;
try {
proc = Runtime.getRuntime().exec("/bin/bash", null, wd);
}
catch (IOException e) {
e.printStackTrace();
}
if (proc != null) {
BufferedReader in = new BufferedReader(new InputStreamReader(proc.getInputStream()));
PrintWriter out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(proc.getOutputStream())), true);
out.println("cd /home/");
out.println(". batchstart.sh");
out.println("exit");
try {
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
proc.waitFor();
in.close();
out.close();
proc.destroy();
}
catch (Exception e) {
e.printStackTrace();
}
}

When run from the command line, using a dot at the start of a script indicates that the script should be run in the current environment, instead of spawning a new subshell and using a new copy of the current environment. This allows you to export a new value of an environment variable to be used by commands run later from the same interactive shell.
Obviously, this technique only works if you are running your batchstart.sh script from an actual shell. Java does not know how this mechanism works and so the dot means nothing to it. A script cannot modify the environment of the Java process it was called from.
If your goal is to change the value of the DISPLAY environment variable for other commands run by your Java process, consider using the ProcessBuilder class to specify a new environment for the child process. Java does not contain a built-in way to modify variables in its own environment.

The source command (".") is a shell built-in. You have to explicitly run /bin/ksh, passing your script name as the argument (followed by any script arguments).
You have a larger problem if you need to source the script. That usually means that environment changes happen in the context of the current shell, not a subshell.
This won't work with Java since Java's not a shell. You'll need to figure out how to change the environment with Java.
Of course, if I'm wrong and there's more to that script that just setting DISPLAY, it may work as suggested.
The method you're going to have to use depends on what you're trying to achieve(as in "Are you running other programs using exec() that rely on DISPLAY being set?" or "Does your Java program need DISPLAY to be set?").
If, as you state in your comment, it's only your Java program that needs DISPLAY set, just set it outside before your program runs. Create a cmd (or bash) file which sets the DISPLAY variable then calls the JRE to run your program.
#/bin/ksh
export DISPLAY-:0.0
/usr/bin/jre/java your_program blah.blah.blah
I would also modify your main() to check that it's set to something and exit gracefully if not:
if (System.getenv ("DISPLAY") == null)
// doesn't exist, exit gracefully.

The period "." is a shell built-in, and executes the script "in-place", analogous to #include in C/C++. Using "." outside of a shell-script has no meaning.
If you want to run the script from Java, you have to execute the script interpreter (/bin/ksh):
Runtime.getRuntime().exec("/bin/ksh batchstart.sh", ...)
but note that this is not semantically equivalent, since you're executing batchstart.sh as a sub-process instead of sourcing it.