In my program, I need to run a external command in a Ubuntu environment (ntpdate) using java. Currently my code looks like this:
Runtime rt = Runtime.getRuntime();
byte[] readBuffer = new byte[131072];
// Exec a process to do the query
Process p = null;
try {
p = rt.exec("ntpdate -q " + ip);
} catch (Exception ex) {
ex.printStackTrace();
}
if(p!= null){
try {
Thread.sleep(1000);
} catch (Exception e) {
}
// Read the input stream, copy it to the file
InputStream in = p.getInputStream();
try {
int count = 0, rc;
while ((rc = in.read(readBuffer, count, readBuffer.length - count)) != -1) {
count += rc;
if (count >= readBuffer.length) {
p.destroy();
break;
}
}
p.destroy();
result = processOutput(readBuffer, count);
} catch (IOException ex) {
ex.printStackTrace();
}
p.destroy();
This code need to be ran simultaneously on multiple threads in order to maximize performance (I need to test a list of 1.000.000 addresses using ntpdate). However, it runs very slowly, barely consuming machine processing. What am I doing wrong? How could I make this more efficient?
The same problem arises when trying to execute "dig" using .exec(), so I doubt it is because of the specific program being called. Is there some restriction in using Runtime.exec() in a multi Threaded environment?
Is Java the most appropriate approach here? Perhaps this would be better in a shell script, which calls ntpdate in the background multiple times? I'm not sure what benefit you're getting from this code snippet by doing this in Java.
What are you doing with the InputStream from the process?
A bash script could do this like:
for ip in #...IP list
do
ntpdate -q $ip > $ip.txt &
done
Why are you waiting for 1 second at each time ?
try {
Thread.sleep(1000);
} catch (Exception e) {
}
This will do nothing but slowing the execution of your application.
Not sure why it's slow but you need to do a lot more to close your resources. Runtime.exec() needs quite a bit of care and attention to avoid hang-ups and leaking of file descriptors.
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html
Are you sure the issue isn't ntpdate? If ntpdate is just sitting there waiting for a server response and has a large timeout value, then your application is going to sit there too.
Try calling ntpdate with a timeout of 0.2 and see if it makes a difference.
Also, as you're opening streams in your code, you definitely want to explicitly .close() them when you're done. Otherwise it might not happen until a GC which could be a very long time away.
I think I found the solution, and that is that there is no solution using java's Runtime.exec(). The problem seems to be that all calls to start a process are synchronized. Indeed, if you start each process alone (via synchronization) you get the exact same result of starting all processes together.
Are there any alternatives to exec? Otherwise, I will need to get some solution without linux's ntpdate...
I notice that both of the commands you tried involve network round-trips. How is the speed if you call something like echo or cat instead?
Related
I've had a bit of an issue, and I'm pretty new at Java. For context, I'm making a GUI in Javafx, with a Batch backend. The GUI only has to call the Batch script(core.bat) once, and know when it has ended after it has been called. Currently, I call the Batch script using the following code:
Runtime runtime = Runtime.getRuntime();
try {
Process p1 = runtime.exec("cmd /c start core.bat");
} catch(IOException ioException) {}
The issue comes into play when I need to know when the Batch script has ended. I would give examples of what I've tried so far, but I've tried so many things over the past three hours and overwritten my code so many times that I just don't know anymore.
Any solution will do. I just need some way to determine when the Batch script has ended, after which it will close itself. Due to this, knowing when the Batch script ends is not the only possible method.
Knowing when the instance of command prompt running the script ends is also an option.
Any solutions are helpful, thank you.
Take a look at Process#waitFor. It waits until the process has finished.
You might need to start a new Thread if you don't want to wait for it but only get notified/execute code when the program is finished(blocking seems not like a good idea in JavaFX):
Runtime runtime = Runtime.getRuntime();
try {
Process p1 = runtime.exec("core.bat");
Thread t=new Thread(()->{
try{
p1.waitFor();
//your code
}catch(InterruptedException e){
Thread.currentThread.interrupt();//not actually needed, but I think it is a good practise and...SonarLint :)
}
});
t.setDaemon(true);
t.start();
} catch(IOException ioException) {}
Also, you executed the cmd command start, that starts a new process that you cannot control that easily. Just execute core.bat or cmd /c core.bat
setDaemon(true); marks your Thread as a daemon Thread, that does not affect the end of the Program(ends if all Threads that are no daemon Threads finished).
I'm working on a big system, which I have written entirely in Java. But I'm also at some point communicating with a C program by running it as a process, communicating some input to it through the file system and then waiting for it to finish to read the results from it. For now, I can't do anything but this. There is no time for me to link them via an API or anything like that.
This worked well until I needed to invoke this process twice to get the results. One I make the first invocation, it works just fine. However, for the second invocation the process just hangs! I don't if it is sleeping and waiting for a signal but I don't see why it should do so.
Here is the method to do the invocation:
public synchronized boolean processCommand(List command) {
try {
ProcessBuilder pb = new ProcessBuilder(command);
Process p = pb.start();
p.waitFor();
p.destroy();
} catch(Exception ex) { return false; }
return true;
}
I really do not need to communicate to the stdout or stdin. I just need the process to run and finish its job. But it just hangs when the calling process is waiting for it ONLY the second time I call it!
My calling code is just simply making the command list and calling this method from another java object.
Both calls to the method processCommand(List command) work fine when the input to the C program is smaller. Would that be some issue with the stdin or stdout?
It's just driving me crazy! Anybody has an insight into this? I appreciate your commend :)
Update:
Here is the solution based on what #Gray mentioned:
I just need to drain the InputStream and possibly the ErrorStream:
public synchronized boolean processCommand(List command) {
try {
ProcessBuilder pb = new ProcessBuilder(command);
Process p = pb.start();
handleStream(p.getInputStream);
handleStream(p.getErrorStream);
p.waitFor();
p.destroy();
} catch(Exception ex) { return false; }
return true;
}
public void handleStream(InputStream input) {
try {
int c;
while( (c=input.read())!= -1) { //Anything }
} catch(Exception ex) { }
}
Try following the solution suggested here (The original link seems to be unavailable; an archived version can be found here, although the referenced link to the example code still points to the original site...)
The best strategy for handling this situation is to kick off a thread just prior to calling waitFor that will interrupt the current thread after a suitable time period. TimerTask was designed for precisely this sort of situation, and waitFor is very responsive to interrupts in most cases. [Follow link in original for an example].
[...]
Java 6 API clearly states that failure to promptly “read the output stream of the subprocess may cause the subprocess to block, and even deadlock.”
[...]
The safe bet for handling this problem today is to explicitly clean up every instance of Process by calling close on each stream made available through Process.getOutputSteam, Process.getInputStream, and Process.getErrorStream, and then call Process.destroy even if the process is already terminated.
I am writing this question which is related to my previous topic:
Run bat file from java code to get desired result in txt file - no can do :(
In a shortcut: i wrote a program in java that runs a bat file. This bat file runs TestComplete8 script that performs desktop application test. After test is finished, bat file generates file called result.txt and prints information about test to it.
I'm stuck with another issue right now: Now from my java code i would like to wait until the bat run is finished. I do that by looping until the file called result.txt exists. Not the nicesest solution i guess but I thought it could work, also tried different solutions. What happens is that it will loop fine and wait until file exists, but testcomplete doesn't perform the test. It is very strange, because testcomplete runs, i can see that test starts, my AUT starts as well, but than nothing happens. Testcomplete is waiting for any object and doesn't click anywhere just waits until predefined time for action runs out. When i run the test without any waiting done in code, everything is fine. I just don't understand why nothing happens during the test when waiting is enabled and why it works fine when i just remove any do - while or waitFor(), or even i tried running it in seperate threads. :(
I have a feeling that it may be somehow related to the OS and have something to do with processes as it runs something like a bat as process and than bat runs it's child process as testcomplete or sth like that.
Thanks for any answers
Source code as asked:
Right now i was trying a solution with modified bat file:
#ECHO OFF
"C:\Program Files (x86)\Automated QA\TestComplete 8\Bin\TestComplete.exe" "C:..." /r /p:projname PathToApp="C:\...p" Login=... Password=B1 /t:"KeywordTests|..." /exit
and the code to run and wait in latest version is:
new Thread(new Runnable() {
public void run() {
File file = new File("D:\\");
int exitValue = -1;
try {
Process process = Runtime.getRuntime().exec(batch, null, file);
while (true) {
try {
exitValue = process.exitValue();
System.out.println(exitValue);
break;
} catch (IllegalThreadStateException e) {
// e.printStackTrace();
try {
Thread.sleep(1000);
} catch (InterruptedException e1) {
e1.printStackTrace();
}
System.out.println("Waiting for process...");
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
}).start();
The most likely cause, without knowing more about the problem, is a common one faced when launching external processes from Java. When launching an external process three streams are created between the parent and child process, input, output, error.
You can liken these to System.in, System.out and System.err. If the parent process (Java) does not actively consume the data on the out and error streams the child process may block as the OS will reach a buffer limit on the stream and prevent any more being written until it is consumed. This is quite likely if your script writes to standard out or standard error.
I would recommend using apache commons-exec to handle Java process launching.
Here's a code sample that I know works.
CommandLine commandLine = new CommandLine( "TestComplete8.bat" );
commandLine.addArgument( ... );
commandLine.addArgument( ... );
DefaultExecutor executor = new DefaultExecutor();
executor.setExitValue( 0 );
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ByteArrayOutputStream errorStream = new ByteArrayOutputStream();
executor.setStreamHandler( new PumpStreamHandler( outputStream, errorStream ) );
try
{
executor.execute( commandLine );
}
catch ( ExecuteException e )
{
// TODO: ...
}
catch ( IOException e )
{
// TODO: ...
}
Then you can examine the output/error streams if you wish when execute returns.
I want to create a utility in Java to automate our building process. The current process involve
Opening 2 consoles for servers. ( I want to open these consoles from java program )
Running mulitple bat files in consoles and based on one batch file output, running other commands.
I need head start, what libraries should i use. Can i open 2 consoles from Java (independently). Like even if my program closes those consoles keep running. (consoles are bea server, startWebLogic.cmd).
Alee, yes you can do that with Runtime.getRuntime().exec("file.bat"); and then you have 2 options, you can capture the output of the execution
for example:
public static void main(String[] args) {
System.out.println("hello");
try {
Process p = Runtime.getRuntime().exec("./prog.sh");
InputStream in = p.getInputStream();
System.out.println("OUTPUT");
StringBuilder sb = new StringBuilder();
int c;
while( (c = in.read() ) > 0 ) {
sb.append((char) c);
}
//here the script finished
String output = sb.toString();
if( output.contains("Exception")) {
System.out.println("script failed");
}
if( p.exitValue() == 0) {
System.out.println("The script run without errors");
} else {
System.out.println("The script failed");
}
} catch (IOException e) {
e.printStackTrace();
}
}
this captures both scenarios, where you need to capture the output and then decide whether the script run successfully, or if you can use the exit status from the script.
The exitValue code is 0 for success and any other number for failure.
Surely you can open as many consoles as you want. If you wish to do it simultaneously creaete separate threads and then create process using Runtime.exec() or ProcessBuilder.
But why do you want to do this? There is a good old ant that is a build tool dedicated for such tasks. It supports everything and it is extendable using custom tasks.
Moreover if you suddenly remember that it is 2011 now, use newer tools like Apache Buildr.
Java's Runtime class has methods to launch Processes which have programmatic interfaces to their input/output streams and other management. However, I'm not sure how Windows handles processes whose parents have died.
You should also consider using the Windows Script Host via VB or JScript as you will probably have finer control.
I have a script (KSH script to be precise) that downloads 3 files from a FTP server, using the curl command.
When I run manually my script, i.e. by executing the command ./ftp_download.sh XXX (XXX are the parameters for the script), the download finishes correctly.
As I want to run the script from a Java program, I have made a short Java class that contains exactly that:
public class Run {
private static final String CMD = "/.../sh/ftp_download.sh XXX";
public static void main(String[] args) {
System.out.println("========================================================");
BufferedReader out = null;
try {
long startTime = System.currentTimeMillis();
String strOutputline;
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Date now = new Date();
Process processus = Runtime.getRuntime().exec(CMD);
out = new BufferedReader(new InputStreamReader(processus.getInputStream()));
while ((strOutputline = out.readLine()) != null) {
now.setTime(System.currentTimeMillis());
System.out.println(sdf.format(now) + " " + strOutputline);
}
System.out.println("RESULT : " + processus.waitFor());
out.close();
processus.destroy();
long duration = System.currentTimeMillis() - startTime;
System.out.println("Duration : " + duration);
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (out != null) {
out.close();
}
} catch (Exception e) {
e.printStackTrace();
}
}
System.out.println("========================================================");
System.out.println("END");
}
}
However, when I run this simple program, it simply freezes after exactly 3m20 (this duration is always the same, even if I run the Java program several times).
By freezing, I mean that the Java program is still running (the curl process too) but the downloaded file is not growing anymore (i.e. curl does not continue to download any data)...
Thus, I never get the RESULT: xxx line printed in the console...
What can explain this strange behavior?
ps: In the near future, I will change my project in order to download these files using the Apache commons-net library, but I really want to understand this strange behavior!
Thanks to derobert, I finally manage to solve this issue. Some explanations: in normal mode, curl is displaying a progress information (a table with the amount of data downloaded, the remaining time, etc.). After some time, the buffer seems to be completly filled, and that's why the process freezes...
I'm not a Java person, but rather a Unix one, and one thing seems obvious: The buffer on either stdout or stderr is filling up, and then curl is blocking.
Does it work if you run curl in silent mode, as in curl --silent?
Checking the Java documentation, it looks like you want to also use getErrorStream in addition to getInputStream.
From Process:
The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
So you need to get the standard output and the error output of the process and read them until they’re empty (i.e. returning -1 on read()).