I'm working on a big system, which I have written entirely in Java. But I'm also at some point communicating with a C program by running it as a process, communicating some input to it through the file system and then waiting for it to finish to read the results from it. For now, I can't do anything but this. There is no time for me to link them via an API or anything like that.
This worked well until I needed to invoke this process twice to get the results. One I make the first invocation, it works just fine. However, for the second invocation the process just hangs! I don't if it is sleeping and waiting for a signal but I don't see why it should do so.
Here is the method to do the invocation:
public synchronized boolean processCommand(List command) {
try {
ProcessBuilder pb = new ProcessBuilder(command);
Process p = pb.start();
p.waitFor();
p.destroy();
} catch(Exception ex) { return false; }
return true;
}
I really do not need to communicate to the stdout or stdin. I just need the process to run and finish its job. But it just hangs when the calling process is waiting for it ONLY the second time I call it!
My calling code is just simply making the command list and calling this method from another java object.
Both calls to the method processCommand(List command) work fine when the input to the C program is smaller. Would that be some issue with the stdin or stdout?
It's just driving me crazy! Anybody has an insight into this? I appreciate your commend :)
Update:
Here is the solution based on what #Gray mentioned:
I just need to drain the InputStream and possibly the ErrorStream:
public synchronized boolean processCommand(List command) {
try {
ProcessBuilder pb = new ProcessBuilder(command);
Process p = pb.start();
handleStream(p.getInputStream);
handleStream(p.getErrorStream);
p.waitFor();
p.destroy();
} catch(Exception ex) { return false; }
return true;
}
public void handleStream(InputStream input) {
try {
int c;
while( (c=input.read())!= -1) { //Anything }
} catch(Exception ex) { }
}
Try following the solution suggested here (The original link seems to be unavailable; an archived version can be found here, although the referenced link to the example code still points to the original site...)
The best strategy for handling this situation is to kick off a thread just prior to calling waitFor that will interrupt the current thread after a suitable time period. TimerTask was designed for precisely this sort of situation, and waitFor is very responsive to interrupts in most cases. [Follow link in original for an example].
[...]
Java 6 API clearly states that failure to promptly “read the output stream of the subprocess may cause the subprocess to block, and even deadlock.”
[...]
The safe bet for handling this problem today is to explicitly clean up every instance of Process by calling close on each stream made available through Process.getOutputSteam, Process.getInputStream, and Process.getErrorStream, and then call Process.destroy even if the process is already terminated.
Related
I've had a bit of an issue, and I'm pretty new at Java. For context, I'm making a GUI in Javafx, with a Batch backend. The GUI only has to call the Batch script(core.bat) once, and know when it has ended after it has been called. Currently, I call the Batch script using the following code:
Runtime runtime = Runtime.getRuntime();
try {
Process p1 = runtime.exec("cmd /c start core.bat");
} catch(IOException ioException) {}
The issue comes into play when I need to know when the Batch script has ended. I would give examples of what I've tried so far, but I've tried so many things over the past three hours and overwritten my code so many times that I just don't know anymore.
Any solution will do. I just need some way to determine when the Batch script has ended, after which it will close itself. Due to this, knowing when the Batch script ends is not the only possible method.
Knowing when the instance of command prompt running the script ends is also an option.
Any solutions are helpful, thank you.
Take a look at Process#waitFor. It waits until the process has finished.
You might need to start a new Thread if you don't want to wait for it but only get notified/execute code when the program is finished(blocking seems not like a good idea in JavaFX):
Runtime runtime = Runtime.getRuntime();
try {
Process p1 = runtime.exec("core.bat");
Thread t=new Thread(()->{
try{
p1.waitFor();
//your code
}catch(InterruptedException e){
Thread.currentThread.interrupt();//not actually needed, but I think it is a good practise and...SonarLint :)
}
});
t.setDaemon(true);
t.start();
} catch(IOException ioException) {}
Also, you executed the cmd command start, that starts a new process that you cannot control that easily. Just execute core.bat or cmd /c core.bat
setDaemon(true); marks your Thread as a daemon Thread, that does not affect the end of the Program(ends if all Threads that are no daemon Threads finished).
How do I run an external command (via a shell) from a Java program, such that no redirection takes place, and wait for the command to end? I want the file descriptors of the external program to be the same as those of the Java program. In particular I do not want the output to be redirected to a pipe that the Java program is reading. Having the Java program relay the output is not a solution.
This means that a plain invocation of java.lang.Runtime.exec is not the solution. I presume that java.lang.ProcessBuilder is involved, but how do I specify that output and error streams must be the same as the calling Java process?
class A {
public static void main(String[] args) {
try {
ProcessBuilder pb = new ProcessBuilder("echo", "foo");
/*TODO: pb.out = System.out; pb.err = System.err;*/
Process p = pb.start();
p.waitFor();
} catch (Exception e) {
System.err.println(e);
System.exit(1);
}
}
}
(This may or may not be the right approach.)
In other words, I'm looking for Java's system, but all I can find is (roughly) popen.
Here's an example of a situation where relaying cannot work: if the subprocess writes to both stdout and stderr and the Java program is relaying, then the Java program has no way to know the order of the write calls in the subprocess. So the order of the output on stdout and stderr from the Java program will be observably different if the two streams end up in the same file. Mixing stdout and stderr is of course not a solution because the caller may want to keep them separate.
While I think this question is of general interest, a Linux-specific solution would solve my immediate problem.
This is the intent of ProcessBuilder.redirectError/redirectOutput which were introduced in Java 7. Using Redirect.INHERIT will make the child process share stderr/stdout with the Java process:
class A {
public static void main(String[] args) {
try {
ProcessBuilder builder = new ProcessBuilder("echo", "foo");
builder.redirectError(ProcessBuilder.Redirect.INHERIT);
builder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
Process p = builder.start();
p.waitFor();
} catch (Exception e) {
System.err.println(e);
System.exit(1);
}
}
}
You might take a look at the NuProcess project. Disclaimer: I wrote it. It provides non-blocking I/O from spawned processes. You still have to relay in Java (you receive callbacks), but because it is using epoll() in the case of Linux, I would expect it to preserve the order of the underlying program. Only a single thread is epoll()'ing the pipes so you won't get any thread scheduling order issues.
I'm 100% order would be preserved on MacOS X, or any BSD variant, because it uses a kqueue which is definitely ordered. Anyway, you might want to give it a shot, it's trivial to code and test.
You can't. By default all standard I/O of the child process are redirected to the parent process (the jvm running your java program).
from the javadoc of the Process class:
By default, the created subprocess does not have its own terminal or
console. All its standard I/O (i.e. stdin, stdout, stderr)
operations will be redirected to the parent process, where they can be
accessed via the streams obtained using the methods getOutputStream(),
getInputStream(), and getErrorStream(). The parent process uses these
streams to feed input to and get output from the subprocess. Because
some native platforms only provide limited buffer size for standard
input and output streams, failure to promptly write the input stream
or read the output stream of the subprocess may cause the subprocess
to block, or even deadlock.
Let me explain my software. What my software simply does is that it creates 10 threads and assigns a number of tasks to each thread. Each thread then creates a Runtime Process that will start a cmd batch file which in turn will start a program that will telnet to a device (I have about 200 of them) to poll its configuration. Here is the code for my process creation:
Process p1 = java.lang.Runtime.getRuntime().exec("cmd /c start /b /wait " + batchFile);
int returnVal = p1.waitFor();
batchFile is the full path of the batch file. Don't get me wrong, the software works fine up to 100% of the execution and it hanged ONLY ONE TIME at about 95%, so I'm trying to find a solution for that. Why it hanged is not my issue now, but rather how to deal with hangups later on..!
Now the problem is that I need to wait for the process to finish because my telnet client will write to a file that I will read later on in the thread; and hence the use of the .waitFor() . My question is how can I get the thread to understand that the external program hanged? In other words, can I give the external program some time limit to finish; and if it does not the thread will kill the process?
Also I have read about reading the error and output streams; however, I don't think it is applicable here, or is it?
Also I have read about reading the error and output streams; however, I don't think it is applicable here, or is it?
Almost certainly yes, it is applicable. Try using ProcessBuilder instead of Runtime.exec and read all the output before calling waitFor. For example (exception handling omitted, in particular waitFor may throw InterruptedException before the process actually exited)
ProcessBuilder pb = new ProcessBuilder(
"cmd", "/c", "start", "/b", "/wait", batchFile);
pb.redirectErrorStream(true);
Process p = pb.start();
IOUtils.closeQuietly(p.getOutputStream());
IOUtils.copy(p.getInputStream(), System.out);
// or alternatively throw away the output using
// IOUtils.copy(p.getInputStream(), NullOutputStream.NULL_OUTPUT_STREAM);
IOUtils.closeQuietly(p.getInputStream());
int returnVal = p.waitFor();
(IOUtils and NullOutputStream are from Apache commons-io).
To answer the actual question in the title, if your process still hangs even after properly reading its output, you may need to use p.destroy() to forcibly terminate the process. You could define a timer task something like
public class TimeoutProcessKiller extends TimerTask {
private Process p;
public TimeoutProcessKiller(Process p) {
this.p = p;
}
public void run() {
p.destroy();
}
}
and then do
// single shared timer instance
Timer t = new Timer();
// for each device
Process p = pb.start();
TimerTask killer = new TimeoutProcessKiller(p);
t.schedule(killer, 30000);
// ... read output stream as before ...
int returnVal = p.waitFor();
killer.cancel();
This will cause the process to be killed if it has been running for more than 30 seconds (adjust the schedule call to change the timeout).
I am having an issue with some process wrapping, and it's only occurring in Windows XP. This code works perfectly in Windows 7. I'm really stumped as to why the streams are empty in XP. I've also tried using the String[] version of Process.Exec() and it made no difference.
I am using the following class to read from the process' STDOUT and STDERR (an instance for each stream):
import java.util.*;
import java.io.*;
public class ThreadedStreamReader extends Thread{
InputStream in;
Queue messageQueue;
public ThreadedStreamReader(InputStream s, Queue q)
{
in = s;
messageQueue = q;
}
public void run()
{
try
{
BufferedReader r = new BufferedReader(new InputStreamReader(in));
String line = null;
while((line = r.readLine()) != null)
{
synchronized(messageQueue)
{
messageQueue.add(line);
}
}
}catch(Exception e)
{
System.err.println("Bad things happened while reading from a stream");
}
}
}
And I use it here:
Process p = Runtime.getRuntime().exec("test.exe");
Queue<String> q = new LinkedList<String>();
ThreadedStreamReader stdout = new ThreadedStreamReader(p.getInputStream(), q);
ThreadedStreamReader stderr = new ThreadedStreamReader(p.getErrorStream(), q);
stdout.start();
stderr.start();
while(true)
{
while(q.size() > 0)
{
System.out.println(q.remove());
}
}
Anyone have any ideas? Thanks!
Edit: Added synchronization
Edit: Just as an update, the parent stream readers are blocked on their read operation. If I kill the child processes, with the task manager, they read in the null from the closing of the stream.
You need to use a threadsafe data structure; I don't think LinkedList is threadsafe.
One mistake that strikes me is that LinkedList is not synchronized, but you're trying to write to it in 2 threads.
Another thing to keep in mind is Process.getInputStream() returns the stdout stream of the process, so you should rename the variable currently called stdin to stdout to prevent confusion.
There are known bugs in pre-Vista Windows operating systems where loading DLLs can cause a hang in IO.
e.g. see http://weblogs.java.net/blog/kohsuke/archive/2009/09/28/reading-stdin-may-cause-your-jvm-hang and https://connect.microsoft.com/VisualStudio/feedback/details/94701/loadlibrary-deadlocks-with-a-pipe-read
I'm not sure if this is what you are running in to, but it may be related.
Also, I vaguely recall some issues in getting a valid stdin and stdout from non-console windows applications. If your call to 'test.jar' is using 'javaw' rather than 'java', then this could be the cause of your problem, too.
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
In my program, I need to run a external command in a Ubuntu environment (ntpdate) using java. Currently my code looks like this:
Runtime rt = Runtime.getRuntime();
byte[] readBuffer = new byte[131072];
// Exec a process to do the query
Process p = null;
try {
p = rt.exec("ntpdate -q " + ip);
} catch (Exception ex) {
ex.printStackTrace();
}
if(p!= null){
try {
Thread.sleep(1000);
} catch (Exception e) {
}
// Read the input stream, copy it to the file
InputStream in = p.getInputStream();
try {
int count = 0, rc;
while ((rc = in.read(readBuffer, count, readBuffer.length - count)) != -1) {
count += rc;
if (count >= readBuffer.length) {
p.destroy();
break;
}
}
p.destroy();
result = processOutput(readBuffer, count);
} catch (IOException ex) {
ex.printStackTrace();
}
p.destroy();
This code need to be ran simultaneously on multiple threads in order to maximize performance (I need to test a list of 1.000.000 addresses using ntpdate). However, it runs very slowly, barely consuming machine processing. What am I doing wrong? How could I make this more efficient?
The same problem arises when trying to execute "dig" using .exec(), so I doubt it is because of the specific program being called. Is there some restriction in using Runtime.exec() in a multi Threaded environment?
Is Java the most appropriate approach here? Perhaps this would be better in a shell script, which calls ntpdate in the background multiple times? I'm not sure what benefit you're getting from this code snippet by doing this in Java.
What are you doing with the InputStream from the process?
A bash script could do this like:
for ip in #...IP list
do
ntpdate -q $ip > $ip.txt &
done
Why are you waiting for 1 second at each time ?
try {
Thread.sleep(1000);
} catch (Exception e) {
}
This will do nothing but slowing the execution of your application.
Not sure why it's slow but you need to do a lot more to close your resources. Runtime.exec() needs quite a bit of care and attention to avoid hang-ups and leaking of file descriptors.
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html
Are you sure the issue isn't ntpdate? If ntpdate is just sitting there waiting for a server response and has a large timeout value, then your application is going to sit there too.
Try calling ntpdate with a timeout of 0.2 and see if it makes a difference.
Also, as you're opening streams in your code, you definitely want to explicitly .close() them when you're done. Otherwise it might not happen until a GC which could be a very long time away.
I think I found the solution, and that is that there is no solution using java's Runtime.exec(). The problem seems to be that all calls to start a process are synchronized. Indeed, if you start each process alone (via synchronization) you get the exact same result of starting all processes together.
Are there any alternatives to exec? Otherwise, I will need to get some solution without linux's ntpdate...
I notice that both of the commands you tried involve network round-trips. How is the speed if you call something like echo or cat instead?