I hava command line which process more than 5 mins. when I invoke command line with ProcessBuilder, it works the command completes the job with in 5 mins.
Whereas the process hangs if it takes more than 5 mins and shows no improvement on process until I quit the process.
p = new ProcessBuilder("myprogram","with","parameter").start();
p.waitFor();
Please let me know if you doesn't understand the above question?
The problem might be, that command "myprogram" produces some output, and you are not reading it. This means that the process is blocked as soon as the buffer is full and waits for your process to continue reading. Your process in turn waits for the other process to finish (which it won't because it waits for your process, ...). This is a classical deadlock situation.
You need to continually read from the processes input stream to ensure that it doesn't block.
Javadocs says:
Class Process
Because some native platforms only provide limited buffer size for
standard input and output streams, failure to promptly write the input
stream or read the output stream of the subprocess may cause the
subprocess to block, and even deadlock.
Fail to clear the buffer of input stream (which pipes to the output
stream of subprocess) from Process may lead to a subprocess blocking.
Related
I have a Python script wherein a JAR is called. After the JAR is called, two shell scripts are called. Initially I was doing this:
proc = subprocess.Popen(jar_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.wait()
output, errors = proc.communicate()
proc = subprocess.Popen(prune_command, shell=True)
proc.wait()
proc = subprocess.call(push_command, shell=True)
I have to wait for the first two processes to finish so I use Popen() and the final one I can let it run in the background, so I call() it. I pass shell=True because I want the called shell scripts to have access to environment variables.
The above works, however, I don't get any logging from the JAR process. I've tried calling it this way:
proc = subprocess.call(jar_command)
This logs as I would expect, but the two shell scripts that follow are not executed. Initially I thought the logs just weren't going to stdout but it turns out they're not being executed at all. I.E. not removing superfluous files or pushing to a database.
Why are the followup shell scripts being ignored?
If you are certain your shell scripts are not running at all, and with the first code everything works - then it must be the java command deadlocks or not terminates correctly using the call() function.
You can validate that by adding a dummy file creation in your bash scripts. Put it in the first line of the script, so if it is executed you'll get the dummy file created. If it's not created, that means the scripts weren't executed, probably due to something with the java execution.
I would have try couple things:
First I would return the Popen instead of call. Instead of using wait(), use communicate():
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
communicate() returns a tuple (stdoutdata, stderrdata).
proc = subprocess.Popen(jar_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.communicate()
Make sure to check both streams for data (stdout and stderr). You might miss an error the java process raises.
Next I would try disabling the buffer by providing bufsize=0 to Popen. It will eliminate the option it relates to python buffering.
If both options still don't work, try to see if there is an exception by using check_call():
proc = subprocess.check_call(jar_command)
Run command with arguments. Wait for command to complete. If the return code was zero then return, otherwise raise CalledProcessError.
These options might have the answer; if not, they would help the debugging process. Feel free to comment how this progress.
Most likely, you are forgetting that the processes streams are in fact OS-level buffers with some finite capacity.
For example, if you run a process that produces a lot of output in PIPE mode, and you wait for it to finish before trying to consume whatever that process wrote to output, you have a deadlock:
The process has filled up the output buffer and is now blocked on writing more data to its output. Until somebody empties the buffer by reading from pipe, the process cannot continue.
Your program is waiting for the subprocess to finish before you read the data from its buffer.
The correct way is to start a thread in your program that will "drain" the pipe constantly as the process is running and while your main thread is waiting. You must first start the process, then start the drain threads, then wait for process to finish.
For differential diagnosis, check whether the subprocess will run fine with little output (i.e. as long as the buffer does not fill up, such as a line or two).
The documentation for subprocess has a note about this.
I know that the process will be blocked if its error and input streams are not completely read.
Does this mean that the process is finished (all his commands were executed) when its streams have been completely read (no data in the stream, the end of the stream has been reached)?
Is it possible that the streams are terminated but the process not?
How can I be sure that the process is complete?
I could not find information about this in the documentation.
Another Question:
As mentioned below, it is possible that the streams are terminated but the process not. In this case: could these kinds of processes contain any commands or actions to do or are they only processes without actions or anything to do? And the second question: how can i terminate a process after a certain timeout although Process.waitFor() is used?
Use Process.waitFor() method to wait until process completion
Does this mean that the process is finished (all his commands were executed) when its streams have been completely read (no data in the stream, the end of the stream has been reached)?
No, it does not. Conversely:
is it possible that the streams are terminated but the process not?
Yes, this is perfectly possible.
How can I be sure that the process is complete? I could not find information about this in the documentation.
Using Process.waitFor().
i'm trying to execute 2 commands via java programme with process
Process p = Runtime.getRuntime().exec(command1);
Process p2 = Runtime.getRuntime().exec(command2);
the problem is that the first one is ok but the seconde on cant be established
it is always bloqed in waitfor()
You might be running into the dreaded "need to empty the streams" problem. See When Runtime.exec() won't for details on it.
Also in the same article is some info on other traps you can run into if you're treating getRuntime().exec() like the command line.
When running an external procss that prints anything to stdout/stderr, you should read what it writes - otherwise it will block once it's buffer fills up.
you basically needs a thread to read from stdout and a thread to read from stderr of each process.
I have a little executable program in C that produce a lot of output to a file.
When I call this program with Runtime, like this:
Runtime r = Runtime.getRuntime();
Process p = null;
p = r.exec("./my_program -in input.file -out output.file", null, new File(System.getProperty("java.io.tmpdir")));
When the program produce low output everything is ok, but when I call "*my_program*" with a large input it will produce a large quantity of output to the output.file, but in this case my program in Java freeze and nothing happen...
I test "*my_program*" in terminal with a lot of large inputs and everything is ok, but when I call the program in Java with Runtime.exec, the Java program freeze.
--
Thanks in advance
Make sure you're reading from the Process's .getOutputStream() and .getErrorStream() if you aren't already. Looking at your code snippet, it appears that you're just executing .exec(...) (and maybe waiting for it to complete with a call not shown to .waitFor()?).
Per http://download.oracle.com/javase/6/docs/api/java/lang/Process.html (emphasis added):
The parent process uses these streams to feed input to and get output
from the subprocess. Because some native platforms only provide
limited buffer size for standard input and output streams, failure to
promptly write the input stream or read the output stream of the
subprocess may cause the subprocess to block, and even deadlock.
Is there a thread-safe way to concurrently consume the stdout from an external process, using ProcessBuilder in Java 1.6?
Background: I need to invoke pbzip2 to unzip large files to stdout and to process each line as the file is decompressed (pbzip2 utilizes multiple CPUs, unlike other implementations).
The logical approach is to create a child thread to loop over the InputStream (i.e. stdout; don't you just love the naming?), as follows:
while((line = reader.readLine()) != null)
{
// do stuff
}
However, unzipping is slow, so what I really need is for the reader.readLine method to quietly wait for the next line(s) to become available, instead of exiting.
Is there a good way to do this?
You should be able to wrap your input stream with an InputStreamReader and BufferedReader. You can then call readLine() and that will block as required.
Note that you should have a corresponding reader for the stderr. You don't have to do anything with it, but you will need to consume the stderr stream, otherwise your spawned process may well block. See this answer for links etc.
You more or less have the solution yourself. You just create a new thread which reads the next line in a loop from the stream of your external process and processes that line.
readLine() will block and wait until an entire new line is available. If you're on a multicore/processor machine, your external process can happily continue unzipping while your thread processes a line. Atleast unzipping can continue until the OS pipes/buffers becomes full.
Just note that if your processing is slower than unzipping, you'll block the unzipping, and at this point it becomes a memory vs speed issue. e.g. you could create one thread that does nothing but read lines(so unzipping will not block), buffer them up in a queue in memory and a another thread - or even several, that consumes said queue.
readLine method to quietly wait for
the next line(s) to become available,
instead of exiting
nd that's exactly what readLine should do, it will just block until a whole line is available.
Yes.
I have written some code that kicks off a time consuming job (ffmpeg) in a Process (spawned by process builder), and it in turn kicks off my OutputStreamReaderclass that is an extention of Thread that consumes the stdio and does some magic with it.
The catch (for me) was redirecting the error stream. Here is my code snippet:
procbbuilder.redirectErrorStream(true);
proc = pb.start();
err = new MyOutputStreamReader(this, proc.getInputStream()); //extenion of thread
err.start();
int exitCode = proc.waitFor();