Develop Reference

Strange execution patterns with subprocess.Popen

I have a Python script wherein a JAR is called. After the JAR is called, two shell scripts are called. Initially I was doing this:
proc = subprocess.Popen(jar_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.wait()
output, errors = proc.communicate()
proc = subprocess.Popen(prune_command, shell=True)
proc.wait()
proc = subprocess.call(push_command, shell=True)
I have to wait for the first two processes to finish so I use Popen() and the final one I can let it run in the background, so I call() it. I pass shell=True because I want the called shell scripts to have access to environment variables.
The above works, however, I don't get any logging from the JAR process. I've tried calling it this way:
proc = subprocess.call(jar_command)
This logs as I would expect, but the two shell scripts that follow are not executed. Initially I thought the logs just weren't going to stdout but it turns out they're not being executed at all. I.E. not removing superfluous files or pushing to a database.
Why are the followup shell scripts being ignored?

If you are certain your shell scripts are not running at all, and with the first code everything works - then it must be the java command deadlocks or not terminates correctly using the call() function.
You can validate that by adding a dummy file creation in your bash scripts. Put it in the first line of the script, so if it is executed you'll get the dummy file created. If it's not created, that means the scripts weren't executed, probably due to something with the java execution.
I would have try couple things:
First I would return the Popen instead of call. Instead of using wait(), use communicate():
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
communicate() returns a tuple (stdoutdata, stderrdata).
proc = subprocess.Popen(jar_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.communicate()
Make sure to check both streams for data (stdout and stderr). You might miss an error the java process raises.
Next I would try disabling the buffer by providing bufsize=0 to Popen. It will eliminate the option it relates to python buffering.
If both options still don't work, try to see if there is an exception by using check_call():
proc = subprocess.check_call(jar_command)
Run command with arguments. Wait for command to complete. If the return code was zero then return, otherwise raise CalledProcessError.
These options might have the answer; if not, they would help the debugging process. Feel free to comment how this progress.

Most likely, you are forgetting that the processes streams are in fact OS-level buffers with some finite capacity.
For example, if you run a process that produces a lot of output in PIPE mode, and you wait for it to finish before trying to consume whatever that process wrote to output, you have a deadlock:
The process has filled up the output buffer and is now blocked on writing more data to its output. Until somebody empties the buffer by reading from pipe, the process cannot continue.
Your program is waiting for the subprocess to finish before you read the data from its buffer.
The correct way is to start a thread in your program that will "drain" the pipe constantly as the process is running and while your main thread is waiting. You must first start the process, then start the drain threads, then wait for process to finish.
For differential diagnosis, check whether the subprocess will run fine with little output (i.e. as long as the buffer does not fill up, such as a line or two).
The documentation for subprocess has a note about this.

Java Command Prompt Emulator

I'm trying to make a java program that commands through cmd.exe and prints their output. To do this, I'm using this code:
cmdLine = Runtime.getRuntime().exec("cmd.exe");
cmdLineOut = new BufferedReader(new InputStreamReader(cmdLine.getInputStream()));
cmdLineIn = new PrintWriter(cmdLine.getOutputStream());
// ...
cmdLineIn.println(command);
cmdLineIn.flush();
String s = null;
while ((s = cmdLineOut.readLine()) != null)
System.out.println(s);
Although, when input is given, the output is never printed.
EDIT: Solved
The cmdLineOut.readLine() doesn't return null when the input is empty, it freezes. Since readLine freezes at the end no other code is executed, I just put the printing of the readLine in a seperate thread.
If somebody wants to answer this better, go ahead.

You never actually execute the user's command, at least in the snippet you posted. Also, nearly all command prompt "commands" are actually just programs that are on the default program search path. You should probably just Runtime.getRuntime().exec(user_command) for each command. This means that you will have to set up the input and output streams like you have already done for each command. You are right to get input in a separate thread, since attempting to read input will block the current thread until there is actually input to read.
However, some commands, even under UNIX or Linux systems, are "built-in" (like cd), meaning that the command prompt (aka "shell") handles the commands internally. Your program will have to test the user input to see if they are calling a built-in, and specially handle calls to built-in commands. Your program should actually be portable to non-Windows computers. Of course, the user would use different commands (cp instead ofcopy), but the only part you would have to add would be handling for other OS' shells' lists of built-ins (or simply have your program implement a "cross-platform" set of built-ins - this is your shell program, you get to make the rules).

Executed C binary from java and reading from output stream of the process

Ok, So i am trying to read the output of a c binary from java code and I am unable to figure out whether the communication channel is blocking or non blocking.
The setup is such:
A java class (A.java) is run
A.java runs a c binary (B.o) using Runtime.getRuntime().exec("B.o"). At this point I have the Process object (returned by Runtime.exec)
A.java reads from the input stream of the Process object using a bufferedreader
A.java outputs the data read from the input stream to a file (output.txt)
The B.o binary simply prints random lines using printf function call.
Now, if I run the above setup, I receive all the data sent by B.o flawlessly. Then to test (the blocking / nonblocking thing), I changed the A.java to sleep for 5 milliseconds after every read from the inputstream of the Process object of B.o. As it turned out, now I wasn't receiving the complete data in A.java send by B.o. This indicates that the communication channel being used is non-blocking (as per my weak understanding).
Then just to make sure, I started looking at the source code of java to see if I was right. And I have found the following so far:
Every call to Runtime.getRuntime().exec(...) ends up in forkAndExec() method in ProcessImpl_md.c. In ProcessImpl_md.c the command is executed, a process is created, and PIPES are setup for communication (using the pipe function call in c). I can't find anywhere in the source code where the PIPES are being set to nonblocking mode (as indicated by my code). I am assuming the PIPES are blocking by default.
I know this is a very bad way to check what I want to check. I am way out of my depth here and I am just head-banging uselessly, I think.
Can anyone point me in the right direction or tell me:
Are the PIPES of a process created through java runtime API are blocking or non-blocking?
When I make A.java sleep after reading from the input stream, why all data is not received? (Assumption being that the PIPE is blocking)
Any non-programmatic way (i.e. I don't have to change the source code of java and etc!) to figure out if the PIPES of a process are blocking or non-blocking?
Thank you.
EDIT: (added code)
Following is not the actual (or even compilable) code but it shows what i am trying to do.
Source of "B.o":
#include <stdio.h>
void main(int argc, char*argv[]){
int a = 0;
for(; a<9000000; a++){
printf("%s", argv[1]);
}
}
Source of "A.java":
<java imports>
public class A{
public static void main(String[] args) throws Exception{
Process p = Runtime.getRuntime().exec("./B.o");
BufferedReader br = new
BufferedReader(new InputStreamReader(p.getInputStream()));
int a = 0;
while(br.readLine() != null){
a++;
Thread.sleep(5);//data missed if this line not commented out
}
br.close();
System.out.println(a);
}
}
PLEASE CHECK MY ANSWER. USELESS QUESTION BY ME.

Whether the communication channels between Java and the external program (there are three, one from Java to native, and two coming back) are operating in blocking or non-blocking mode is not directly relevant to whether all data will be successfully transmitted across each. Likewise, delays between read requests are not directly relevant to whether all data will be successfully transmitted, regardless of blocking vs. non-blocking I/O in your particular implementation of java.lang.Process.
Really, your efforts to probe blocking vs. non-blocking inter-process I/O are futile, because the I/O interface provided to your Java program is based on InputStream and OutputStream, which provide only for blocking I/O. Even if non-blocking I/O were involved at some low level of the implementation, I can't think of any way for your program to detect that.
With respect to your specific questions, however:
Are the PIPES of a process created through java runtime API are blocking or non-blocking?
They could be either, but they are more likely blocking because that better matches the interface presented to the Process user.
When I make A.java sleep after reading from the input stream, why all data is not received? (Assumption being that the PIPE is blocking)
I can only speculate, but the problem is likely in the external program. Possibly it goes to sleep when its output buffer fills, and nothing happens to wake it up. It might help to invoke myProcess.getOutputStream().close() if your Java program is not sending data to the external program. It's in any case a good idea to close that stream once you've written to it everything you're ever going to write.
Any non-programmatic way (i.e. I don't have to change the source code of java and etc!) to figure out if the PIPES of a process are blocking or non-blocking?
Potentially you could run the VM under strace or connect a native debugger to it, and analyze the VM's behavior that way. If you mean to do this from inside Java then the answer is a resounding "NO". Your Java program will see blocking behavior under all circumstances because the contracts of InputStream and OutputStream demand it.

I was making a big blunder and was completely off base. Posting this answer to clear things up (though I would like to delete the question altogether). I wanted to know if the communication channels between a C binary run from Java code are blocking or non-blocking. And I mentioned that the data was missing when I made my java code sleep after reading from the input stream of the created process (of C code). The data wasn't missing because of that. I had actually put a timer in Java code after which to terminate the process of the C binary. And since the PIPES are blocking, it wasn't able to receive all the data before the timer expired. I was misinterpreting this loss of data to mean that the PIPES were non-blocking. Confirmed this by running STRACE on the created C binary process. There were no EAGAIN errors on the write syscalls. My bad. But thank you very much to all for taking out the time to respond.

UNIX STDOUT end symbol

I want to execute multiple commands from Java Process but I don't want to spawn a new process for executing every command. So I made an Object called Shell that holds InputStream and OutputStream for Process.
The problem is that if I don't terminate a process by appending
"exit\n"
I can't tell where is the end of the InputStream and the InputStream gets into waiting state when I've read the whole output so I need to know when to stop doing next read.
Is there some kind of a standard symbol at the end of the output?
Because what I came up with is
final String outputTerminationSignal = checksum(command);
command += ";echo \"" + outputTerminationSignal + "\";echo $?\n"
This way when I get the outputTerminationSignal line I can get the exit code and stop reading.
final String line = bufferedReader.readLine();
if (line != null && line.equals(outputTerminationSignal)) {
final String exitCode = bufferedReader.readLine();
}
Of course this is exploitable and error-prone because the real output in some case may match my generated outputTerminationSignal and the app will stop reading when it shouldn't.
I wonder if there is some standard so called "outputTerminationSignal" comming from the output I am not aware of.

Unix doesn't use a special character or symbol to indicate the end of a stream. In java, if you try to read from a stream that's at end-of-file, then you'll get an EOFException.
Having said that, if you're reading from a stream connected to a running program, then you won't get an EOFException just because the other program is idle. You would only get an EOFException if the other program has exited, or if it explicitly closes its output stream (that you are reading from). The situation you describe sounds like the shell is just idle waiting for another command. You won't get an EOF indication from the stream in this case.
You could try getting the shell to print a command prompt when it's waiting for a command, then look for the command prompt as an "end of command" indicator. Shells normally print command prompts only when they're interactive, but you might be able to find a way around that.
If you want to make the shell process exit without sending it the "exit" command, you could try closing the stream that you're using to write to the shell process. The shell should see that as an end-of-file and exit.

You could ask the shell for the PID of the spawned child, and monitor its state

How to Send a Password to Process in Java

I am launching a process from java to run a command for me. This process runs for a little while, then needs a password to continue. Now I know that I can write to the in stream of the proces, but I am not quite sure how to detect when I need to write to it.
Possible solutions:
Is there a way that I can detect that the process is blocking?
Can I just write to the standard in immediately after executing the command and when the process hits a point when it needs it, it can just read from it?
Any other ideas?

It is not necessary to detect if the child process is blocking or not. If the child process is designed to block until input is provided to it via stdin, it will block until such input is provided.
It it necessary to keep in mind that the standard input, output and error buffer sizes are limited, and therefore it would be necessary for the child process to process the contents of the input buffer, and for the parent process to process the contents of the output and error buffers as soon as possible. Not doing so will result in the child process hanging.

Maybe you should get around the runas problem but not using runas. Google found me this: http://www.source-code.biz/snippets/c/1.htm Lets you pass your password at runtime....