This is in the context of a local Processing program. I would like to run an external program to get some data. Is there a popen() or equivalent function I can use?
Process process = Runtime.getRuntime().exec("your command");
Then you can read and write the data using the Process streams.
JDK5 introduced ProcessBuilder for more control over the process generation.
Process process = new ProcessBuilder(command).start()
Be aware of the fact, that internally forkAndExec is invoked, and fork 'makes a copy of the entire parents address space', so that even a little command can lead to OutOfMemoryErrors, when the parent process has big amount of memory space acquired.
see here
A close friend of popen() is to make a named pipe as input and/or output, like in UNIX:
mknod /tmp/mypipe.12345 p ; sort -o /tmp/mypipe.12345 /tmp/mypipe.12345 &
Then open /tmp/mypipe.12345, write, close, open /tmp/mypipe.12345, read, close. Since a sort cannot write anything until EOF on input, the output open will occur after the input close. The popen() call cannot do this!
For simpler scenarios, the named pipe can just be read or written.
Of course, you still need to spin it off, as in a system(...) call.
You want to remove the named pipe when you are done. On some UNIX systems, /tmp is cleared upon reboot.
/tmp is shared so name collisions are quite possible. You can generate a partly random pipe file name (numeric part of /tmp/mypipe.12345) in Java to generally prevent this. In some systems, Bash creates named pipes in /var/tmp for every <(...) or >(...) use. Unfortunately, it is a bit of a challenge to determine when they can be removed without effect (fuser?)!
Related
I know how to "deamonize" a process (not to confused with Thread.setDaemon). There are some answers here and here and I'm using my own perl wrapper, which works fine.
But what I'd like to get now, is the parent process waiting for the Java process until it says "OK", i.e., until it has really started successfully (not only process started, but everything up and running well).
I could indicate this by writing a file, binding to a socket or alike, but it's ugly. Out of the eight items on the deamonize list, I only need the following three in a slightly simplified form:
Close standard input, standard output and standard error.
Run in the background (i.e., fork)
Ignore SIGHUP.
The first and last item can be done before process start, so only forking the process remains. Googling for "Java fork" is hopeless since the ForkJoinPool exists. Before I get my hands dirty, I'd like to know if
it's supported in Java 9 (then I'd simply wait)
someone did it already using JNA and what problems they ran in
there's a better solution
I don't care about Windows as it's for a Linux server.
I don't know how well this translates to Java, but from a syscall perspective:
When both the parent and child are under your full control, you should call pipe or socketpair to create your own communication channel, and specify it to the child via environment variables or command-line arguments. Remember to immediately close one half in each process (either between fork and exec if you have control there, or via the CLOEXEC flag - this means the child executable never has to know about the parent's end at all).
I'm not sure why you seem hesitant to use sockets - perhaps you're under the impression that local sockets take up ports (they don't) - though if all the data travels in one direction I'd prefer pipes just for clarity.
If it is possible for the child to create its own children, you should set the CLOEXEC flag on the inherited pipe as soon as possible.
You must send a positive message to indicate success; closing the pipe early must be considered an error (though you may also have explicit errors). Note that this means you don't have to track the exit value (useful if the "child" is actually a grandchild).
Alternatively, perhaps you should make something else do the work: your init system, X11, and DBUS? E.g., what were you thinking you should do if the child crashes?
Ok, So i am trying to read the output of a c binary from java code and I am unable to figure out whether the communication channel is blocking or non blocking.
The setup is such:
A java class (A.java) is run
A.java runs a c binary (B.o) using Runtime.getRuntime().exec("B.o"). At this point I have the Process object (returned by Runtime.exec)
A.java reads from the input stream of the Process object using a bufferedreader
A.java outputs the data read from the input stream to a file (output.txt)
The B.o binary simply prints random lines using printf function call.
Now, if I run the above setup, I receive all the data sent by B.o flawlessly. Then to test (the blocking / nonblocking thing), I changed the A.java to sleep for 5 milliseconds after every read from the inputstream of the Process object of B.o. As it turned out, now I wasn't receiving the complete data in A.java send by B.o. This indicates that the communication channel being used is non-blocking (as per my weak understanding).
Then just to make sure, I started looking at the source code of java to see if I was right. And I have found the following so far:
Every call to Runtime.getRuntime().exec(...) ends up in forkAndExec() method in ProcessImpl_md.c. In ProcessImpl_md.c the command is executed, a process is created, and PIPES are setup for communication (using the pipe function call in c). I can't find anywhere in the source code where the PIPES are being set to nonblocking mode (as indicated by my code). I am assuming the PIPES are blocking by default.
I know this is a very bad way to check what I want to check. I am way out of my depth here and I am just head-banging uselessly, I think.
Can anyone point me in the right direction or tell me:
Are the PIPES of a process created through java runtime API are blocking or non-blocking?
When I make A.java sleep after reading from the input stream, why all data is not received? (Assumption being that the PIPE is blocking)
Any non-programmatic way (i.e. I don't have to change the source code of java and etc!) to figure out if the PIPES of a process are blocking or non-blocking?
Thank you.
EDIT: (added code)
Following is not the actual (or even compilable) code but it shows what i am trying to do.
Source of "B.o":
#include <stdio.h>
void main(int argc, char*argv[]){
int a = 0;
for(; a<9000000; a++){
printf("%s", argv[1]);
}
}
Source of "A.java":
<java imports>
public class A{
public static void main(String[] args) throws Exception{
Process p = Runtime.getRuntime().exec("./B.o");
BufferedReader br = new
BufferedReader(new InputStreamReader(p.getInputStream()));
int a = 0;
while(br.readLine() != null){
a++;
Thread.sleep(5);//data missed if this line not commented out
}
br.close();
System.out.println(a);
}
}
PLEASE CHECK MY ANSWER. USELESS QUESTION BY ME.
Whether the communication channels between Java and the external program (there are three, one from Java to native, and two coming back) are operating in blocking or non-blocking mode is not directly relevant to whether all data will be successfully transmitted across each. Likewise, delays between read requests are not directly relevant to whether all data will be successfully transmitted, regardless of blocking vs. non-blocking I/O in your particular implementation of java.lang.Process.
Really, your efforts to probe blocking vs. non-blocking inter-process I/O are futile, because the I/O interface provided to your Java program is based on InputStream and OutputStream, which provide only for blocking I/O. Even if non-blocking I/O were involved at some low level of the implementation, I can't think of any way for your program to detect that.
With respect to your specific questions, however:
Are the PIPES of a process created through java runtime API are blocking or non-blocking?
They could be either, but they are more likely blocking because that better matches the interface presented to the Process user.
When I make A.java sleep after reading from the input stream, why all data is not received? (Assumption being that the PIPE is blocking)
I can only speculate, but the problem is likely in the external program. Possibly it goes to sleep when its output buffer fills, and nothing happens to wake it up. It might help to invoke myProcess.getOutputStream().close() if your Java program is not sending data to the external program. It's in any case a good idea to close that stream once you've written to it everything you're ever going to write.
Any non-programmatic way (i.e. I don't have to change the source code of java and etc!) to figure out if the PIPES of a process are blocking or non-blocking?
Potentially you could run the VM under strace or connect a native debugger to it, and analyze the VM's behavior that way. If you mean to do this from inside Java then the answer is a resounding "NO". Your Java program will see blocking behavior under all circumstances because the contracts of InputStream and OutputStream demand it.
I was making a big blunder and was completely off base. Posting this answer to clear things up (though I would like to delete the question altogether). I wanted to know if the communication channels between a C binary run from Java code are blocking or non-blocking. And I mentioned that the data was missing when I made my java code sleep after reading from the input stream of the created process (of C code). The data wasn't missing because of that. I had actually put a timer in Java code after which to terminate the process of the C binary. And since the PIPES are blocking, it wasn't able to receive all the data before the timer expired. I was misinterpreting this loss of data to mean that the PIPES were non-blocking. Confirmed this by running STRACE on the created C binary process. There were no EAGAIN errors on the write syscalls. My bad. But thank you very much to all for taking out the time to respond.
I am monitoring and Minecraft server and I am making a setup file in Python. I need to be able to run two threads, one running the minecraft_server.jar in the console window, while a second thread is constantly checking the output of the minecraft_server. Also, how would I input into the console from Python after starting the Java process?
Example:
thread1 = threading.Thread(target=listener)
thread2 = minecraft_server.jar
def listener():
if minecraft_server.jarOutput == "Server can't keep up!":
sendToTheJavaProccessAsUserInputSomeCommandsToRestartTheServer
It's pretty hard to tell here, but I think what you're asking is how to:
Launch a program in the background.
Send it input, as if it came from a user on the console.
Read its output that it tries to display to a user on the console.
At the same time, run another thread that does other stuff.
The last one is pretty easy; in fact, you've mostly written it, you just need to add a thread1.start() somewhere.
The subprocess module lets you launch a program and control its input and output. It's easiest if you want to just feed in all the input at once, wait until it's done, then process all the output, but obviously that's not your case here, so it's a bit more involved:
minecraft = subprocess.Popen(['java', 'path/to/minecraft_server.jar', '-other', 'args],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
I'm merging stdout and stderr together into one pipe; if you want to read them separately, or send stderr to /dev/null, or whatever, see the docs; it's all pretty simple. While we're making assumptions here, I'm going to assume that minecraft_server uses a simple line-based protocol, where every command, every response, and every info message is exactly one line (that is, under 1K of text ending in a \n).
Now, to send it input, you just do this:
minecraft.stdin.write('Make me a sandwich\n')
Or, in Python 3.x:
minecraft.stdin.write(b'Make me a sandwich\n')
To read its output, you do this:
response = minecraft.stdout.readline()
That works just like a regular file. But note that it works like a binary file. In Python 2.x, the only difference is that newlines don't get automatically converted, but in Python 3.x, it means you can only write bytes (and compatible objects), not strs, and you will receive bytes back. There are good reasons for that, but if you want to get pipes that act like text files instead, see the universal_newlines (and possibly bufsize) arguments under Frequently Used Arguments and Popen Constructor.
Also, it works like a blocking file. With a regular file, this rarely matters, but with a pipe, it's quite possible that there will be data later, but there isn't data yet (because the server hasn't written it yet). So, if there is no output yet (or not a complete line's worth, since I used readline()), your thread just blocks, waiting until there is.
If you don't want that, you probably want to create another thread to service stdout. And its function can actually look pretty similar to what you've got:
def listener():
for line in minecraft.stdout:
if line.strip() == "Server can't keep up!":
minecraft.stdin.write("Restart Universe\n")
Now that thread can block all day and there's no problem, because your other threads are still going.
Well, not quite no problem.
First it's going to be hard to cleanly shut down your program.
More seriously, the pipes between processes have a fixed size; if you don't service stdout fast enough, or the child doesn't service stdin fast enough, the pipe can block. And, the way I've written things, if the stdin pipe blocks, we'll be blocked forever in that stdin.write and won't get to the next read off stdout, so that can block too, and suddenly we're both waiting on each other forever.
You can solve this by having another thread to service stdout. The subprocess module itself includes an example, in the Popen._communicate function used by all the higher-level functions. (Make sure to look at Python 3.3 or later, because earlier versions had bugs.)
If you're in Python 3.4+ (or 3.3 with a backport off PyPI), you can instead use asyncio to rewrite your program around an event loop and handle the input and output the same way you'd write a reactor-based network server. That's what all the cool kids are doing in 2017, but back in late 2014 many people still thought it looked new and scary.
If all of this is sounding like a lot more work than you signed on for, you may want to consider using pexpect, which wraps up a lot of the tedious details, and makes some simplifying assumptions that are probably true in your case.
I am launching a process from java to run a command for me. This process runs for a little while, then needs a password to continue. Now I know that I can write to the in stream of the proces, but I am not quite sure how to detect when I need to write to it.
Possible solutions:
Is there a way that I can detect that the process is blocking?
Can I just write to the standard in immediately after executing the command and when the process hits a point when it needs it, it can just read from it?
Any other ideas?
It is not necessary to detect if the child process is blocking or not. If the child process is designed to block until input is provided to it via stdin, it will block until such input is provided.
It it necessary to keep in mind that the standard input, output and error buffer sizes are limited, and therefore it would be necessary for the child process to process the contents of the input buffer, and for the parent process to process the contents of the output and error buffers as soon as possible. Not doing so will result in the child process hanging.
Maybe you should get around the runas problem but not using runas. Google found me this: http://www.source-code.biz/snippets/c/1.htm Lets you pass your password at runtime....
So I have a problem with a process that I am running, whenever I try to stop it using process.destroy(), it does not stop.
I want to create a file (ProcessHandler) that extends Process and do the following:
ProcessHandler process = (ProcessHandler)Runtime.getRuntime().exec("cmd.exe /c \"java net/com/codeusa/Server 43594\"");
So, my problem is trying to convert Process to ProcessHandler where I can override the destroy() command, to make it TSKILL itself. I have figured out how to do everything but when I try the above like of code, I get a ClassCastException..
Anyone have an idea how I can make these be compatible. BTW the exec(String) command returns an instance of Process.
Firstly, if you are using Java 5 or later, I would recommend using ProcessBuilder instead of Runtime.getRuntime().exec(). For one thing, you don't have to worry about quoting arguments. Each separate command-line argument is a separate parameter. For example:
ProcessBuilder builder = new ProcessBuilder("cmd.exe", "/C", "java net/com/codeusa/Server 43594");
Process process = builder.start();
When starting a process using ProcessBuilder or Runtime.getRuntime().exec(), it's entirely up to the JVM to instantiate and return a subclass of Process of its choice, and there's no way to influence its decision. I assume your ProcessHandler class is one you've written yourself (I can't find a Java API class with that name). It might subclass Process, but even if it does there's no way for the JVM to return an instance of it when you use ProcessBuilder or Runtime.getRuntime().exec(). So your line of code above is guaranteed to throw a ClassCastException, assuming it doesn't throw some other exception.
I have had some experience in the past of processes that didn't respond to destroy() methods. Usually this was because the standard output or standard error being written by the process wasn't being read, and the process had ground to a halt because one or more of its I/O buffers had filled up. Does the process above write anything to its standard output or standard error, and if so, are you reading it?
Reading both the standard output and standard error streams is easier with ProcessBuilder: if you add the line builder.redirectErrorStream(true); between the two lines above, then you only need to read from the process's standard output. If you're stuck with Java 1.4 or earlier and Runtime.getRuntime().exec(), you'll have to set up two different objects in two different threads, one that reads from each stream.
I'm not sure what you are trying to achieve with your ProcessHandler class - you haven't provided the source code for it. Besides, I've never had the need to kill a process more forcibly than by using the destroy() method.
I figured out a whole new thing!! When I call the destroy() method for process, it destroys the cmd.exe process.. but I replaced cmd.exe with "java" and now when I call destroy(), the java.exe process terminates.. HURAY