I am using Java Process API to write a class that receives binary input from the network (say via TCP port A), processes it and writes binary output to the network (say via TCP port B). I am using Windows XP. The code looks like this. There are two functions called run() and receive(): run is called once at the start, while receive is called whenever there is a new input received via the network. Run and receive are called from different threads.
The run process starts an exe and receives the input and output stream of the exe. Run also starts a new thread to write output from the exe on to the port B.
public void run() {
try {
Process prc = // some exe is `start`ed using ProcessBuilder
OutputStream procStdIn = new BufferedOutputStream(prc.getOutputStream());
InputStream procStdOut = new BufferedInputStream(prc.getInputStream());
Thread t = new Thread(new ProcStdOutputToPort(procStdOut));
t.start();
prc.waitFor();
t.join();
procStdIn.close();
procStdOut.close();
} catch (Exception e) {
e.printStackTrace();
printError("Error : " + e.getMessage());
}
}
The receive forwards the received input from the port A to the exe.
public void receive(byte[] b) throws Exception {
procStdIn.write(b);
}
class ProcStdOutputToPort implements Runnable {
private BufferedInputStream bis;
public ProcStdOutputToPort(BufferedInputStream bis) {
this.bis = bis;
}
public void run() {
try {
int bytesRead;
int bufLen = 1024;
byte[] buffer = new byte[bufLen];
while ((bytesRead = bis.read(buffer)) != -1) {
// write output to the network
}
} catch (IOException ex) {
Logger.getLogger().log(Level.SEVERE, null, ex);
}
}
}
The problem is that I am getting the following stack inside receive() and the prc.waitfor() returns immediately afterwards. The line number shows that the stack is while writing to the exe.
The pipe has been ended
java.io.IOException: The pipe has been ended
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at xxx.receive(xxx.java:86)
Any advice about this will be appreciated.
This means you are writing to the pipe after the other end has already closed it.
That indicates a major error in your application protocol.
I have had the same problem recently and I have found a solution.
First of all, "The pipe has been ended" error is not a Java error - it comes from Windows system. According to MSDN:
The using process has closed the pipe or, if you are trying to write
to the pipe, there are no available readers.
Not very informative. However, if process has closed the pipe itself, it may mean that some errors occurred in process.
To check this, redirect errors coming from process, for instance, to a file:
File f = new File("errors.txt");
pb.redirectError(f);
In my case (I've been trying to execute SrcML parser) file contained this:
.\libs\srcML-Win\src2srcml.exe: unrecognised option `--language Java'
Try 'src2srcml --help' for more information.
Fixing this solved the problem.
Related
Background
I am having a simple java program which reads data from std input. I am executing it though from a bash script executer.sh
python2 readLines.py | tee logfile | java MsgReader
readLines.py reads data from a file line by line and throws it on the stdout
MsgReader.java
import kafka.javaapi.producer.Producer;
public void process() {
String msg = "";
BufferedReader stdinReader = new BufferedReader(new InputStreamReader(System.in)
Producer producer = new Producer();//it takes config file as parameter which is not related to the question here
try {
while ((msg = stdinReader.readLine()) != null) {
producer.send(message);
}
stdinReader.close();
} catch (IOException|FailedToSendMessageException e) {
System.out.println("Send message failed!");
e.printStackTrace();
}
}
public static void main(String args[]) throws JSONException,IOException {
try{
Date now = new Date();
System.out.println("Start : " + now);
process();// Continuously reads logs
now = new Date();
System.out.println("After : " + now);
} catch(Exception e){
e.printStackTrace();
return;
}
}
Execution
./executer.sh & : executing in the background
Problem
After reading all the lines readLines.py ends, but executer.sh and MsgReader.java are stll executing as verified by ps command. The statement System.out.println("After : " + now); os recorded in the log file, indicating the program has reached the last statement in the main function.
How to terminate the java program and bash script gracefully.
I do not want to add System.exit() here.
Platform
CentOS release 6.7 (Final)
java 1.7.0_95
python 2.7.6
You did not close the Producer. From the Javadoc for KafkaProducer (the implementation of Producer):
The producer consists of a pool of buffer space that holds records that haven't yet been transmitted to the server as well as a background I/O thread that is responsible for turning these records into requests and transmitting them to the cluster. Failure to close the producer after use will leak these resources.
The background thread is still running and will prevent the process from ending until you close the Producer, preferably in a finally block after the try/catch. Also, that stdinReader.close(); belongs in the finally as well.
I am trying to port a C program to android using JNI. I have been able to set up the program and get java and c working fine together. The problem is I need to be able to use STDIN since the C program reads input from STDIN and returns a response through STDOUT (C program is a server-client application).
I don't know if it is worth mentioning but the C program uses STDIN_FILENO file descriptor for reading input from STDIN.
How do I read from STDOUT and WRITE to STDIN using Java?
I did some research and found some vague explanation at the following link: https://groups.google.com/forum/#!topic/android-ndk/Brm6jPr4C0Y that I don't understand.
Here's is the C code https://github.com/unekwu/kemi/blob/master/jni/dnscat/dnscat.c#L1270
More Details
The C program is usually run from command line like
dnscat --dns <ip> <port>
After which it starts listening for messages from the user. Normally entered from stdin.
Now in my Android app,
I'm able to run it with JNI by calling main and different name and passing ann array of strings to it. I'v verify the program starts up correcting. The problem is how I'll send the messages to the program since there's no stdin on android.
I've created a project in github, which you can download from here.
It creates 2 named pipes (FIFO), one for input and other for output.
It opens up one end of the pipe in write only mode in native code and other end of the pipe in read only mode in Java code. The file descriptor in native code is mapped to STDOUT i.e. 1, thereafter any writes to STDOUT in native code, will be redirected to other end of pipe which can read in Java code.
It opens up one end of the pipe in read only mode in native code and other end of the pipe in write only mode in Java code. The file descriptor in native code is mapped to STDIN i.e. 0, thereafter any writes to other end of pipe in Java code, will be read by native code using STDIN.
To achieve STDOUT redirection:
Native Code:
/*
* Step 1: Make a named pipe
* Step 2: Open the pipe in Write only mode. Java code will open it in Read only mode.
* Step 3: Make STDOUT i.e. 1, a duplicate of opened pipe file descriptor.
* Step 4: Any writes from now on to STDOUT will be redirected to the the pipe and can be read by Java code.
*/
int out = mkfifo(outfile, 0664);
int fdo = open(outfile, O_WRONLY);
dup2(fdo, 1);
setbuf(stdout, NULL);
fprintf(stdout, "This string will be written to %s", outfile);
fprintf(stdout, "\n");
fflush(stdout);
close(fdo);
Java Code:
/*
* This thread is used for reading content which will be written by native code using STDOUT.
*/
new Thread(new Runnable() {
#Override
public void run() {
BufferedReader in = null;
try {
in = new BufferedReader(new FileReader(mOutfile));
while(in.ready()) {
final String str = in.readLine();
mHandler.post(new Runnable() {
#Override
public void run() {
Toast.makeText(RedirectionJni.this, str, Toast.LENGTH_LONG).show();
}
});
}
in.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}).start();
To achieve STDIN redirection:
Native Code:
/*
* Step 1: Make a named pipe
* Step 2: Open the pipe in Read only mode. Java code will open it in Write only mode.
* Step 3: Make STDIN i.e. 0, a duplicate of opened pipe file descriptor.
* Step 4: Any reads from STDIN, will be actually read from the pipe and JAVA code will perform write operations.
*/
int in = mkfifo(infile, 0664);
int fdi = open(infile, O_RDONLY);
dup2(fdi, 0);
char buf[256] = "";
fscanf(stdin, "%*s %99[^\n]", buf); // Use this format to read white spaces.
close(fdi);
Java Code:
/*
* This thread is used for writing content which will be read by native code using STDIN.
*/
new Thread(new Runnable() {
#Override
public void run() {
BufferedWriter out = null;
try {
out = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(mInfile)));
String content = "This content is written to " + mInfile;
out.write(content.toCharArray(), 0, content.toCharArray().length);
out.flush();
out.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}).start();
Let me know if you need any help.
In the Java Code, get a command line by
Process p;
p = Runtime.getRuntime().exec("su"); // or execute something else, su is just to get root
DataOutputStream dos = new DataOutputStream(p.getOutputStream());
dos.writeBytes(ForExampleABinaryPath+" &\n");
dos.flush();
// to read (might has to run parallel in different thread)
InputStream is = p.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
is.readLine() // will get you a line o null
using write && flush and read in parallel, you might be able to achieve your goals.
good luck
I am trying to run a batch file with Runtime.exec() and then output its InputStream into a JTextArea. What I have works, but only partially. What happens is the batch file runs, but if it executes a command other than something like "echo" that command immediately terminates and the next line executes. For example, let's say I try to run a simple batch file like this:
#echo off
echo hello. waiting 5 seconds.
timeout /t 5 /nobreak > NUL
echo finished. goodbye.
The batch file executes, and the JTextArea says
hello. waiting 5 seconds.
finished. goodbye.
but it doesn't wait for 5 seconds in the middle.
I can't figure out why it's doing this. Here's what I use to run the batch file and read its InputStream.
private class ScriptRunner implements Runnable {
private final GUI.InfoGUI gui; // the name of my GUI class
private final String script;
public ScriptRunner(final GUI.InfoGUI gui, final File script) {
this.gui = gui;
this.script = script.getAbsolutePath();
}
#Override
public void run() {
try {
final Process p = Runtime.getRuntime().exec(script);
StreamReader output = new StreamReader(p.getInputStream(), gui);
Thread t = new Thread(output);
t.start();
int exit = p.waitFor();
output.setComplete(true);
while (t.isAlive()) {
sleep(500);
}
System.out.println("Processed finished with exit code " + exit);
} catch (final Exception e) {
e.printStackTrace();
}
}
}
private class StreamReader implements Runnable {
private final InputStream is;
private final GUI.InfoGUI gui;
private boolean complete = false;
public StreamReader(InputStream is, GUI.InfoGUI gui) {
this.is = is;
this.gui = gui;
}
#Override
public void run() {
BufferedReader in = new BufferedReader(new InputStreamReader(is));
try {
while (!complete || in.ready()) {
while (in.ready()) {
gui.setTextAreaText(in.readLine() + "\n");
}
sleep(250);
}
} catch (final Exception e) {
e.printStackTrace();
} finally {
try {
in.close();
} catch (final Exception e) {
e.printStackTrace();
}
}
public void setComplete(final boolean complete) {
this.complete = complete;
}
}
public void sleep(final long ms) {
try {
Thread.sleep(ms);
} catch (final InterruptedException ie) {
}
}
I know my code is pretty messy, and I'm sure it contains grammatical errors.
Thanks for anything you can do to help!
You're creating a Process but you're not reading from its standard error stream. The process might be writing messages to its standard error to tell you that there's a problem, but if you're not reading its standard error, you won't be able to read these messages.
You have two options here:
Since you already have a class that reads from a stream (StreamReader), wire up another one of these to the process's standard error stream (p.getErrorStream()) and run it in another Thread. You'll also need to call setComplete on the error StreamReader when the call to p.waitFor() returns, and wait for the Thread running it to die.
Replace your use of Runtime.getRuntime().exec() with a ProcessBuilder. This class is new in Java 5 and provides an alternative way to run external processes. In my opinion its most significant improvement over Runtime.getRuntime().exec() is the ability to redirect the process's standard error into its standard output, so you only have one stream to read from.
I would strongly recommend going for the second option and choosing to redirect the process's standard error into its standard output.
I took your code and replaced the line
final Process p = Runtime.getRuntime().exec(script);
with
final ProcessBuilder pb = new ProcessBuilder(script);
pb.redirectErrorStream(true);
final Process p = pb.start();
Also, I don't have your GUI code to hand, so I wrote the output of the process to System.out instead.
When I ran your code, I got the following output:
hello. waiting 5 seconds.
ERROR: Input redirection is not supported, exiting the process immediately.
finished. goodbye.
Processed finished with exit code 0
Had you seen that error message, you might have twigged that something was up with the timeout command.
Incidentally, I noticed in one of your comments that none of the commands suggested by ughzan worked. I replaced the timeout line with ping -n 5 127.0.0.1 > NUL and the script ran as expected. I couldn't reproduce a problem with this.
The problem is definitely in timeout.exe. If you add echo %errorlevel% after line with timeout, you will see that it returns 1 if running from java. And 0 if running in usual way. Probably, it requires some specific console functionality (i.e. cursor positioning) that is suppressed when running from java process.
Is there anything I can do to get this to work while running from Java
If you don't need ability to run any batch file then consider to replace timeout with ping. Otherwise... I've tried to run batch file with JNA trough Kernel32.CreateProcess and timeout runs fine. But then you need to implement reading of process output trough native calls also.
I hope someone will suggest better way.
The ready method only tells if the stream can guarantee that something can be read immediately, without blocking. You can't really trust it because always returning false is a valid implementation. Streams with buffers may return true only when they have something buffered. So I suspect your problem is here:
while (!complete || in.ready()) {
while (in.ready()) {
gui.setTextAreaText(in.readLine() + "\n");
}
sleep(250);
}
It should rather read something like this:
String line;
while (!complete || (line=in.readLine()) != null) {
gui.setTextAreaText(line + "\n");
}
It's probably because your "timeout ..." command returned with an error.
Three ways to test it:
Check if the "timeout ..." command works in the Windows command prompt.
Replace "timeout ..." in the script with "ping -n 5 127.0.0.1 > NUL" (it essentially does the same thing)
Remove everything but "timeout /t 5 /nobreak > NUL" from your script. The process should return with an error (1) if the timeout failed because it is the last command executed.
I'm trying to provide communication between a C# app and a Java app on windows using named pipes with the method described by v01ver in this question: How to open a Windows named pipe from Java?
I'm running into a problem on the Java side because I have a reader thread constantly waiting for input on the pipe and when I try to write to the pipe from my main thread it gets stuck forever.
final RandomAccessFile pipe;
try {
pipe = new RandomAccessFile("\\\\.\\pipe\\mypipe", "rw");
}
catch (FileNotFoundException ex) {
ex.printStackTrace();
return;
}
Thread readerThread = new Thread(new Runnable() {
#Override
public void run() {
String line = null;
try {
while (null != (line = pipe.readLine())) {
System.out.println(line);
}
}
catch (IOException ex) {
ex.printStackTrace();
}
}
});
readerThread.start();
try { Thread.sleep(500); } catch (InterruptedException e) {}
try {
System.out.println("Writing a message...");
pipe.write("Hello there.\n".getBytes());
System.out.println("Finished.");
}
catch (IOException ex) {
ex.printStackTrace();
}
The output is: Writing a message...
and then it waits forever.
How can I write to a named pipe while waiting for input in another thread?
This is expected behaviour of pipes. It is supposed to hang untill other process connects to the pipe and reads it.
I have a same problem -- communication between a C#/Python app and a Java app on windows using named pipes:
We have example of Client Code written on Java, but in line String echoResponse = pipe.readLine(); tread waits forever.
try {
// Connect to the pipe
RandomAccessFile pipe = new RandomAccessFile("\\\\.\\pipe\\testpipe", "rw");
String echoText = "Hello word\n";
// write to pipe
pipe.write ( echoText.getBytes() );
// read response
String echoResponse = pipe.readLine();
System.out.println("Response: " + echoResponse );
pipe.close();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Solution of problem:
I have a ServerPipe code written on Python from here Example Code - Named Pipes:
and run its on Python 2.6.6
from ctypes import *
PIPE_ACCESS_DUPLEX = 0x3
PIPE_TYPE_MESSAGE = 0x4
PIPE_READMODE_MESSAGE = 0x2
PIPE_WAIT = 0
PIPE_UNLIMITED_INSTANCES = 255
BUFSIZE = 4096
NMPWAIT_USE_DEFAULT_WAIT = 0
INVALID_HANDLE_VALUE = -1
ERROR_PIPE_CONNECTED = 535
MESSAGE = "Default answer from server\0"
szPipename = "\\\\.\\pipe\\mynamedpipe"
def ReadWrite_ClientPipe_Thread(hPipe):
chBuf = create_string_buffer(BUFSIZE)
cbRead = c_ulong(0)
while 1:
fSuccess = windll.kernel32.ReadFile(hPipe, chBuf, BUFSIZE,
byref(cbRead), None)
if ((fSuccess ==1) or (cbRead.value != 0)):
print chBuf.value
cbWritten = c_ulong(0)
fSuccess = windll.kernel32.WriteFile(hPipe,
c_char_p(MESSAGE),
len(MESSAGE),
byref(cbWritten),
None
)
else:
break
if ( (not fSuccess) or (len(MESSAGE) != cbWritten.value)):
print "Could not reply to the client's request from the
pipe"
break
else:
print "Number of bytes written:", cbWritten.value
windll.kernel32.FlushFileBuffers(hPipe)
windll.kernel32.DisconnectNamedPipe(hPipe)
windll.kernel32.CloseHandle(hPipe)
return 0
def main():
THREADFUNC = CFUNCTYPE(c_int, c_int)
thread_func = THREADFUNC(ReadWrite_ClientPipe_Thread)
while 1:
hPipe = windll.kernel32.CreateNamedPipeA(szPipename,
PIPE_ACCESS_DUPLEX,
PIPE_TYPE_MESSAGE |
PIPE_READMODE_MESSAGE
|
PIPE_WAIT,
PIPE_UNLIMITED_INSTANCES,
BUFSIZE, BUFSIZE,
NMPWAIT_USE_DEFAULT_WAIT,
None
)
if (hPipe == INVALID_HANDLE_VALUE):
print "Error in creating Named Pipe"
return 0
fConnected = windll.kernel32.ConnectNamedPipe(hPipe, None)
if ((fConnected == 0) and (windll.kernel32.GetLastError() ==
ERROR_PIPE_CONNECTED)):
fConnected = 1
if (fConnected == 1):
dwThreadId = c_ulong(0)
hThread = windll.kernel32.CreateThread(None, 0,
thread_func, hPipe, 0, byref(dwThreadId))
if (hThread == -1):
print "Create Thread failed"
return 0
else:
windll.kernel32.CloseHandle(hThread)
else:
print "Could not connect to the Named Pipe"
windll.kernel32.CloseHandle(hPipe)
return 0
if __name__ == "__main__":
main()
After server have start you can use slightly modified version of the Java Client code:
try {
// Connect to the pipe
RandomAccessFile pipe = new RandomAccessFile("\\\\.\\pipe\\mynamedpipe", "rw");
String echoText = "Hello world\n";
// write to pipe
pipe.write(echoText.getBytes());
//String aChar;
StringBuffer fullString = new StringBuffer();
while(true){
int charCode = pipe.read();
if(charCode == 0) break;
//aChar = new Character((char)charCode).toString();
fullString.append((char)charCode);
}
System.out.println("Response: " + fullString);
pipe.close();
}
catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
It works well in NetBeans 6.9.1.
I suppose that RandomAccessFile is not the right API here. Try a FileInputStream + FileOutputStream on the Java side. But that is only a guess, as I last used the Windows API in times when named pipes didn't yet exist.
Don't worry, using RandomAccessFile to access a named pipe is correct. A named pipe is a file system object. Under Linux/Unix it is also called "fifo". Those objects are readable just like a file. (and not the same as pipes used between processes which are abstracted by Java Pipe class).
However I see two problems with your program. I cannot test it currently as I would need your test server (feel free to publish). Your reader thread waits for answers from the other side (i.e. the server). It uses readLine(), I would use a different method (for debugging reading char by char might be the best).
With Java (without JNI) you cannot actually create a named pipe (server side). Opening a named pipe with the generic method used by RandomAccessFile you will get a byte-type stream which can be one-way or duplex.
BTW: JTDS (the free JDBC driver for SQL Server) can optionally use a named pipe to access SQL server, even over the network. And it is using exactly the RandomAccessFile method.
BTW2: there is a makepipe.exe test server on older MS SQL Server installation media, however I did not find a trusted source to get that file.
I'm not familiar with JAVA, and my C# is pretty elementary too. However I'm had a similar problem with a multithreaded C++ client that I fixed by opening the pipe for overlapped IO. Until I did this, Windows serialized reads and writes, effectively causing an unsatisfied (blocking) ReadFile to prevent completion of a subsequent WriteFile until the read was done.
See CreateFile function
FILE_FLAG_OVERLAPPED
So I have a an application which is running on top of gridgain and does so quite successfully for about 12-24 hours of stress testing before it starts to act funny. After this period of time the application will suddenly start replying to all queries with the exception java.nio.channels.ClosedByInterruptException (full stack trace is at http://pastie.org/664717
The method that is failing from is (edited to use #stephenc feedback)
public static com.vlc.edge.FileChannel createChannel(final File file) {
FileChannel channel = null;
try {
channel = new FileInputStream(file).getChannel();
channel.position(0);
final com.vlc.edge.FileChannel fileChannel = new FileChannelImpl(channel);
channel = null;
return fileChannel;
} catch (FileNotFoundException e) {
throw new VlcRuntimeException("Failed to open file: " + file, e);
} catch (IOException e) {
throw new VlcRuntimeException(e);
} finally {
if (channel != null) {
try {
channel.close();
} catch (IOException e){
// noop
LOGGER.error("There was a problem closing the file: " + file);
}
}
}
}
and the calling function correctly closes the object
private void fillContactBuffer(final File signFile) {
contactBuffer = ByteBuffer.allocate((int) signFile.length());
final FileChannel channel = FileUtils.createChannel(signFile);
try {
channel.read(contactBuffer);
} finally {
channel.close();
}
contactBuffer.rewind();
}
The application basically serves as a distributed file parser so it does a lot of these types of operations (will typically open about 10 such channels per query per node). It seems that after a certain period it stops being able to open files and I'm at a loss to explain why this could be happening and would greatly appreciate any one who can tell me what could be causing this and how I could go about tracking it down and fixing it. If it is possibly related to file handle exhaustion, I'd love to hear any tips for finding out for sure... i.e. querying the JVM while it's running or using linux command line tools to find out more information about what handles are currently open.
update: I've been using command line tools to interrogate the output of lsof and haven't been able to see any evidence that file handles are being held open... each node in the grid has a very stable profile of openned files which I can see changing as the above code is executed... but it always returns to a stable number of open files.
Related to this question: Freeing java file handles
There are a couple of scenarios where file handles might not be being closed:
There might be some other code that opens files.
There might be some other bit of code that calls createChannel(...) and doesn't call fillContactBuffer(...)
If channel.position(0) throws an exception, the channel won't be closed. The fix is to rearrange the code so that the following statements are inside the try block.
channel.position(0);
return new FileChannelImpl(channel);
EDIT: Looking at the stack trace, it seems that the two methods are in different code-bases. I'd point the finger of blame at the createChannel method. It is potentially leaky, even if it is not the source of your problems. It needs an in internal finally clause to make sure that the channel is closed in the event of an exception.
Something like this should do the trick. Note that you need to make sure that the finally block does not closes the channel on success!
public static com.vlc.edge.FileChannel createChannel(final File file) {
final FileChannel channel = null;
try {
channel = new FileInputStream(file).getChannel();
channel.position(0);
FileChannel res = new FileChannelImpl(channel);
channel = null;
return res;
} catch (FileNotFoundException e) {
throw new VlcRuntimeException("Failed to open file: " + file, e);
} catch (IOException e) {
throw new VlcRuntimeException(e);
} finally {
if (channel != null) {
try {
channel.close();
} catch (...) {
...
}
}
}
}
FOLLOWUP much later
Given that file handle leakage has been eliminated as a possible cause, my next theory would be that the server side is actually interrupting its own threads using Thread.interrupt(). Some low-level I/O calls respond to an interrupt by throwing an exception, and the root exception being thrown here looks like one such exception.
This doesn't explain why this is happening, but at a wild guess I'd say that it was the server-side framework trying to resolve an overload or deadlock problem.