Here's my scenario:
process A spawns child process B and spins threads to drain B's outputs.
process B spawns daemon process C and drains its outputs, too.
process B finishes, daemon process still lives.
process A finds out that process B exited via process.waitFor(). However, it's stuck on reading the input streams of process B. It's because B has started a daemon. The input stream receives EOF only when the process C exits.
This only happens on Windows. I'm using the ProcessBuilder. Here're the solutions I came up with and I'd like to hear your feedback as none of the solutions I really like:
I can use jna to spawn the daemon process C. This way I can create a process that is 'detached enough' and process A is not stuck on draining the streams from B. It works but I'm not very keen on that solution because it means some native code (and lots of that since I'm keen on consuming the inputs). Some inspiration how to do it via JNA is here: http://yajsw.sourceforge.net (however it contains way more stuff than mere process starting).
Run on jre7. Jdk7 brings some new goodies to the ProcessBuilder, e.g. inheritIO() stuff that also solves my problem. Apparently, when inheritIO() is turned on, I can simply close all streams in the daemon process C (which I do anyway because it'a daemon) and that solves the problem. However, I need to run on jre5+
Close the System.out and System.err in process B before spawning the daemon process C. Again, it solves the problem but I really need those streams to work in process B as I write useful stuff to them. No good. I hoped I could take advantage of this characteristic by placing some kind of a bootstrap process between B & C but that didn't solve the problem.
I don't have that problem on linux so could I only run on linux? No, I can't.
Made the process A drain the outputs of process B in a non-blocking way. This somewhat works but it's not convenient. E.g. inputStream.read() is not interruptible. I could use inputStream.available() but it doesn't distinguish between EOF and zero-bytes-available. So the solution is only good if process A is never interested in B's output EOF. Also, this solution seems to be more CPU intensive and generally... feels awkward and not really bullet proof.
Run process C in a --dry-run mode where it just checks if it can be started. It tries to start, sends welcome message and exits. It's no longer long-running so it will not block reads. Process B can gain enough confidence that C can be started and we can use relatively simple JNA code to spawn detached process without consuming its outputs (it's the consuming of outputs makes the JNA-related code messy and heavyweight). The only problem is that we no longer consumer process' C outputs but it can be solved by making C write to a well known file that process B can consume. This solution is more like a big and ugly workaround but kind of workable for us. Anyways, we are trying the solution 1) at the moment.
I would really appreciate any hints!
I just encountered the same problem. I think I have a workaround to the problem. In process A, I have the following code fragment after Process.waitFor(), where outT and errT are the threads to read process B's stdout and stderr, respectively:
try {
outT.join(1000);
if (outT.isAlive()) {
errmsg("stdout reader still alive, interrupting", null);
outT.interrupt();
}
} catch (Exception e) {
errmsg("Exception caught from out stream reader: "+e, e);
}
try {
errT.join(1000);
if (errT.isAlive()) {
errmsg("stderr reader still alive, interrupting", null);
errT.interrupt();
}
} catch (Exception e) {
errmsg("Exception caught from err stream reader: "+e, e);
}
p.destroy();
Not sure if p.destroy() is needed, but I have been trying all kinds of combinations to deal with the problem.
Anyway, in the run() method of the outT/errT threads, I have the following, where the 'pipe' variable is a Writer instance I am capturing stdout/stderr of the sub-process to. The 'in' variable is the stdout, or stderr, stream obtained from Process:
try {
r = new BufferedReader(new InputStreamReader(in, enc));
String line;
while (true) {
if (Thread.currentThread().isInterrupted()) {
errmsg("Text stream reader interrupted", null);
break;
}
if (r.ready()) {
line = r.readLine();
if (line == null) {
break;
}
pipe.write(line);
pipe.write(SystemUtil.EOL);
if (autoFlush) {
pipe.flush();
}
}
}
pipe.flush();
} catch (Throwable t) {
errmsg("Exception caught: "+t, t);
try { pipe.flush(); } catch (Exception noop) {}
} finally {
IOUtil.closeQuietly(in);
IOUtil.closeQuietly(r);
}
It seems that I never get an EOF indication from any sub-process, even after the sub-process terminates, hence, all the chicanery above to prevent stale threads and blocking.
Related
I have a parallel running java application that consumes huge log files and applies some custom logic. Each log row is processed in a separate thread using fire-and-forget approach.
However sometimes the java process just stops processing, what I mean with that is that the java application doesn't get assigned CPU to execute the process even if the application is still hasn't finished consuming the file.
Running top I get quite low load average considering 16 cores that I have:
Running vmstat I can see that non of the user processes are running neither the kernel processes, rather it's idle 99%
The output of iostat shows me that there are no pending IO tasks running either:
I also haven't spotted any deadlocks or starvation taking a thread dump. The most of the threads are WAITING or RUNNABLE.
What am I missing? I got lost, and I don;t really know where to investigate further.
=UPDATE=
This is the part that initiates parallel execution, after this there are thousand lines of code applying modification incl. elasticsearch, akka etc
So I don't really know what the relevant code would be that might causes any troubles.
BlockingQueue<Runnable> workQueue = new ArrayBlockingQueue<Runnable>(100);
ExecutorService executorService = new MetricsThreadPoolExecutor(numThreadCore, numThreadCore, idleTime, TimeUnit.SECONDS, workQueue, new ThreadPoolExecutor.AbortPolicy(), "process.concurrent", metrics);
FileInputStream fileStream = new FileInputStream(file);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(new GZIPInputStream(fileStream));
String strRow = bufferedReader.readLine();
while (strRow != null) {
final Row row = new Row(strRow);
try {
executorService.submit(new Runnable() {
#Override
public void run() {
if (!StringUtil.isBlank(row.getLine())) {
processor.process(row);
}
}
});
strRow = bufferedReader.readLine();
} catch (RejectedExecutionException ree) {
try {
logger.warn(ree.getMessage());
Thread.sleep(50L);
} catch (InterruptedException ie) {
logger.warn("Wait interrupted", ie);
}
}
However sometimes the java process just stops processing, what I mean with that is that the java application doesn't get assigned CPU to execute the process even if the application is still hasn't finished consuming the file.
Don't think about this at the CPU/vmstat/iostat level. That's just confusing the debugging of the problem. You should think about this in terms of threads only and trust the OS to schedule them appropriately.
I see no reason why the main thread shouldn't finish after all of the rows have been submitted for processing. As an aside, you may instead want to just block the producer instead of regenerating the rows in your spin/sleep loop like you are doing. See: RejectedExecutionException free threads but full queue
If you application is not completing then either one of the worker threads is hung while processing the row or maybe the MetricsThreadPoolExecutor has not been shutdown. I suspect the latter. The producer thread, after it exits the while (strRow != null) { loop should call executorService.shutdown(). Otherwise the threads will be waiting for more rows to be added.
You could do a thread-dump on your application to see if it is stuck in a worker. You could add logging when the producer thread finishes which should let you know if it completed it's work. Both might help figure out where the problem lies.
I have a Thread (let's say T1) which reads data from socket:
public void run() {
while (running) {
try {
BufferedReader reader = new BufferedReader( new InputStreamReader(socket.getInputStream()) );
String input = reader.readLine();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Another Thread (lets say T2) try to finish the program in one of its method. Therefore T2 does the following:
T1.running = false;
socket.close();
Here is this scenario for which i couldn't find a solution:
T1 is active and waiting for some input to read i.e. blocking.
context switching
T2 is active and sets running to false, closes the socket
context switching
because T1 was blocking and T2 closed the socket, T1 throws an Exception. What i want is to catch this SocketException. i can't put a try/catch(SocketException) in T1.run(). So how can i catch it in T1's running-method? If it's not possible to catch it in T1's running, then how can i catch it elsewhere?
PS: "Another question about the Thread Debugging"
Normally when i debug the code step by step, i lose the 'active running line' on a context switch. Let's say i'm in line 20 of T1, context switch happens, let's assume the program continues from the 30.line of T2, but the debugger does not go/show to the 30.line of T2, instead the 'active running line' vanishes. So i lose the control over the code. I use Eclipse for Java and Visual Studio for C#. So what is the best way to track the code while debugging on a context switch ?
For your problem assuming you are using a Thread Pool maybe you should make a ThreadFactory that installs a Thread.UncaughtExceptionHandler on all Threads and then invoke your work with execute() on the ExecutorService instead of submit().
For you problem with debugging maybe you should read
http://msdn.microsoft.com/en-us/library/ms164746.aspx
Your code has several other problems so I'll address them all at the same time.
You must create the BufferedReader outside the loop. Otherwise you will lose data in the buffers being discarded each time around the loop.
You must test the result of readLine() for null. If you get it, you must close the BufferedReader and exit the loop.
If you get any exception you must also close the BufferedReader and exit the loop.
What i want is to catch this SocketException.
So catch it.
I can't put a try/catch(SocketException) in T1.run().
You must. No choice. You have to completely rewrite it anyway because of the above items.
I am currently trying to make a litlle handy tool, you see I am a network administrator and my boss told me that he wanted me to monitor the network and block certain sites and ip for some game servers, so for the monitoring part we are going to redirect all traffic on the network to a server where we can monitor the traffic before sending it to the gateway.
For this we are going to use arpspoof in linux and I have finished a solution for the blocking of sites and servers, and what I am going to make is a GUI that makes it easier for me to handle and control these things and when I tried running arpspoof from java using a ProcessBuilder it does not work and I get no output?
It also does not enter the while loop. I can't really think of more to write atm, but if I can think of more I will update this thread.
My code:
new Thread() {
public void run() {
try {
System.out.println("running arpspoof...");
Process prb = new ProcessBuilder("gksudo", "arpspoof", "-i", "wlan0", Gateway).start();
InputStream is = prb.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
while ((line = br.readLine()) != null) {
System.out.println("Output: " + line);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}.start();
I have never used gksudo, but I googled it and it says it's a GUI version of sudo. I'm guessing that you just launched a GUI app which does not write anything to stdout and which does not return. If so, then the code is doing what I would expect. It is blocking until the process writes a line of text that it can read - which never occurs so it blocks indefinitely.
First test your ProcessBuilder code using a trivial command like "echo" to make sure your Java code is working as expected. Then work your way back. Try running your program as root so you don't need the sudo argument and see if that works. Then finally try to run it using sudo instead of gksudo.
I think #user is on the right track, but there are a couple of other possible explanations.
The gksudo command could be asking for a password. I'm not sure where it would ask, but there's a good chance that it won't be the "stdout" stream of the "gksudo" process.
If "gksudo" or the command that you are "gksudo"-ing fails to launch, there is a good chance that it will write an error message to its "stderr" stream. But you are not reading "stderr".
To help diagnose this, you need to try the following:
Look in the log file that for "sudo" - it is "/var/log/secure" on my box.
Use "ps -efl" (or similar) to see what processes exist while your application is blocked waiting for output. (If that is happening ...)
Look to see if "gksudo" is prompting for a password in an unexpected place.
Try temporarily tweaking the "sudoers" file to allow the "arpspoof" command to be "sudo"-ed without a password.
I have 2 classes where I have one to send commands to a socket, another which receives (and sometimes answers commands) .
Do I have to synchronize this? Or is this not necessary?
Both classes run in their own threads with the socket object passed down to each of them as argument upon thread.start();
Is this the proper way to do it or could I do something more efficient?
Would this have chances of causing errors?
The sending part:
public void run(){
send_chatline("roomf");
int vvv = 0;
while (this.socket.isConnected()){
try{
Thread.sleep(10000);
vvv++;
Thread.sleep(10000);
send_chatline("alive");
Thread.sleep(10000);
if (vvv == 1) {
this.socket.flush();
this.socket.send("T,-1," + this.playerid * 3);
this.socket.flush();
}
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
}
But remember! The recieveFromSock class also writes sometimes when specific commands appear.
The only function of sendTosock is to keep the connection alive (being able to remain online).
The only guarantee you have is that a byte sent, is sent once, and that a byte read is read once. Others may say that the synchronization is taken care of by the runtime-library, or the underlying operating system. Any such statement is runtime-implementation dependent and/or operating system dependent (and should in my opinion be disregarded from).
So, if you have one thread reading from the socket, and one thread writing to the socket, you should be fine without synchronization.
If more than one thread may write to the socket, you should synchronize their actions to make sure the output sent from the two threads doesn't get interleaved.
Same applies if two threads read from the socket. You should synchronize the reads to make sure that data read by one thread doesn't have "gaps" due to reads of another thread.
Similar question, same conclusion:
in what way is java.net.Socket threadsafe?
Do I have to synchronize this?
No. SocketImpl has its own internal synchronization. You won't be surprised to learn that sending isn't synchronized but receiving is.
You don't need to synchronize, as the synchronization is done on the socket. The question is: are the two classes/threads communicating through sockets living in the same JVM? If so, there are several more efficient options, the most simple being having a BlockingQueue where the sender class adds its commands and the receiver takes them.
I was curious too for the nio classes and it IS synchronized(for nio only). Just look at the write method here in the sun JVM code
http://www.docjar.com/html/api/sun/nio/ch/SocketChannelImpl.java.html
Checking the old i/o socket code however shows no synchronization though there is an acquireFD and releaseFD...but no sycnhronization there...that appears to prevent a close until all writes on all threads are done(it is like they assume writes from multiple threads are okay in every OS...but I don't know if that is true OR they synchronize in the JVM native code so the developer knows they can do that)....we would need a JVM developer to tell us if there is a synch block in the windows JVM, linux JVM and mac JVM, etc. etc....
private void socketWrite(byte b[], int off, int len) throws IOException {
if (len <= 0 || off < 0 || off + len > b.length) {
if (len == 0) {
return;
}
throw new ArrayIndexOutOfBoundsException();
}
FileDescriptor fd = impl.acquireFD();
try {
socketWrite0(fd, b, off, len);
} catch (SocketException se) {
if (se instanceof sun.net.ConnectionResetException) {
impl.setConnectionResetPending();
se = new SocketException("Connection reset");
}
if (impl.isClosedOrPending()) {
throw new SocketException("Socket closed");
} else {
throw se;
}
} finally {
impl.releaseFD();
}
}
Runtime.getRuntime.exex("abc.exe -parameters");
using .waitFor() does not help to determine the completion of process.
Looks like JDK8 introduces Process.isAlive(). Surprised it took so long...
In the meantime, the best option seems to be to poll Process.exitValue(), wrapped in a try-catch:
// somewhere previous...
String[] cmd = { "abc.exe", "-p1", "-p2" };
Process process = Runtime.getRuntime.exec(cmd);
// call this method repeatedly until it returns true
private boolean processIsTerminated () {
try {
process.exitValue();
} catch (IllegalThreadStateException itse) {
return false;
}
return true;
}
Alternately, a similar method could return the exit value if the process had terminated, or some other specified value if not.
Process.waitFor() (javadoc) should work. If it doesn't work then either:
there's a bug in the JVM or the OS (highly unlikely for something like this), or
there is something about the process and/or your Java code that means that the process won't exit.
In current releases of Java you can also use Process.isAlive (javadoc) to test the process status without blocking until it finishes. For Java 7 and older there is a hacky solution that entails polling the process return code and catching an exception, but this is inefficient. You should upgrade to Java 8 or later as soon as possible!
Once the task is finished its goes for an indefinite wait. (I don't know why).
If this happening, then neither waitFor() or isAlive() will help.
The most likely reasons that a process launched from Java won't / can't exit are:
the process is blocked waiting for your Java application to give it some input (via its stdin),
the process is blocked waiting for your Java application to read its output (i.e. its stdout or stderr),
it is blocked waiting on some external event; e.g. if it is trying to talk remote server that is not responding,
something has sent it a STOP signal of some kind, or
it is just taking a looong time to run.
The first two of these reasons / causes can be addressed by (respectively) closing the Java output stream connected to its standard input, and reading (and possibly discarding) the Java input streams connected to its standard output and standard error. The other causes are intractable, and your only options are to "wait it out" or attempt to kill off the process.
Bottom line - you need to find out why your process isn't completing. The blocked Process.waitFor() call is a symptom, not the disease.
I have a similar issue and neither of the methods written here works for me. This is my code:
public void startCCleaner() {
System.out.println("Starting ccleaner...");
try {
Process process = new ProcessBuilder("C:\\Program Files\\CCleaner\\CCleaner64.exe").start();
if(process.waitFor() == 0 ){
System.out.println("Process terminated ");
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
If you don't want to use waitFor(), which apparently you don't you can always test the exit value directly.
import java.util.*;
import java.io.*;
public class ProcExitTest
{
public static void main(String args[])
{
try
{
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec("<....>");
int exitVal = proc.exitValue();
System.out.println("Process exitValue: " + exitVal);
}
catch (InterruptedException ie)
{
ie.printStackTrace();
}
}
}
exit code 0 means normal termination.