I want to run an external programs repeated N times, waiting for output each time and process it. Since it's too slow to run sequentially, I tried multithreading.
The code looks like this:
public class ThreadsGen {
public static void main(String[] pArgs) throws Exception {
for (int i =0;i < N ; i++ )
{
new TestThread().start();
}
}
static class TestThread extends Thread {
public void run() {
String cmd = "programX";
String arg = "exArgs";
Process pr;
try {
pr = new ProcessBuilder(cmd,arg).start();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
try {
pr.waitFor();
} catch (InterruptedException e) {
e.printStackTrace();
}
//process output files from programX.
//...
}
However, it seems to me that only one thread is running at a time (by checking CPU usage).
What I want to do is getting all threads (except the one that is waiting for programX to finish) working? What's wrong with my code?
Is it because pr.waitFor(); makes the main thread wait on each subthread?
The waitFor() calls are not your problem here (and are actually causing the spawned Threads to wait on the completion of the spawned external programs rather than the main Thread to wait on the spawned Threads).
There are no guarantees around when Java will start the execution of Threads. It is quite likely, therefore, that if the external program(s) that you are running finish quickly then some of the Threads running them will complete before all the programs are launched.
Also note that CPU usage is not necessarily a good guide to concurrent execution as your Java program is doing nothing but waiting for the external programs to complete. More usefully you could look at the number of executed programs (using ps or Task Manager or whatever).
Isn't yours the same problem as in this thread: How to wait for all threads to finish, using ExecutorService?
Related
Our Java application is having an issue where it blocks indefinitely when it tries to write to a log file located on a NFS share and the NFS share is down.
I was wondering whether we could solve this problem by having a Future execute the write operation with a timeout. Here is a little test program I wrote:
public class write_with_future {
public static void main(String[] args) {
int iteration=0;
while (true) {
System.out.println("iteration " + ++iteration);
ExecutorService executorService = Executors.newSingleThreadExecutor();
Future future = executorService.submit(new Runnable() {
public void run() {
try {
Category fileLogCategory = Category.getInstance("name");
FileAppender fileAppender = new FileAppender(new SimpleLayout(), "/usr/local/app/log/write_with_future.log");
fileLogCategory.addAppender(fileAppender);
fileLogCategory.log(Priority.INFO, System.currentTimeMillis());
fileLogCategory.removeAppender(fileAppender);
fileAppender.close();
}
catch (IOException e) {
System.out.println("IOException: " + e);
}
}
});
try {
future.get(100L, TimeUnit.MILLISECONDS);
}
catch (InterruptedException ie) {
System.out.println("Current thread interrupted while waiting for task to complete: " + ie);
}
catch (ExecutionException ee) {
System.out.println("Exception from task: " + ee);
}
catch (TimeoutException te) {
System.out.println("Task timed out: " + te);
}
finally {
future.cancel(true);
}
executorService.shutdownNow();
}
}
}
When I ran this program with a maximum heap size of 1 MB, and the NFS share was up, this program was able to execute over 1 million iterations before I stopped it.
But when I ran the program with a maximum heap size of 1 MB, and the NFS share was down, the program executed 584 iterations, getting a TimeoutException each time, and then it failed with a java.lang.OutOfMemoryError error. So I am thinking that even though future.cancel(true) and executorService.shutdownNow() are being called, the executor threads are blocked on the write and not responding to the interrupts, and the program eventually runs out of memory.
Is there any way to clean up the executor threads that are blocked?
If appears that Thread.interrupt() does not interrupt threads blocked in an I/O operation on an NFS file. You might want check the NFS mount options, but I suspect that you won't be able to fix that problem.
However, you could certainly prevent it from causing OOME's. The reason you are getting those is that you are not using ExecutorServices as they are designed to be used. What you are doing is repeatedly creating and shutting down single thread services. What you should be doing is creating on instance with a bounded thread pool and using that for all of the tasks. If you do it that way, if one of the threads takes a long time ... or is blocked in I/O ... you won't get a build-up of threads, and run out of memory. Instead, the backlogged tasks will sit in the ExecutorService's work queue until one of the worker thread unblocks.
Hi Every JAVA Developper,
I have juste a simple Question about JVM, i want to know how long the JVM will wait for a thread ?
For Example, take a look at this code :
public static void main(String[] args) {
p = Runtime.getRuntime().exec("myShellCommand -p1 v1 -p2 v2");
p.waitFor();
System.out.println("End ....:)");
}
Suppose that "myShellCommand" running for ever, whats happen then ? the JVM still also waiting for ever ?
the waitFor method causes the current thread to wait, if necessary, until the process represented by this Process object has terminated. This method returns immediately if the subprocess has already terminated. If the subprocess has not yet terminated, the calling thread will be blocked until the subprocess exits.(From the Javadoc).
Based on the documentation, I think that it will run forever.
In your case, the JVM would continue to wait for the launched process to terminate.
However, you could launch another "process monitor" thread and that could wait/sleep for a reasonable time and then interrupt the main thread.
As per the javadoc of the waitFor method
if the current thread is interrupted by another thread while it is
waiting, then the wait is ended and an InterruptedException is thrown.
#Kaoutar
If your requirement is to launch the process and then exit the JVM after a reasonable time period, e.g. 10 minutes, then you could do something like this:
public static void main(String[] args)
{
Thread subProcessThread = new Thread(new Runnable()
{
#Override
public void run()
{
Process p;
try
{
p = Runtime.getRuntime().exec("myShellCommand -p1 v1 -p2 v2");
p.waitFor();
System.out.println("End ....:)");
}
catch (IOException e)
{
e.printStackTrace();
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
});
subProcessThread.start();
long waitTimeInMillis = 10 * 60 * 1000;
subProcessThread.join(waitTimeInMillis);
}
Starting multiple threads and having each exec() then destroy() a running java process result in some of the process not being destroyed and still running after program exit. Here is some code that reproduce the issue. I noticed the more threads you start, the more processes stay alive. And the more sleep before destroy(), the less processes stay alive. (I used InfiniteLoop as an example. Any running process will do the trick.)
EDIT : Bug has been reported to Oracle, waiting for an answer. Feel free to share any knowledge/experiments on the subject.
for(int i = 0; i < 100; i++)
{
new Thread(new Runnable()
{
public void run()
{
try
{
Process p = Runtime.getRuntime().exec(new String[]{"java", "InfiniteLoop"});
Thread.sleep(1);
p.destroy();
}catch(IOException | InterruptedException e){e.printStackTrace();}
}
}).start();
}
Use a p.waitFor(); before p.destroy(); ,
this will ensure the completion of the previous process. I think you p.destroy command gets invoked sooner than the exec() command performs the action. Therefore it becomes useless.
If subprocesses write anything to stdout or stderr (intentionally or not), that could cause trouble:
"Because some native platforms only provide limited buffer size for
standard input and output streams, failure to promptly write the input
stream or read the output stream of the subprocess may cause the
subprocess to block, and even deadlock."
Source: http://www.javaworld.com/jw-12-2000/jw-1229-traps.html
The whole article is IMO worth reading if you need to use Runtime.exec().
This is simply because before the threads execute the destroy call, your main program terminates and all the associated threads leaving the started processes running. To verify this, simply add a System.out call after the destroy and you will find it is not executed. To overcome this add a Thread.sleep at the end of your main method and you will not have the orphaned processes. The below does not leave any process running.
public class ProcessTest {
public static final void main (String[] args) throws Exception {
for(int i = 0; i < 100; i++) {
new Thread(new Runnable()
{
public void run() {
try {
Process p = Runtime.getRuntime().exec(new String[]{"java", "InfiniteLoop"});
Thread.sleep(1);
p.destroy();
System.out.println("Destroyed");
}catch(IOException e) {
System.err.println("exception: " + e.getMessage());
} catch(InterruptedException e){
System.err.println("exception: " + e.getMessage());
}
}
}).start();
}
Thread.sleep(1000);
}
}
You should close the input/output/error streams to the process. We saw some issues in the past where the forked process was not completing properly due to those streams not being closed (even if they weren't being used).
An exemplary solution:
p.destroy();
p.getInputStream().close();
p.getOutputStream().close();
p.getErrorStream().close();
I believe that according to link, a distinct process is spawned by the operating system in response to this call. This process has a lifetime independent of your Java program and threads within it so you would expect it to continue running after your program has exited. I just tried it on my machine and it appeared to work as expected:
import java.io.*;
class Mp {
public static void main(String []args) {
for(int i = 0; i < 100; i++) {
new Thread(new Runnable() {
public void run() {
try {
System.out.println("1");
Process p = Runtime.getRuntime().exec
(new String[]{"notepad", ""});
System.out.println("2");
Thread.sleep(5);
System.out.println("3");
p.destroy();
System.out.println("4");
}
catch(IOException | InterruptedException e) {
e.printStackTrace();
}
}
}).start();
}
}
}
This is not an answer; I am posting complete source for my own attempt at recreating this problem as per discussion in question comments.
I cannot reproduce this problem on Ubuntu 12.04; OpenJDK 6b_27 (however, see below).
ProcessTest.java:
import java.io.*;
public class ProcessTest {
public static final void main (String[] args) throws Exception {
for(int i = 0; i < 100; i++) {
new Thread(new Runnable()
{
public void run() {
try {
Process p = Runtime.getRuntime().exec(new String[]{"java", "InfiniteLoop"});
Thread.sleep(1);
p.destroy();
}catch(IOException e) {
System.err.println("exception: " + e.getMessage());
} catch(InterruptedException e){
System.err.println("exception: " + e.getMessage());
}
}
}).start();
}
}
}
InfiniteLoop.java
public class InfiniteLoop {
public static final void main (String[] args) {
while (true) ;
}
}
I cannot reproduce the issue where processes remaining running after the JVM terminates. However, if I add a long delay in the main thread after starting the threads but before returning from main, I do see roughly a dozen running java processes that stick around (although they are terminated when the main program terminates).
Update:
I just had it leave about 5 processes running after it terminated. It doesn't always happen. Weird. I want to know more about this too. I have a hunch that it has something to do with destroying the process too quickly or some kind of race condition; maybe java forks something off or does something to create a new process that destroy() doesn't take care of if called too quickly / at the wrong time.
I found an old bug (but it is not mark resolved) stating that if a process spawns subprocesses they may not be killed by destroy(). bugs.sun.com/bugdatabase/view_bug.do?bug_id=4770092 What version of the JDK are you using.
Here's another reference to what looks like a similar issue: Java tool/method to force-kill a child process And I want to apologize if I've only added confusion to your life, I don't actually use Process that much and am not familiar with the quirks. Hopefully somebody else will step in with a definitive answer. It seems like it doesn't handle subprocesses well, and I'm presuming java forks something off. That's all I got.
There is a race condition between the time Runtime.exec kicks off a new thread to start a Process and when you tell that process to destroy itself.
I'm on a linux machine so I will use the UNIXProcess.class file to illustrate.
Runtime.exec(...) will create a new ProcessBuilder and start it which on a unix machine creates a new UNIXProcess instance. In the constructor of UNIXProcess there is this block of code which actually executes the process in a background (forked) thread:
java.security.AccessController.doPrivileged(
new java.security.PrivilegedAction() {
public Object run() {
Thread t = new Thread("process reaper") {
public void run() {
try {
pid = forkAndExec(prog,
argBlock, argc,
envBlock, envc,
dir,
redirectErrorStream,
stdin_fd, stdout_fd, stderr_fd);
} catch (IOException e) {
gate.setException(e); /*remember to rethrow later*/
gate.exit();
return;
}
java.security.AccessController.doPrivileged(
new java.security.PrivilegedAction() {
public Object run() {
stdin_stream = new BufferedOutputStream(new
FileOutputStream(stdin_fd));
stdout_stream = new BufferedInputStream(new
FileInputStream(stdout_fd));
stderr_stream = new FileInputStream(stderr_fd);
return null;
}
});
gate.exit(); /* exit from constructor */
int res = waitForProcessExit(pid);
synchronized (UNIXProcess.this) {
hasExited = true;
exitcode = res;
UNIXProcess.this.notifyAll();
}
}
};
t.setDaemon(true);
t.start();
return null;
}
});
Notice that the background thread sets the field pid which is the UNIX process id. This will be used by destroy() to tell the OS which process to kill.
Because there is no way to make sure that this background thread has run when destroy() is called, we may try to kill the process before it has run OR we may try to kill the process before pid field has been set; pid is uninitialized and therefore is 0. So I think calling destroy too early will do the equivalent of a kill -9 0
There is even a comment in the UNIXProcess destroy() that alludes to this but only considers calling destroy after the process has already finished, not before it has started:
// There is a risk that pid will be recycled, causing us to
// kill the wrong process! So we only terminate processes
// that appear to still be running. Even with this check,
// there is an unavoidable race condition here, but the window
// is very small, and OSes try hard to not recycle pids too
// soon, so this is quite safe.
The pid field is not even marked as volatile so we may not even see the most recent value all the time.
I had a very similar issue and the problem with destroy() not working was manifesting even with a single thread.
Process process = processBuilder(ForeverRunningMain.class).start()
long endTime = System.currentTimeMillis() + TIMEOUT_MS;
while (System.currentTimeMillis() < endTime) {
sleep(50);
}
process.destroy();
The process was not always destroyed if TIMEOUT_MS was too low. Adding an additional sleep() before destroy() fixed it (even though I don't have an explanation why):
Thread.sleep(300);
process.destroy();
I'm doing an optimization problem in which I want to execute each Solver thread (one at a time) with random parameters for a fixed period of time. If any of the thread successfully finds a solution, it would return and the program will exit.
I have the code below where I used an ExecutorService and Future to help me accomplish this. However, for some reason, the memory usage of the program increases linearly as time goes on, and the program will terminate with an OutOfMemory error before it gets very far. My Solver code is certainly not the issue as it has no static variables and uses a constant amount of memory. I'm wondering if it's because I'm not cleaning up the threads or handling the exceptions properly, but I can't seem to find any egregious problem from the code.
public class RandomizedSolver {
public static void main(String[] args) {
try {
for (int i = 0; i < 300; i++) {
ExecutorService executor = Executors.newSingleThreadExecutor();
try {
System.out.println("Starting new thread");
Future<Void> future = executor.submit(new Solver(args));
future.get(1, TimeUnit.SECONDS);
executor.shutdownNow();
break;
} catch (TimeoutException e) {
System.out.println("Thread timeout.");
executor.shutdownNow();
continue;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
The point of using ExecutorServices is to reuse their threads, not to keep recreating them. You should revise your design and have only one ExecutorService, with the appropriate number of underlying threads, and submit all you tasks to that unique ExecutorService.
Also note that if your tasks take more than 1 seconds and if they do not terminate promptly when interrupted, you could have up to 300 ExecutorServices and 300 Solver tasks running at the same time. Depending on how much memory you Solver takes, that could result in a OOME.
I want to implement something like this.
1.A background process which will be running forever
2.The background process will check the database for any requests in pending state. If any found,will assign a separate thread to process the request.So one thread per request.Max threads at any point of time should be 10. Once the thread has finished execution,the status of the request will be updated to something,say "completed".
My code outline looks something like this.
public class SimpleDaemon {
private static final int MAXTHREADS = 10;
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(MAXTHREADS);
RequestService requestService = null; //init code omitted
while(true){
List<Request> pending = requestService.findPendingRequests();
List<Future<MyAppResponse>> completed = new ArrayList<Future<MyAppResponse>>(pending.size());
for (Request req:pending) {
Callable<MyAppResponse> worker = new MyCallable(req);
Future<MyAppResponse> submit = executor.submit(worker);
completed.add(submit);
}
// Now retrieve the result
for (Future<MyAppResponse> future : completed) {
try {
requestService.updateStatus(future.getRequestId());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
try {
Thread.sleep(10000); // Sleep sometime
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
Can anyone spend sometime to review this and comment any suggestion/optimization (from multi threading perspective) ? Thanks.
Using a max threads of ten seems somewhat arbitrary. Is this the maximum available connections to your database?
I'm a little confused as to why you are purposefully introducing latency into your applications. Why aren't pending requests submitted to the Executor immediately?
The task submitted to the Executor could then update the RequestService, or you could have a separate worker Thread belonging to the RequestService which calls poll on a BlockingQueue of Future<MyAppResponse>.
You have no shutdown/termination strategy. Nothing indicates that main is run on a Thread that is set to Daemon. If it is, I think the ExecutorService's worker threads will inherit the daemon status, but then your application could shutdown with live connection to the database, no? Isn't that bad?
If the thread isn't really a Daemon, then you need to handle that InterruptedException and treat it as an indication that you are being asked to exit the application.
Your calls to requestService appear to be single threaded resulted in any long running queries preventing completed queries from being completed.
Unless the updateStatus has to be called in a specific order, I suggest you call this as part of your query in MyCallable. This could simplify your code and allow results to be processed as they become available.
You need to handle the potential throwing of a RejectedExecutionException by executor.submit() because the thread-pool has a finite number of threads.
You'd probably be better off using an ExecutorCompletionService rather than an ExecutorService because the former can tell you when a task completes.
I strongly recommend reading Brian Goetz's book "Java Concurrency in Practice".