I'm doing an optimization problem in which I want to execute each Solver thread (one at a time) with random parameters for a fixed period of time. If any of the thread successfully finds a solution, it would return and the program will exit.
I have the code below where I used an ExecutorService and Future to help me accomplish this. However, for some reason, the memory usage of the program increases linearly as time goes on, and the program will terminate with an OutOfMemory error before it gets very far. My Solver code is certainly not the issue as it has no static variables and uses a constant amount of memory. I'm wondering if it's because I'm not cleaning up the threads or handling the exceptions properly, but I can't seem to find any egregious problem from the code.
public class RandomizedSolver {
public static void main(String[] args) {
try {
for (int i = 0; i < 300; i++) {
ExecutorService executor = Executors.newSingleThreadExecutor();
try {
System.out.println("Starting new thread");
Future<Void> future = executor.submit(new Solver(args));
future.get(1, TimeUnit.SECONDS);
executor.shutdownNow();
break;
} catch (TimeoutException e) {
System.out.println("Thread timeout.");
executor.shutdownNow();
continue;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
The point of using ExecutorServices is to reuse their threads, not to keep recreating them. You should revise your design and have only one ExecutorService, with the appropriate number of underlying threads, and submit all you tasks to that unique ExecutorService.
Also note that if your tasks take more than 1 seconds and if they do not terminate promptly when interrupted, you could have up to 300 ExecutorServices and 300 Solver tasks running at the same time. Depending on how much memory you Solver takes, that could result in a OOME.
Related
Our Java application is having an issue where it blocks indefinitely when it tries to write to a log file located on a NFS share and the NFS share is down.
I was wondering whether we could solve this problem by having a Future execute the write operation with a timeout. Here is a little test program I wrote:
public class write_with_future {
public static void main(String[] args) {
int iteration=0;
while (true) {
System.out.println("iteration " + ++iteration);
ExecutorService executorService = Executors.newSingleThreadExecutor();
Future future = executorService.submit(new Runnable() {
public void run() {
try {
Category fileLogCategory = Category.getInstance("name");
FileAppender fileAppender = new FileAppender(new SimpleLayout(), "/usr/local/app/log/write_with_future.log");
fileLogCategory.addAppender(fileAppender);
fileLogCategory.log(Priority.INFO, System.currentTimeMillis());
fileLogCategory.removeAppender(fileAppender);
fileAppender.close();
}
catch (IOException e) {
System.out.println("IOException: " + e);
}
}
});
try {
future.get(100L, TimeUnit.MILLISECONDS);
}
catch (InterruptedException ie) {
System.out.println("Current thread interrupted while waiting for task to complete: " + ie);
}
catch (ExecutionException ee) {
System.out.println("Exception from task: " + ee);
}
catch (TimeoutException te) {
System.out.println("Task timed out: " + te);
}
finally {
future.cancel(true);
}
executorService.shutdownNow();
}
}
}
When I ran this program with a maximum heap size of 1 MB, and the NFS share was up, this program was able to execute over 1 million iterations before I stopped it.
But when I ran the program with a maximum heap size of 1 MB, and the NFS share was down, the program executed 584 iterations, getting a TimeoutException each time, and then it failed with a java.lang.OutOfMemoryError error. So I am thinking that even though future.cancel(true) and executorService.shutdownNow() are being called, the executor threads are blocked on the write and not responding to the interrupts, and the program eventually runs out of memory.
Is there any way to clean up the executor threads that are blocked?
If appears that Thread.interrupt() does not interrupt threads blocked in an I/O operation on an NFS file. You might want check the NFS mount options, but I suspect that you won't be able to fix that problem.
However, you could certainly prevent it from causing OOME's. The reason you are getting those is that you are not using ExecutorServices as they are designed to be used. What you are doing is repeatedly creating and shutting down single thread services. What you should be doing is creating on instance with a bounded thread pool and using that for all of the tasks. If you do it that way, if one of the threads takes a long time ... or is blocked in I/O ... you won't get a build-up of threads, and run out of memory. Instead, the backlogged tasks will sit in the ExecutorService's work queue until one of the worker thread unblocks.
I am using ExecutorService for creating Thread. In the run method, its processing a time consuming operations. It takes nearly upto 10 seconds to complete it. For testing, here I am using Thread.sleep(10000);
My question is, If I use newFixedThreadPool as 2000, will it really execute 2000 threads at a time?
public class ThreadPoolTest {
public static void main(String[] args) {
System.out.println(new Date());
ExecutorService executor = Executors.newFixedThreadPool(2000);
IntStream.range(0, 2000).forEach(
i -> {
Runnable worker = new WorkerThread("WorkerThread-" + i);
executor.submit(worker);//calling execute method of ExecutorService
}
);
executor.shutdown();
while (!executor.isTerminated()) { }
System.out.println("Finished all threads");
System.out.println("Main thread finished");
System.out.println(new Date());
}
}
public class WorkerThread implements Runnable{
private String name;
public WorkerThread(String s){
this.name=s;
}
public void run() {
System.out.println(Thread.currentThread().getName()+" (Start) message = "+name);
processData();
}
private void processData(){
try { Thread.sleep(10000); } catch (InterruptedException e) { e.printStackTrace(); }
}
}
I am printing the time in this code. Its showing that, it took only 10 seconds to complete the entire process. That means all 2000 threads executed parallelly? I have heard that the number of actual threads running will be based on the number of cores in the system. But how did all 2000 threads run parallely?
The number of logical CPUs determines the number of threads running at a given moment.
However, a CPU can switch threads every 100 microseconds or about 10,000 times per second giving the illusion more threads are running at once. Calling sleep is likely to cause the CPU to context switch until after the timeout.
NOTE: System.out.println holds a lock for the output, so in reality, only one thread at a time is doing real work, and even then the program is probably being slowed down by the speed your screen will update. Try removing the println (and the sleep) and it should complete in a fraction of the time.
But how did all 2000 threads run parallely?
Almost all of them were asleep. The only busy thread was likely to be the one updating the screen with the buffered println messages.
This question already has answers here:
ExecutorService that interrupts tasks after a timeout
(11 answers)
Closed 7 years ago.
This is just an example to explain my problem...
I am using ExecutorService with 20 active threads and 75K max queued items...
In my case, a normal task should not take more than 10 seconds, if it takes more time that means there's some problem with the task.
If all the threads are hung due to problematic tasks my RejectionHandler would restart the entire service.
I have two questions here:
I do not like the idea of restarting the service, instead if there's
way to detect hanging thread and we could just restart that hung
thread that would be great. I have gone through couple of articles to handle hung threads with ThreadManager but have not found anything
with ExecutorService.
I am very much fascinated about the Executors.newCachedThredPool()
because on peak days we are heavily loaded with incoming tasks, and
on other days they are very few. Any suggestions would be greatly
appreciated.
public class HangingThreadTest {
// ExecutorService executorService = Executors.newCachedThreadPool()
private static ExecutorService executorService = new ThreadPoolExecutor(10,
20, 5L, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(75000));
public static void main(String... arg0) {
for (int i = 0; i < 50000; i++) {
executorService.submit(new Task());
}
}
}
/**
* Task to be completed
*/
class Task implements Runnable {
private static int count = 0;
#Override
public void run() {
count++;
if (count%5 == 0) {
try {
System.out.println("Hanging Thread task that needs to be reprocessed: "
+ Thread.currentThread().getName()+" count: "+count);
Thread.sleep(11000);
} catch (InterruptedException e) {
// Do something
}
}
else{
System.out.println("Normal Thread: "
+ Thread.currentThread().getName()+" count: "+count);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
//Do something
}
}
}
}
There is no build-in mechanism in Executors framework that would help terminate a thread if it has been running for more than a threshold value.
But we can achieve this with some extra code as below:
Get the Future object returned by the executorService.submit(...);.
Future future = executorService.submit(new Task());
Call the get method on this future object to and make it wait only for threshold interval for task completion. Below, an example that is waits for only 2 secs.
try {
f.get(2, TimeUnit.SECONDS);
} catch (TimeoutException e) {
f.cancel(true);
} catch (Exception e) {}
The above code waits for 2 seconds for task completion it throws a TimeoutException if it doesn't get completed during that time. Subsequently we can call cancel method on the future object. This results in setting the interrupt flag in the thread that is executing the task.
Now the final change is, in the Task class code we need to check at necessary points (application dependent), whether the interrupt flag has been set to true using isInterrupted() method of Thread class. If interrupted==true, we can do the necessary clean up and return from the run method immediately. The critical piece here is to identify the necessary points in your Task class where you want to check for this interrupted flag.
This makes the thread available for processing next task.
You may have a look at this article, it was very helpful for me before when I was facing the same problem : Java Hanging Thread Detection
I want to run an external programs repeated N times, waiting for output each time and process it. Since it's too slow to run sequentially, I tried multithreading.
The code looks like this:
public class ThreadsGen {
public static void main(String[] pArgs) throws Exception {
for (int i =0;i < N ; i++ )
{
new TestThread().start();
}
}
static class TestThread extends Thread {
public void run() {
String cmd = "programX";
String arg = "exArgs";
Process pr;
try {
pr = new ProcessBuilder(cmd,arg).start();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
try {
pr.waitFor();
} catch (InterruptedException e) {
e.printStackTrace();
}
//process output files from programX.
//...
}
However, it seems to me that only one thread is running at a time (by checking CPU usage).
What I want to do is getting all threads (except the one that is waiting for programX to finish) working? What's wrong with my code?
Is it because pr.waitFor(); makes the main thread wait on each subthread?
The waitFor() calls are not your problem here (and are actually causing the spawned Threads to wait on the completion of the spawned external programs rather than the main Thread to wait on the spawned Threads).
There are no guarantees around when Java will start the execution of Threads. It is quite likely, therefore, that if the external program(s) that you are running finish quickly then some of the Threads running them will complete before all the programs are launched.
Also note that CPU usage is not necessarily a good guide to concurrent execution as your Java program is doing nothing but waiting for the external programs to complete. More usefully you could look at the number of executed programs (using ps or Task Manager or whatever).
Isn't yours the same problem as in this thread: How to wait for all threads to finish, using ExecutorService?
I want to implement something like this.
1.A background process which will be running forever
2.The background process will check the database for any requests in pending state. If any found,will assign a separate thread to process the request.So one thread per request.Max threads at any point of time should be 10. Once the thread has finished execution,the status of the request will be updated to something,say "completed".
My code outline looks something like this.
public class SimpleDaemon {
private static final int MAXTHREADS = 10;
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(MAXTHREADS);
RequestService requestService = null; //init code omitted
while(true){
List<Request> pending = requestService.findPendingRequests();
List<Future<MyAppResponse>> completed = new ArrayList<Future<MyAppResponse>>(pending.size());
for (Request req:pending) {
Callable<MyAppResponse> worker = new MyCallable(req);
Future<MyAppResponse> submit = executor.submit(worker);
completed.add(submit);
}
// Now retrieve the result
for (Future<MyAppResponse> future : completed) {
try {
requestService.updateStatus(future.getRequestId());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
try {
Thread.sleep(10000); // Sleep sometime
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
Can anyone spend sometime to review this and comment any suggestion/optimization (from multi threading perspective) ? Thanks.
Using a max threads of ten seems somewhat arbitrary. Is this the maximum available connections to your database?
I'm a little confused as to why you are purposefully introducing latency into your applications. Why aren't pending requests submitted to the Executor immediately?
The task submitted to the Executor could then update the RequestService, or you could have a separate worker Thread belonging to the RequestService which calls poll on a BlockingQueue of Future<MyAppResponse>.
You have no shutdown/termination strategy. Nothing indicates that main is run on a Thread that is set to Daemon. If it is, I think the ExecutorService's worker threads will inherit the daemon status, but then your application could shutdown with live connection to the database, no? Isn't that bad?
If the thread isn't really a Daemon, then you need to handle that InterruptedException and treat it as an indication that you are being asked to exit the application.
Your calls to requestService appear to be single threaded resulted in any long running queries preventing completed queries from being completed.
Unless the updateStatus has to be called in a specific order, I suggest you call this as part of your query in MyCallable. This could simplify your code and allow results to be processed as they become available.
You need to handle the potential throwing of a RejectedExecutionException by executor.submit() because the thread-pool has a finite number of threads.
You'd probably be better off using an ExecutorCompletionService rather than an ExecutorService because the former can tell you when a task completes.
I strongly recommend reading Brian Goetz's book "Java Concurrency in Practice".