I have a java code which I want to run. If the job does not complete within, say, 2 hours, then it should be killed automatically (basically, some kind of timed batch).
How to achieve this in Java?
If you are on Java 9 or higher, you can do the timeout batching as below:-
CompletableFuture<Integer> future = CompletableFuture.supplyAsync(this::longRunningTask)
.orTimeout(2, TimeUnit.SECONDS);
future.get(); // j.u.c.ExecutionException after waiting for 2 second
If it completes within the timeout limit, it will return the value (here an Integer object in response to future.get() method)
And, this batching is asynchronous (If you don't call get method explicitly.)
NOTE: This does not prevent the thread from completing the task, it just completes a future in main thread with a Timeout Exception so that main thread can continue. The background task/thread is still continues to finish. (look #Andreas comment)
Some Samples:-
final CompletableFuture<Void> future =
CompletableFuture.supplyAsync(this::longRunningTask)
.orTimeout(2, TimeUnit.SECONDS);
future.get(); // j.u.c.ExecutionException after waiting for 2 second
And the longRunningTask() :-
private Void longRunningTask(){
log.info("Thread name" + Thread.currentThread().getName());
try {
log.info("Going to sleep for 10 sec...");
Thread.sleep(10*1000);
log.info("Sleep Completed. Task Completed.");
} catch (InterruptedException e) {
log.info("Exception Occurred");
}
finally{
log.info("Final Cleanup in progress.");
}
log.info("Finishing the long task.");
return null;
}
If you run above code, it will give Execution Exception in main thread (where future.get() is called) but the longRunningTask will still print Sleep Completed. Task Completed. after completing 10 sec sleep.
If you carefully notice, the longRunnigThread is never interrupted (does not enter in catch block) so continues normally, but main thread gets an exception on get().
Workaround/Solution:
Use ExecutorService and submit the longRunnigTask future with this Exceutor, if timeout occurs, shutdown the executor or else, shutdown after successful get() in case of no timeout exception.
Sample:
ExecutorService myWorkers = Executors.newFixedThreadPool(1);
final CompletableFuture<Void> longTask =
CompletableFuture.supplyAsync(this::longRunningTask, myWorkers)
.orTimeout(2, TimeUnit.SECONDS);
try {
longTask.get();
} catch (InterruptedException | ExecutionException e) {
log.info("EE... Kill the executor thread/s.");
myWorkers.shutdownNow(); // this will interrupt the thread, catch the IntrExcep in thread and return the execution there
}
and the slightly modified longRunnigTask
private Void longRunningTask(){
log.info("Thread name" + Thread.currentThread().getName());
try {
log.info("Going to sleep for 10 sec...");
Thread.sleep(10*1000);
log.info("Sleep Completed. Task Completed.");
} catch (InterruptedException e) {
log.info("Exception Occurred");
return null; // this will finish the thread exce releasing all locking resources. Can be GCed then.
}
finally{
log.info("Final Cleanup in progress.");
}
log.info("Finishing the long task.");
return null;
}
With this approach, it won't complete the task inf timeout is occurred (you won't see Sleep completed. Task completed. in logs..), and would see, exception occurred in the longRunningTask thread (because of interrupt caused by myWorker.shutdown).
Related
So I have a function which looks like this
ExecutorService executorService = Executors.newFixedThreadPool(2000);
Boolean getMore = true;
try{
While (getMore) {
JSONObject response = getPaginatedResponse();
int[] ar = response.get("something");
if (ar.length > 0) {
// loop through the array and invoke executorService.submit() for each
}
else { getMore = false; }
}
executorService.shutdown();
try {
System.out.println("waiting for tasks to complete, termination starting at : "+java.time.LocalDateTime.now());
executorService.awaitTermination(15, TimeUnit.MINUTES);
} catch (InterruptedException e) {
throw new Exception("loading was interrupted... thread pool timed out!");
}
} catch (Exception) {
System.out.println("Fatal error");
}
My issue is that the each of these threads invoke x number of threads, which in turn each call an API and processes its response, the implementation stops execution after all the "First-level" threads gets fired, but not necessarily all the second level ones, which is crucial for my program, how or where can I invoke the executerService.shutdown() to make sure that all the threads were called.
you can put executorService.shutdown(); inside finally block of exception
So my threads are working as expected, and I just wanted to add some extra sauce to my homework.
I made a while loop that checks uses the isShutdown which returns false unless shutdown(); has been called.
So i call shutdown at the end of my method, but it won't ever exit the while loop.
public void runParrallel() throws InterruptedException {
System.out.println("Submitting Task ...");
ExecutorService executor = Executors.newFixedThreadPool(5);
List<Future<TagCounter>> counters = new ArrayList();
counters.add(executor.submit(new TagCounterCallable("https//www.fck.dk")));
counters.add(executor.submit(new TagCounterCallable("https://www.google.com")));
counters.add(executor.submit(new TagCounterCallable("https://politiken.dk")));
counters.add(executor.submit(new TagCounterCallable("https://cphbusiness.dk")));
System.out.println("Task is submitted");
while (!executor.isShutdown()) {
System.out.println("Task is not completed yet....");
Thread.sleep(1000);
}
for (Future<TagCounter> future : counters) {
try {
TagCounter tc = future.get();
System.out.println("Title: " + tc.getTitle());
System.out.println("Div's: " + tc.getDivCount());
System.out.println("Body's: " + tc.getBodyCount());
System.out.println("----------------------------------");
} catch (ExecutionException ex) {
System.out.println("Exception: " + ex);
}
}
executor.shutdown();
}
The while-loop is before you ever call shutdown(). The condition cannot possibly evaluate to false, thus you are stuck with an infinite loop. I'd suggest moving the while loop to the point after you call shutdown().
See also this question on how to shut down an ExecutorService.
Correct me if I'm wrong, but it looks like you want to wait until all tasks that were submitted to your ExecutorService have finished. If you know that they're going to finish in a timely manner, then you can use ExecutorService#shutdown in conjunction with ExecutorService#awaitTermination to block the executing thread until all tasks are complete.
This can be done with the following:
public void runParrallel() throws InterruptedException {
// Same code to submit tasks.
System.out.println("Task is submitted");
executor.shutdown();
executor.awaitTermination(1, TimeUnit.DAYS);
// At this point, the ExecutorService has been shut down successfully
// and all tasks have finished.
for (Future<TagCounter> future : counters) {
try {
TagCounter tc = future.get();
System.out.println("Title: " + tc.getTitle());
System.out.println("Div's: " + tc.getDivCount());
System.out.println("Body's: " + tc.getBodyCount());
System.out.println("----------------------------------");
} catch (ExecutionException ex) {
System.out.println("Exception: " + ex);
}
}
}
With this solution, the while loop can be removed.
Your while-loop is running infinitely because there is nothing that activates the executor.shutdown() inside the while-loop. The code wont progress to the end where you call executor.shutdown() because the while-loop's condition returns back to the start of the while-loop.
Put an if-statement inside the while-loop. The if-statement checks if the task is submitted, and if it is, the executor.shutdown() will be called.
Following is just an example:
while (!executor.isShutdown()) {
System.out.println("Task is not completed yet....");
Thread.sleep(1000);
if(TaskIsCompleted){
executor.shutdown();
}
}
The thread pool executor is being executed in parallel to the main thread. The main thread is not waiting until the shutdown of the executor.
public static void main(String[] args) {
Date jobStartTime = null;
LOGGER.info("MainApp::Job started");
try {
MainApp obj = new MainApp();
// Getting the job Id of the job
String jobName=args[0]; //batch name
String fileName=args[1]; //sqoop file
LOGGER.info("MainApp::jobName: "+jobName+" fileName "+fileName);
currentJobID = obj.getMaxJobId(jobName);
LOGGER.info("MainApp:Job Id is" + currentJobID);
// Getting the start time of the job
jobStartTime = commonDB.getTime();
LOGGER.info("MainApp:Job Start time is" + jobStartTime);
JobDetails job=new JobDetails(currentJobID,jobName,fileName);
// Reading and parsing the sqoop file and executing the sqoop commands
CommandGenerator exec=new CommandGenerator();
List<TableInfo> commandList = exec.parseAndExec(job);
ThreadPoolExecutor tp = (ThreadPoolExecutor) Executors.newFixedThreadPool(10);
for (final TableInfo table : commandList) {
ParallelExecutor pe = new ParallelExecutor(table);
tp.execute(pe);
}
tp.shutdown();
while(!tp.isShutdown()){
}
job=new JobDetails(currentJobID,jobName,fileName,jobStartTime);
//put everything in one method
StatusAndMailUtils status=new StatusAndMailUtils();
status.onJobCompletion(job);
} catch (Exception e) {
// TODO Auto-generated catch block
LOGGER.info("MainApp::Exception");
e.printStackTrace();
}
}
I have used the while loop to keep the main thread waiting mean while the executor threads are in progress. But, it is not helping. Please let me know how to make the main thread wait.
while(!tp.isShutdown()){
}
After having called shutdown(), you can use awaitTermination(long timeout, TimeUnit unit) to block the calling thread until all tasks have completed execution.
As timeout you can use a value excessively big if you want to wait as long as it is needed for the tasks to complete however as it could make your thread wait forever if a task never ends, it is always better to set a reasonable timeout in order to execute some tasks if it is abnormally too long.
For example:
tp.shutdown();
tp.awaitTermination(Long.MAX_VALUE, TimeUnit.DAYS);
Of course it's not waiting. That's the whole idea of creating a threadpool, so your main thread can perform other tasks while the threadpool is executing additional tasks.
You can use the awaitTermination(long timeout, TimeUnit unit) method to have your main thread pause while the threadpool finishes its tasks.
You can also submit these Runnables and wait them to complete. It is also possible to specify a timeout to wait for the threads to execute before throwing an exception.
List<Future<ParallelExecutor>> tasks = new ArrayList<>();
ExecutorService tp = Executors.newFixedThreadPool(10);
for (final TableInfo table : commandList) {
ParallelExecutor pe = new ParallelExecutor(table);
tasks.add(tp.submit(pe));
}
for (Future<ParallelExecutor > p : tasks) {
p.get(); // with timeout p.get(10, TimeUnit.SECONDS);
}
tp.shutdown();
I need to execute two tasks in parallel and wait for them to complete. Also I need the result from the second task, for that I am using Future.
My question is that DO I need executor.awaitTermination to join the tasks or Future.get() will take care of it. Also is there a better way to achieve this with Java 8?
public class Test {
public static void main(String[] args) {
test();
System.out.println("Exiting Main");
}
public static void test() {
System.out.println("In Test");
ExecutorService executor = Executors.newFixedThreadPool(2);
executor.submit(() -> {
for(int i = 0 ; i< 5 ; i++) {
System.out.print("["+i+"]");
try {
Thread.sleep(1000);
} catch (Exception e) {e.printStackTrace();}
}
});
Future<String> result = executor.submit(() -> {
StringBuilder builder = new StringBuilder();
for(int i = 0 ; i< 10 ; i++) {
System.out.print("("+i+")");
try {
Thread.sleep(1000);
} catch (Exception e) {e.printStackTrace();}
builder.append(i);
}
return builder.toString();
});
System.out.println("shutdown");
executor.shutdown();
// DO I need this code : START
System.out.println("awaitTermination");
try {
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
System.out.println("Error");
}
// DO I need this code : END
System.out.println("Getting result");
try {
System.out.println(result.get());
}
catch (InterruptedException e) {e.printStackTrace();}
catch (ExecutionException e) {e.printStackTrace();}
System.out.println("Exiting Test");
}
}
OUTPUT with awaitTermination:
In Test
[0]shutdown
(0)awaitTermination
[1](1)[2](2)[3](3)[4](4)(5)(6)(7)(8)(9)Getting result
0123456789
Exiting Test
Exiting Main
OUTPUT without awaitTermination:
In Test
[0]shutdown
Getting result
(0)[1](1)[2](2)[3](3)[4](4)(5)(6)(7)(8)(9)0123456789
Exiting Test
Exiting Main
From the get javadoc:
Waits if necessary for the computation to complete, and then retrieves its result.
get will wait for the second task only.
From the awaitTermination javadoc:
Blocks until all tasks have completed execution after a shutdown request, or the timeout occurs, or the current thread is interrupted, whichever happens first.
awaitTermination will wait for all tasks.
You should use CompletableFuture API
You can run a process async like follow:
CompletableFuture.supplyAsync( () -> { ... } );
It returns a future, and you can add a callback which will be called when process is finished and result is available.
For example:
CompletableFuture.runAsync( () -> {
// Here compute your string
return "something";
} ).thenAccept( result -> {
// Here do something with result (ie the computed string)
} );
Note that this statement uses internally the ForkJoinPool#commonPool() to execute the process async, but you can also call this statement with your own ExecutorService if you want. In both case, in order to be sure not exiting before tasks are completed, you need to call either get() (which is blocking) on each future of submitted tasks, or wait for the executor to shutdown.
I'm indexing pdf files using Apache lucene with threads. Some threads take time more than 15 minutes. After 15 minutes of thread execution it will throw Thread interrupted Exception.Is there a way to increase the time limit to avoid this issue.
I got this exception when there is a single thread running and it indexed nearly 76% of its pdf files.
application server is Glassfish
List<Thread> threads = new ArrayList<Thread>();
Thread worker;
for (int a = 1;a <= count; a++) {
IndexManualRunnable indexManualRunnable =
new IndexManualRunnable(indexfileLocation, collections, manual, processId);
worker = new Thread(indexManualRunnable);
worker.setName(manual.getName());
worker.setPriority(Thread.MAX_PRIORITY);
worker.start();
threads.add(worker);
}
for (Thread thread : threads) {
try {
thread.join();
} catch (InterruptedException interruptedException) {
saveReport("", "", "Interrupted Exception", 1, processId, Category.INDEXING, thread.getName());
interruptedException.printStackTrace();
}
}
UPDATE:
I see that you are using Glassfish and are saying this interrupt is occurring every time at 15 minutes. It appears Glassfish is set to timeout at around 900 seconds which is 15 minutes by default - this throws an InterruptException.
Since your application viably needs to process for longer than 15 minutes, update the following server config to a time limit you see fit.
http.request-timeout-seconds
Here is an example asadmin command to update the property but I have not tested it:
# asadmin set server-config.network-config.protocols.protocol.<listener-name>.http.request-timeout-seconds=-1
- NOTE: <listener-name> is the name of the listener they wish to disable the timeout on.
- (-1) means unlimited
To deal with an interrupted thread you can catch and handle the exception yourself. If you want the thread to continue executing regardless you would theoretically do nothing - but to keep the code clean and proper I would implement it as below:
boolean interrupted = false;
try {
while (true) {
try {
return queue.take();
} catch (InterruptedException e) {
interrupted = true;
// fall through and retry
}
}
} finally {
if (interrupted)
Thread.currentThread().interrupt();
}
I like this example because it does not just leave an empty catch clause, it instead preserves the fact that it was called in a boolean. It does this so that when it finally does complete you can check if it was interrupted and if so respect it and interrupt the current thread.
For more information and where the example came from look into the following article:
http://www.ibm.com/developerworks/library/j-jtp05236/
Change the domain.xml in Glassfish
<thread-pools>
<thread-pool name="http-thread-pool" idle-thread-timeout-seconds="1800" />
<thread-pool max-thread-pool-size="200" name="thread-pool-1" idle-thread-timeout-seconds="1800" />
</thread-pools>
increase the idle-thread-timeout-seconds