Develop Reference

How to make a block of code execute without interruption Java?

I'm trying to create a web crawler.
I've created a class to handle all URLs visited and to visit.
This class has to be accessed by multiple threads for retrieving and updating those lists.
The problem I'm facing, or at least I think, is in nextRandom() and probably also in next(). I think what is happening is the threads are interfering with each other since the function is somewhat synchronized but not atomic. Is there a way to make so this block of code is executed without any interruption by other threads?
The URL handler
import java.util.*;
import java.util.concurrent.ThreadLocalRandom;
public class UrlHandler {
private volatile Set<String> visited = new HashSet<String>();
private volatile List<String> toVisit = new ArrayList<String>();
public void addToVisit(String url) {
synchronized (this){
if (!visited.contains(url)) toVisit.add(url);
}
}
public void addToVisit(Collection<String> urls) {
synchronized (this){
for (String url : urls)
if (!visited.contains(url)) toVisit.add(url);
}
}
public void addVisited(String url){
synchronized (this){
visited.add(url);
}
}
public void addVisited(Collection<String> urls){
synchronized (this){
visited.addAll(urls);
}
}
public String next() {
while (toVisit.size() == 0) {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
synchronized (this){
String url = toVisit.get(0);
toVisit.remove(0);
return url;
}
}
public String nextRandom() {
synchronized (this){
int n = 0;
if (toVisit.size() > 1){
n = ThreadLocalRandom.current().nextInt(toVisit.size());
}
String url = toVisit.get(n);
toVisit.remove(n);
return url;
}
}
public List<String> getToVisit() {
synchronized (this){
return toVisit;
}
}
public Set<String> getVisited() {
synchronized (this){
return visited;
}
}
}
Web Crawler
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class WebCrawler {
private final ExecutorService executor;
public WebCrawler(int nOfThreads) {
this.executor = Executors.newFixedThreadPool(nOfThreads);
}
public void add(Runnable runnable) {
this.executor.execute(runnable);
}
//Used to shut down safely and wait also 5 of seconds for not finished tasks
public void shutdown() {
this.executor.shutdown();
try {
this.executor.awaitTermination(5, TimeUnit.SECONDS);
if (!this.executor.isTerminated()) {
System.err.println("Timed out waiting for executor to terminate cleanly. Shutting down.");
this.executor.shutdownNow();
}
} catch (final InterruptedException e) {
System.err.println("Interrupted while waiting for executor shutdown.");
Thread.currentThread().interrupt();
}
}
}
Failing test example
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class UrlHandlerTest {
List<String> testList = new ArrayList<>(List.of("test1", "test2", "test3", "test3"));
List<String> uniqueTestList = new ArrayList<>(List.of("test1", "test2", "test3"));
UrlHandler urlHandler = new UrlHandler();
#Test
public void concurrentAccess(){
urlHandler.addToVisit(testList);
WebCrawler webCrawler = new WebCrawler(10);
for (int i = 0; i < urlHandler.getToVisit().size(); i++) {
webCrawler.add(new Runnable() {
#Override
public void run() {
String url = urlHandler.nextRandom();
urlHandler.addVisited(url);
System.out.println("Here thread " + Thread.currentThread().getId() + " working on: " + url);
}
});
}
webCrawler.shutdown();
System.out.println(urlHandler.getVisited());
assertEquals(true, urlHandler.getVisited().containsAll(uniqueTestList));
}
}

In the next method this code is a problem:
while (toVisit.size() == 0) {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
The lock isn't held for this part, so size can be stale. Instead of this, try something like
while (toVisit.size() == 0)
wait();
Do this in a synchronized block so you have the lock held while checking the collection size. Code that adds to the collection should notify in order to wake up the waiting threads.

This piece of code is problematic:
for (int i = 0; i < urlHandler.getToVisit().size(); i++) {
webCrawler.add(new Runnable() {
// ...
});
}
The urlHandler.getToVisit().size() is always changing during the traversal, and there is uncertainty (because the size will be changed asynchronously).
Change to:
int size = urlHandler.getToVisit().size();
for (int i = 0; i < size; i++) {
webCrawler.add(new Runnable() {
// ...
});
}

Simulate the multi-thread process of task T by using Future Class

Here is a task T:
T is composed of many subtasks, and the completion time of each subtask is different.If one of the subtasks fail, the others should be stoped at once and the task T will fail.
So how to simulate the process of task T? (require fast-failure)
Maybe Future Class could solve it. But how?

The easiest fix to this is to separate out these Runnables into their own threadpool so you can then call shutdownNow() on that pool only which interrupts all of the tasks in that pool

Here's an example using FutureTask.
One of the tasks simulates failure by waiting for 1 second and then cancelling the other tasks. Even though they are waiting for 5 seconds, the program completes in 1 second.
You would need to wrap your Runnable in something like this which can cancel if the actual task fails.
To be able to cancel a task, it needs to be in a state where Thread.interrupt() will stop the thread. If it is just sitting in a loop, for instance, then the Future will be cancelled (so get() will return immediately), but the task itself will keep running.
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.FutureTask;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
public class FutureTaskExample {
public static void main(String[] args) throws InterruptedException {
List<FutureTask<String>> tasks = new ArrayList<>();
ExecutorService executorService = Executors.newFixedThreadPool(10);
long start = System.currentTimeMillis();
for (int i = 0; i < 10; ++i) {
final int ii = i;
FutureTask<String> task =
new FutureTask<>(new Callable<String>() {
#Override
public String call() throws Exception {
if (ii == 9) {
Thread.sleep(1000);
tasks.subList(0,9).stream().forEach(t -> t.cancel(true));
return "Failure";
} else {
long start = System.currentTimeMillis();
Thread.sleep(5000);
System.out.println("Task " + ii + " slept for " + (System.currentTimeMillis() - start));
return "Completed";
}
}
});
tasks.add(task);
executorService.execute(task);
}
List<String> results = tasks.stream().map(t -> {
try {
return t.get();
} catch (Exception e) {
// ignore
e.printStackTrace();
return "Interrupted";
}
}).collect(Collectors.toList());
System.out.println("Completed in " + (System.currentTimeMillis() - start) + " " + results);
executorService.shutdown();
executorService.awaitTermination(10, TimeUnit.SECONDS);
System.out.println("Done in " + (System.currentTimeMillis() - start));
}
}

I give a simulation example by using CompletableFuture
I create a Task Class TaskT, make it looks more real. And the Task Class contains three state, runTask() method and cancel() method.
Three state are Success, Cancelling and Cancelled.Success means task run completed.Cancelling means task is cancelling. Cancelled means the task has been Cancelled, advice others task cancel quickly.
runTask() method: I use Thread.sleep(interval) to simulate runing state. And when the task run finish, the code if(cancelled) return Result.CANCELLED; detect whether the state is canceled?
cancel() method: I use double-check lock to decorate the "cancel" logic to ensure it is single instance.
Finally, in the main metod, I use CompletableFuture Class to start thread and return param. thenAccept() method is useful and help SUCCESS TASK advice the other RUNNING TASKs cancel the task.
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
public class Competable {
// three state after run task
static enum Result{
SUCCESS, FAIL, CANCELLED
}
// task list
static List<TaskT> tasks = new ArrayList<>();
/**
* Task List
*/
public static class TaskT{
private String name;
private int timeInSecond;
private Result ret;
volatile boolean cancelling = false;
volatile boolean cancelled = false;
public TaskT(String name, int timeInSecond, Result ret){
this.name = name;
this.timeInSecond = timeInSecond * 1000;
this.ret = ret;
}
/**
* Simulate task runing
* runing time in real work is uncertain
* maybe run in computing，maybe run in IO
*/
public Result runTask(){
int interval = 100;
int total = 0;
try {
for(;;){
Thread.sleep(interval);
total += interval;
if(total>=timeInSecond) break;
if(cancelled) return Result.CANCELLED;
}
}catch (InterruptedException e){
e.printStackTrace();
}
System.out.println(name + "Task End!!!");
return ret;
}
/**
* Simlulate task cancel
* and set cancel time
*/
public void cancel() {
if (!cancelled) {
synchronized (this) {
if (cancelled) return;
cancelling = true;
System.out.println(name + "cancelling!!!");
try {
TimeUnit.MILLISECONDS.sleep(50);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(name + "cancelled!!!");
}
cancelled = true;
}
}
}
/**
* rollback: advice the other thread cancel
*/
public static void callback(Result result, TaskT task){
if(Result.FAIL == result){
for(TaskT _task : tasks){
if(_task!=task){
_task.cancel();
}
}
}
}
public static void main(String[] args) throws IOException {
TaskT subtask1 = new TaskT("task1", 3, Result.SUCCESS);
TaskT subtask2 = new TaskT("task2", 4, Result.SUCCESS);
TaskT subtask3 = new TaskT("task3", 1, Result.FAIL);
tasks.add(subtask1);
tasks.add(subtask2);
tasks.add(subtask3);
for(TaskT task:tasks){
CompletableFuture f = CompletableFuture.supplyAsync(()->task.runTask())
.thenAccept((result -> callback(result, task)));
}
// System.in.read();
}
}

Java timeout multiple tasks in parallel

What is the best practice approach to launch a pool of 1000's of tasks (where up to 4 should be able to execute in parallel) and automatically timeout them if they take more than 3 seconds (individually)?
While I found that ExecutorService seems to be helpful (see SSCE from another post below), I don't see how to make this work for multiple tasks running in parallel (as the future.get(3, TimeUnit.SECONDS) is executing on the same thread than the one launching the tasks, hence no opportunity to launch multiple tasks in parallel):
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
public class Test {
public static void main(String[] args) throws Exception {
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<String> future = executor.submit(new Task());
try {
System.out.println("Started..");
System.out.println(future.get(3, TimeUnit.SECONDS));
System.out.println("Finished!");
} catch (TimeoutException e) {
future.cancel(true);
System.out.println("Terminated!");
}
executor.shutdownNow();
}
}
class Task implements Callable<String> {
#Override
public String call() throws Exception {
Thread.sleep(4000); // Just to demo a long running task of 4 seconds.
return "Ready!";
}
}
Thanks!

If you have to monitor each task to kill it when it exceeds the timeout period, either
the task itself has to keep track of time and quit appropriately, OR
you have to create a second watchdog thread for every task. The watchdog thread sets a timer and sleeps, waking up after the timeout interval expires and then terminating the task if it's still running.

This is a tricky one. Here’s what I came up with:
public class TaskQueue<T> {
private static final Logger logger =
Logger.getLogger(TaskQueue.class.getName());
private final Collection<Callable<T>> tasks;
private final int maxTasks;
private int addsPending;
private final Collection<T> results = new ArrayList<T>();
private final ScheduledExecutorService executor;
public TaskQueue() {
this(4);
}
public TaskQueue(int maxSimultaneousTasks) {
maxTasks = maxSimultaneousTasks;
tasks = new ArrayDeque<>(maxTasks);
executor = Executors.newScheduledThreadPool(maxTasks * 3);
}
private void addWhenAllowed(Callable<T> task)
throws InterruptedException,
ExecutionException {
synchronized (tasks) {
while (tasks.size() >= maxTasks) {
tasks.wait();
}
tasks.add(task);
if (--addsPending <= 0) {
tasks.notifyAll();
}
}
Future<T> future = executor.submit(task);
executor.schedule(() -> future.cancel(true), 3, TimeUnit.SECONDS);
try {
T result = future.get();
synchronized (tasks) {
results.add(result);
}
} catch (CancellationException e) {
logger.log(Level.FINE, "Canceled", e);
} finally {
synchronized (tasks) {
tasks.remove(task);
if (tasks.isEmpty()) {
tasks.notifyAll();
}
}
}
}
public void add(Callable<T> task) {
synchronized (tasks) {
addsPending++;
}
executor.submit(new Callable<Void>() {
#Override
public Void call()
throws InterruptedException,
ExecutionException {
addWhenAllowed(task);
return null;
}
});
}
public Collection<T> getAllResults()
throws InterruptedException {
synchronized (tasks) {
while (addsPending > 0 || !tasks.isEmpty()) {
tasks.wait();
}
return new ArrayList<T>(results);
}
}
public void shutdown() {
executor.shutdown();
}
}
I suspect it could be done more cleanly using Locks and Conditions instead of synchronization.

Use Catch with a timer to return to the function

I am using the GitHub API, but it throws an exception because of the rate limit.
I use catch to catch this exception, but I want to use a timer count 1 hour and continue the function from where it stopped.
try{
// my function
}
catch (NoSuchPageException e ) {
Thread.sleep(3600 * 1000);
System.out.println("Waiting the timer");
}
How can I continue my function after the timer ends?

I suppose you can use a while loop.
boolean continueFunc = false;
while (!continueFunc)
try{
//my function
continueFunc = true;
}
catch (NoSuchPageException e ){
Thread.sleep(3600 * 1000);
System.out.println("Waiting the timer");
}
}

Your question is unclear but if you wish to retrieve html content but continue if you content is not there 1 hour later after calling the function you can use futures:
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.FutureTask;
import java.util.concurrent.TimeUnit;
public class FutureExample {
static class RetrieveHTMLThingy implements Callable<String> {
#Override
public String call() throws Exception {
System.out.println("Getting HTML Stuff");
Thread.sleep(2000);
return "html stuff";
}
}
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(2);
FutureTask<String> getHTML = new FutureTask<String>(new RetrieveHTMLThingy());
executor.execute(getHTML);
try {
System.out.println(getHTML.get(1, TimeUnit.HOURS));
} catch (Exception e) {
e.printStackTrace();
}
}
}

Timeout for a method

My Program looks like below
Main Program (Thread 1)
Create multiple simple java threads (Thead 1.1, 1.2...)
In each thread(1.1 or 1.2..) I'm doing some processing also calling one method which is
sometimes is not responding(CORBA calls). I want to define timer for
this method and thread(1.1 or 1.2 whoever is calling) should wait there itself till I get response or timer expired.
I have written following sample program. I don't think this is the right approach. Is there any better approach? In this prg I'm not sure when the interupt method is invoked.
public class MethodTimeout implements Runnable{
/**
* #param args
*/
public Thread t1 = null;
public int threadnum = 0;
public static void main(String[] args) {
for (int i=0; i<3; i++){
MethodTimeout mt =new MethodTimeout();
Thread t = new Thread(mt,"thread "+(i+1));
mt.t1 = t;
mt.threadnum = (i+1);
t.start();
}
System.out.println("stmt after execution");
}
public Object testTimeout(){
long startTime = System.currentTimeMillis();
try {
System.out.println("in side method start "+t1.getName()+" start time"+startTime);
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
long endtime = System.currentTimeMillis();
System.out.println("in side method end "+t1.getName()+" total time"+(endtime-startTime) );
return null;
}
#Override
public void run() {
Thread timeout = new Thread (){
public void run() {
testTimeout();
};
};
timeout.start();
try {
Thread.sleep(2000);
timeout.interrupt();
Thread.sleep(2000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println(t1.getName() + " is ending");
}
}

This very much sounds like you should implement Callable. This is just an example
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
public class Test {
public static void main(String[] args) throws Exception {
ExecutorService service = Executors.newFixedThreadPool(2);
Future<String> futureResult = service.submit(new MyCall());
try{
String result = futureResult.get(20, TimeUnit.MILLISECONDS);
} catch(TimeoutException timeout){
System.out.println("Timeout");
service.shutdownNow();
}
}
static class MyCall implements Callable<String> {
#Override
public String call() throws Exception {
try{
//Simulate some corba work
Thread.sleep(1000);
}catch(InterruptedException e){
Thread.currentThread().interrupt();
System.out.println("Shutting down the task!");
}
return "The result";
}
}
}

You can also make one minor change to #Eugene's answer, that is instead of calling the shutdownNow() on the ExecutorService itself you can just call cancel(true) on the futureResult that timed out. Here is the code snippet:
public class Test {
public static void main(String[] args) throws Exception {
ExecutorService service = Executors.newFixedThreadPool(2);
Future<String> futureResult = service.submit(new MyCall());
try{
String result = futureResult.get(20, TimeUnit.MILLISECONDS);
} catch(TimeoutException timeout){
System.out.println("Timeout");
} finally {
futureResult.cancel(true);
}
}
This is just to ensure that only the timed out thread is cancelled. As the shutdownNow() prevents waiting tasks from starting in addition to attempting to stop currently executing ones.