Develop Reference

How to make a block of code execute without interruption Java?

I'm trying to create a web crawler.
I've created a class to handle all URLs visited and to visit.
This class has to be accessed by multiple threads for retrieving and updating those lists.
The problem I'm facing, or at least I think, is in nextRandom() and probably also in next(). I think what is happening is the threads are interfering with each other since the function is somewhat synchronized but not atomic. Is there a way to make so this block of code is executed without any interruption by other threads?
The URL handler
import java.util.*;
import java.util.concurrent.ThreadLocalRandom;
public class UrlHandler {
private volatile Set<String> visited = new HashSet<String>();
private volatile List<String> toVisit = new ArrayList<String>();
public void addToVisit(String url) {
synchronized (this){
if (!visited.contains(url)) toVisit.add(url);
}
}
public void addToVisit(Collection<String> urls) {
synchronized (this){
for (String url : urls)
if (!visited.contains(url)) toVisit.add(url);
}
}
public void addVisited(String url){
synchronized (this){
visited.add(url);
}
}
public void addVisited(Collection<String> urls){
synchronized (this){
visited.addAll(urls);
}
}
public String next() {
while (toVisit.size() == 0) {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
synchronized (this){
String url = toVisit.get(0);
toVisit.remove(0);
return url;
}
}
public String nextRandom() {
synchronized (this){
int n = 0;
if (toVisit.size() > 1){
n = ThreadLocalRandom.current().nextInt(toVisit.size());
}
String url = toVisit.get(n);
toVisit.remove(n);
return url;
}
}
public List<String> getToVisit() {
synchronized (this){
return toVisit;
}
}
public Set<String> getVisited() {
synchronized (this){
return visited;
}
}
}
Web Crawler
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class WebCrawler {
private final ExecutorService executor;
public WebCrawler(int nOfThreads) {
this.executor = Executors.newFixedThreadPool(nOfThreads);
}
public void add(Runnable runnable) {
this.executor.execute(runnable);
}
//Used to shut down safely and wait also 5 of seconds for not finished tasks
public void shutdown() {
this.executor.shutdown();
try {
this.executor.awaitTermination(5, TimeUnit.SECONDS);
if (!this.executor.isTerminated()) {
System.err.println("Timed out waiting for executor to terminate cleanly. Shutting down.");
this.executor.shutdownNow();
}
} catch (final InterruptedException e) {
System.err.println("Interrupted while waiting for executor shutdown.");
Thread.currentThread().interrupt();
}
}
}
Failing test example
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class UrlHandlerTest {
List<String> testList = new ArrayList<>(List.of("test1", "test2", "test3", "test3"));
List<String> uniqueTestList = new ArrayList<>(List.of("test1", "test2", "test3"));
UrlHandler urlHandler = new UrlHandler();
#Test
public void concurrentAccess(){
urlHandler.addToVisit(testList);
WebCrawler webCrawler = new WebCrawler(10);
for (int i = 0; i < urlHandler.getToVisit().size(); i++) {
webCrawler.add(new Runnable() {
#Override
public void run() {
String url = urlHandler.nextRandom();
urlHandler.addVisited(url);
System.out.println("Here thread " + Thread.currentThread().getId() + " working on: " + url);
}
});
}
webCrawler.shutdown();
System.out.println(urlHandler.getVisited());
assertEquals(true, urlHandler.getVisited().containsAll(uniqueTestList));
}
}

In the next method this code is a problem:
while (toVisit.size() == 0) {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
The lock isn't held for this part, so size can be stale. Instead of this, try something like
while (toVisit.size() == 0)
wait();
Do this in a synchronized block so you have the lock held while checking the collection size. Code that adds to the collection should notify in order to wake up the waiting threads.

This piece of code is problematic:
for (int i = 0; i < urlHandler.getToVisit().size(); i++) {
webCrawler.add(new Runnable() {
// ...
});
}
The urlHandler.getToVisit().size() is always changing during the traversal, and there is uncertainty (because the size will be changed asynchronously).
Change to:
int size = urlHandler.getToVisit().size();
for (int i = 0; i < size; i++) {
webCrawler.add(new Runnable() {
// ...
});
}

Java Semaphore, acquire() and release()

I am a beginner to java and I was experiment with Semaphore.
I tried to write a code that has writers and reader, I only tried to use acquire() and release():
1) If a writer is writing, then at the same time no other writer can write and no reader can read.
2) Multiple readers can read at the same time, but if there is at least one active reader then writers can't write anything.
So, in summary, there can be either
- one reader and no writer
- multiple readers and no writer
- one writer and no reader
I tried to write the code below, I know it is probably horribly bad but I really struggle to understand the concept and the code is not working and I don't know how to fix it.
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Semaphore;
public class Main
{
public static void main(String [] args)
{
ExecutorService executorService = Executors.newCachedThreadPool();
ReadWriteLock RW = new ReadWriteLock();
executorService.execute(new Writer(RW));
executorService.execute(new Writer(RW));
executorService.execute(new Writer(RW));
executorService.execute(new Writer(RW));
executorService.execute(new Reader(RW));
executorService.execute(new Reader(RW));
executorService.execute(new Reader(RW));
executorService.execute(new Reader(RW));
}
}
class ReadWriteLock
{
private int reader = 0;
private Semaphore write = new Semaphore(1);
public void readLock()
{
if (write.availablePermits() == 1)
{
reader++;
}
}
public void writeLock()
{
if (write.availablePermits() == 1)
{
try
{
write.acquire();
}
catch(InterruptedException e)
{
e.printStackTrace();;
}
}
}
public void readUnLock()
{
if (reader > 0)
reader--;
}
public void writeUnLock()
{
if (write.availablePermits() == 0)
{
write.release();
}
}
}
class Writer implements Runnable
{
private ReadWriteLock RW_lock;
public Writer(ReadWriteLock rw) {
RW_lock = rw;
}
public void run() {
while (true){
RW_lock.writeLock();
RW_lock.writeUnLock();
}
}
}
class Reader implements Runnable
{
private ReadWriteLock RW_lock;
public Reader(ReadWriteLock rw) {
RW_lock = rw;
}
public void run() {
while (true){
RW_lock.readLock();
RW_lock.readUnLock();
}
}
}

You can follow the following solution to the reader writer problem using semaphores. It takes care of all the mentioned conditions and uses the java.util.concurrent.Semaphore package to implement semaphores.
http://www2.hawaii.edu/~walbritt/ics240/synchronization/ReaderWriterSolution.java
You have to define locks on the reader and writer. Following is an interface for all the reader lock.
interface RWLock{
public abstract void acquireReadLock(int readerNum);
public abstract void acquireWriteLock(int writerNum);
public abstract void releaseReadLock(int readerNum);
public abstract void releaseWriteLock(int writerNum);
}
Database.java holds the definition of all the read and write function:
class Database implements RWLock{
private int readerCount; // the number of active readers
private Semaphore mutex; // controls access to readerCount
private Semaphore db; // controls access to the database
public Database() {
readerCount = 0;
mutex = new Semaphore(1);
db = new Semaphore(1);
}
public void acquireReadLock(int readerNum) {
try{
//mutual exclusion for readerCount
mutex.acquire();
}
catch (InterruptedException e) {}
++readerCount;
// if I am the first reader tell all others
// that the database is being read
if (readerCount == 1){
try{
db.acquire();
}
catch (InterruptedException e) {}
}
System.out.println("Reader " + readerNum + " is reading. Reader count = " + readerCount);
//mutual exclusion for readerCount
mutex.release();
}
public void releaseReadLock(int readerNum) {
try{
//mutual exclusion for readerCount
mutex.acquire();
}
catch (InterruptedException e) {}
--readerCount;
// if I am the last reader tell all others
// that the database is no longer being read
if (readerCount == 0){
db.release();
}
System.out.println("Reader " + readerNum + " is done reading. Reader count = " + readerCount);
//mutual exclusion for readerCount
mutex.release();
}
public void acquireWriteLock(int writerNum) {
try{
db.acquire();
}
catch (InterruptedException e) {}
System.out.println("Writer " + writerNum + " is writing.");
}
public void releaseWriteLock(int writerNum) {
System.out.println("Writer " + writerNum + " is done writing.");
db.release();
}
}
Now you will need to implement the reader and writer respectively. By acquiring the locks and following the mentioned conditions:
one reader and no writer
multiple readers and no writer
one writer and no reader
class Reader implements Runnable
{
private RWLock database;
private int readerNum;
public Reader(int readerNum, RWLock database) {
this.readerNum = readerNum;
this.database = database;
}
public void run() {
while (true) {
SleepUtilities.nap();
System.out.println("reader " + readerNum + " wants to read.");
database.acquireReadLock(readerNum);
// you have access to read from the database
// let's read for awhile .....
SleepUtilities.nap();
database.releaseReadLock(readerNum);
}
}
;
}
class Writer implements Runnable
{
private RWLock database;
private int writerNum;
public Writer(int w, RWLock d) {
writerNum = w;
database = d;
}
public void run() {
while (true){
SleepUtilities.nap();
System.out.println("writer " + writerNum + " wants to write.");
database.acquireWriteLock(writerNum);
// you have access to write to the database
// write for awhile ...
SleepUtilities.nap();
database.releaseWriteLock(writerNum);
}
}
}
Finally you need a running class to test the solution
import java.util.concurrent.Semaphore;
public class ReaderWriterSolution{
public static final int NUM_OF_READERS = 3;
public static final int NUM_OF_WRITERS = 2;
public static void main(String args[]){
RWLock database = new Database();
Thread[] readerArray = new Thread[NUM_OF_READERS];
Thread[] writerArray = new Thread[NUM_OF_WRITERS];
for (int i = 0; i < NUM_OF_READERS; i++) {
readerArray[i] = new Thread(new Reader(i, database));
readerArray[i].start();
}
for (int i = 0; i < NUM_OF_WRITERS; i++) {
writerArray[i] = new Thread(new Writer(i, database));
writerArray[i].start();
}
}
}
All the classes have been included in the solution.
Hope it helps!

How to make 2 threads share data? (Java)

I'm currently practicing Threads so I tasked myself to write a program that will create 2 threads. The first one will endlessly print a character and the second one will endlessly wait for input and then pass it to the first Thread. Then Thread #1 should print the passed character. Here's what I wrote:
public class A extends Thread {
public char dif;
Scanner stdin = new Scanner(System.in);
#Override
public void run() {
for (; ; ) {
dif = stdin.nextLine().charAt(0);
MyThread.setCh(dif);
}
}
}
This thread takes input and then passes it to this one:
public class MyThread extends Thread {
public static char ch;
public static void setCh(char cha) {
ch = cha;
}
public static char getCh() {
return ch;
}
#Override
public void run() {
for(;;) {
try {
Thread.sleep(300);
}
catch(InterruptedException e) {
e.printStackTrace();
}
System.out.print(getCh());
}
}
}
And what happens in the main():
MyThread endless = new MyThread();
MyThread.setCh('$');
A set = new A();
endless.start();
set.start();
However, this doesn't work as intended. No matter what I type, the program keeps printing $. Also for some reason the first time I type a character I get an Out of bounds exception.

Probably, the easiest way to approach this, is to use BlockingQueue.
Effectively, in your example the thread, that receives character from System.in is producer and the thread that prints received character is consumer.
So, here is the code that achieves your goal:
import java.util.*;
import java.util.concurrent.*;
class Setup {
public static void main(String[] args) {
BlockingQueue<Character> q = new LinkedBlockingQueue<>();
Producer p = new Producer(q);
Consumer c = new Consumer(q);
new Thread(p).start();
new Thread(c).start();
}
}
class Producer implements Runnable {
private final BlockingQueue<Character> queue;
private final Scanner scanner = new Scanner(System.in);
Producer(BlockingQueue<Character> q) { queue = q; }
public void run() {
try {
while (true) {
queue.put(produce());
}
} catch (InterruptedException ex) {
Thread.currentThread().interrupt(); // set interrupt flag
} finally {
scanner.close();
}
}
Character produce() {
return scanner.nextLine().charAt(0);
}
}
class Consumer implements Runnable {
private final BlockingQueue<Character> queue;
Consumer(BlockingQueue<Character> q) { queue = q; }
public void run() {
try {
while (true) {
consume(queue.take());
}
} catch (InterruptedException ex) {}
}
void consume(Character c) {
System.out.println("Received character: " + c);
}
}

The problem actually you have it is quite small delay in Thread.sleep(300);
Try to set sleep for a few seconds Thread.sleep(5000);. You have to type something before it will print previous char

Multithread programming and incrementing a static variable

I have read almost all posts related to my problem but could not solve my problem. this code is a question in the Big Java - Early Object of Cay Horstmann. The question asks about counting the word of several files by using multithread programming and store the combined words counting which is my problem.
in order to get the combined counter I used a static variable which is incremented on each of iteration of counting words in threads. Also I used a ReentrantLock() to make incrementing available only for one thread at a time.
everything works fine except incrementing. it seems sometimes the static variable does not incremented. I tested the number of lock() and unlock()
by each thread and they match with my expected result however the static variable does not work properly.
is there any explanation for it? thank you for your time and help.
my task class:
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
public class WordCount implements Runnable
{
private String fileName;
Scanner inputFile;
private long counter;
public volatile static long combinedCounter = 0;
Lock ccLock;
public WordCount(String aName) throws FileNotFoundException
{
try
{
this.ccLock = new ReentrantLock();
this.counter = 0;
this.fileName = aName;
this.inputFile = new Scanner(new File(this.fileName));
}
catch (FileNotFoundException e)
{}
}
public void cCount()
{
ccLock.lock();
try
{
combinedCounter++;
/*synchronized (this)
{
combinedCounter++;
}*/
}
finally
{
ccLock.unlock();
}
}
#Override
public void run()
{
try
{
while (inputFile.hasNext() && !Thread.interrupted())
{
synchronized (this)
{
cCount();
}
counter++;
inputFile.next();
Thread.sleep(0);
}
System.out.printf("%s: %d\t\t%d\n", this.fileName,
this.counter,combinedCounter);
}
catch (InterruptedException e)
{}
}
}
This is my client class:
import java.io.FileNotFoundException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class WordCountRunner
{
public static void main(String[] args) throws FileNotFoundException,
InterruptedException
{
String a = "a.txt";
String b = "b.txt";
ExecutorService pool = Executors.newFixedThreadPool(2);
;
try
{
Runnable r1 = new WordCount(a);
Runnable r2 = new WordCount(b);
pool.execute(r1);
pool.execute(r2);
while (!pool.isTerminated())
{
pool.shutdown();
}
Thread.sleep(100);
System.out.print("***" + WordCount.combinedCounter);
}
catch (FileNotFoundException e)
{
}
finally
{
pool.shutdown();
}
}
}

The lock does not work because the ReentrantLock is an instance variable in the WordCount class. So, each instance of that class has its own private lock and they don't synchronize with each other. The easiest change would be to make the lock static, like the variable it's protecting.

Each of your Runnables has its own lock object. For your strategy to work, they all need to share exactly one lock.
For example,
// ...
static Lock ccLock = new ReentrantLock();
public WordCount(String aName) throws FileNotFoundException
{
try
{
// this.ccLock = new ReentrantLock();
this.counter = 0;
this.fileName = aName;
this.inputFile = new Scanner(new File(this.fileName));
// ...
}

how to restart a thread

I tried to write a file monitor which will check the file if a new line is appended,the monitor in fact is a thread which will read the line by a randomaccessfile all the time.
This is the monitor core codes:
public class Monitor {
public static Logger log = Logger.getLogger(Monitor.class);
public static final Monitor instance = new Monitor();
private static final ArrayList<Listener> registers = new ArrayList<Listener>();
private Runnable task = new MonitorTask();
private Thread monitorThread = new Thread(task);
private boolean beStart = true;
private static RandomAccessFile raf = null;
private File monitoredFile = null;
private long lastPos;
public void register(File f, Listener listener) {
this.monitoredFile = f;
registers.add(listener);
monitorThread.start();
}
public void replaceFile(File newFileToBeMonitored) {
this.monitoredFile = newFileToBeMonitored;
// here,how to restart the monitorThread?
}
private void setRandomFile() {
if (!monitoredFile.exists()) {
log.warn("File [" + monitoredFile.getAbsolutePath()
+ "] not exist,will try again after 30 seconds");
try {
Thread.sleep(30 * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
setRandomFile();
return;
}
try {
if (raf != null) {
raf.close();
lastPos = 0;
}
raf = new RandomAccessFile(monitoredFile, "r");
log.info("monitor file " + monitoredFile.getAbsolutePath());
} catch (FileNotFoundException e) {
// The file must exist now
} catch (IOException e) {}
}
private void startRead() {
beStart = true;
String line;
while (beStart) {
try {
raf.seek(lastPos);
while ((line = raf.readLine()) != null) {
fireEvent(new FileEvent(monitoredFile.getAbsolutePath(),
line));
}
lastPos = raf.getFilePointer();
} catch (IOException e1) {}
}
}
private void stopRead() {
this.beStart = false;
}
private void fireEvent(FileEvent event) {
for (Listener lis : registers) {
lis.lineAppended(event);
}
}
private class MonitorTask implements Runnable {
#Override
public void run() {
stopRead();
//why putting the resetReandomAccessFile in this thread method is that it will sleep if the file not exist.
setRandomFile();
startRead();
}
}
}
This is some help classes:
public interface Listener {
void lineAppended(FileEvent event);
}
public class FileEvent {
private String line;
private String source;
public FileEvent(String filepath, String addedLine) {
this.line = addedLine;
this.source = filepath;
}
//getter and setter
}
And this is a example to call the monitor:
public class Client implements Listener {
private static File f = new File("D:/ab.txt");
public static void main(String[] args) {
Monitor.instance.register(f, new Client());
System.out.println(" I am done in the main method");
try {
Thread.sleep(5000);
Monitor.instance.replaceFile(new File("D:/new.txt"));
} catch (InterruptedException e) {
System.out.println(e.getMessage());
}
}
#Override
public void lineAppended(FileEvent event) {
String line = event.getLine();
if (line.length() <= 0)
return;
System.err.println("found in listener:" + line + ":" + line.length());
}
}
Now,my probelm is the code work well if I just call:
Monitor.instance.register(file,listener);
This will monitor the file for line appending,and will notify the listener.
However it does not work when I call the :
Monitor.instance.replaceFile(anotherfile);
This means I want to monitor another file rather than before.
So in my Monitor I have to restart the thread,how to make it?
I have tried the:
monitorThread.interruppt();
It does not wrok.
Anyone can fix it for me or tell me how to do ?
Thanks.
Before I ask,I have googling the "restart java thread",so I know one can not restart a dead thread,but my thread does not return,so I think it can be restarted.

You don't restart a Thread, instead you create a new one each time you want to start a thread.
A better alternative may be to use Executors.newCachedThreadPool() which gives you a pool of thread which will be started/recycle for you.
BTW: You are using recursion rather than a loop to poll if the file exists. Using recursion can mean if you wait too long it will throw a StackOverflowError. IMHO you shouldn't wait at all, the polling thread should repeatedly attempt to open the file until it is told to stop (or the file appears)
Your current implementation also means if the file is replaced, you will have to reopen the file in the background thread anyway.

Instead of explaining, I just coded up a skeleton example. I did not test it terribly well, but it may be of some use.
In order to monitor a(nother) file, just create a new Monitor, passing it a ScheduledExecutorService. Starting and stopping monitoring is straightforward. You can (should) reuse the same executor for multiple monitors.
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public interface Event
{
}
public interface Listener
{
void handle(Event event);
}
public class Monitor
{
private static final int CHECK_EVERY_SECONDS = 10;
private static final int RECHECK_AFTER_IF_NOT_EXISTS_SECONDS = 30;
private File file;
private ScheduledExecutorService executor;
private boolean active;
private List<Listener> listeners;
public Monitor(File file, ScheduledExecutorService executor)
{
super();
this.file = file;
this.executor = executor;
listeners = new ArrayList<Listener>();
}
public synchronized void start()
{
if (active)
{
return;
}
active = true;
executor.execute(new Runnable()
{
public void run()
{
synchronized (Monitor.this)
{
if (!active)
{
System.out.println("not active");
return;
}
}
if (!file.exists())
{
System.out.println("does not exist, rescheduled");
executor.schedule(this, RECHECK_AFTER_IF_NOT_EXISTS_SECONDS, TimeUnit.SECONDS);
return;
}
Event event = doStuff(file);
System.out.println("generated " + event);
updateListeners(event);
System.out.println("updated listeners and rescheduled");
executor.schedule(this, CHECK_EVERY_SECONDS, TimeUnit.SECONDS);
}
});
}
private Event doStuff(final File file)
{
return new Event()
{
public String toString()
{
return "event for " + file;
}
};
}
public synchronized void stop()
{
active = false;
}
public void addListener(Listener listener)
{
synchronized (listeners)
{
listeners.add(listener);
}
}
public void removeListener(Listener listener)
{
synchronized (listeners)
{
listeners.remove(listener);
}
}
private void updateListeners(Event event)
{
synchronized (listeners)
{
for (Listener listener : listeners)
{
listener.handle(event);
}
}
}
public static void main(String[] args) throws IOException
{
ScheduledExecutorService executor = Executors.newScheduledThreadPool(4);
File file = new File("test.png");
Monitor monitor = new Monitor(file, executor);
monitor.addListener(new Listener()
{
public void handle(Event event)
{
System.out.println("handling " + event);
}
});
monitor.start();
System.out.println("started...");
System.in.read();
monitor.stop();
System.out.println("done");
executor.shutdown();
}
}

See this post How to start/stop/restart a thread in Java?

I assume you answered your question
one can not restart a dead thread
This link may be helpful to you How to restart thread in java?

A thread in Java cannot be re-started. Every time you need to restart the thread you must make a new one.
That said, you might want to look at:
private void setRandomFile() {
if (!monitoredFile.exists()) {
log.warn("File [" + monitoredFile.getAbsolutePath()
+ "] not exist,will try again after 30 seconds");
try {
Thread.sleep(30 * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
setRandomFile();
return;
}
// ....
}
Here you sleep for 30 seconds if the file does not exist, then recursively call the same function. Now, I don't know what business requirements you have, but if this recursion ran long enough you will run out of stack space. Perhaps you will be better served with a while loop or even better, a little synchronisation like a Semaphore.