I want to write a program that can calculate the size of a directory and its subdirectories with multi-thread.
I write this:
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.Semaphore;
import java.util.concurrent.ThreadPoolExecutor;
public class Innerclass {
Semaphore semaphore = new Semaphore(1);
private List<String> availableConnections = new ArrayList();
public Innerclass(){
this.availableConnections.add("A");
}
public String acquireConnection() throws InterruptedException {
semaphore.acquire();
System.out.println("Acquiring connection " + Thread.currentThread().getName());
return availableConnections.remove(0);
}
static Long count = 0L;
static File file1 ;
public static void main(String[] args) {
System.out.println(countFilesInDirectory(new File("target directory address")));
}
public static Long countFilesInDirectory(File directory) {
Innerclass connection = new Innerclass();
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(20);
Task task1 = new Task();
//Long count = 0L;
for (File file : directory.listFiles()) {
if (file.isFile()) {
executor.execute(new Runnable() {
#Override
public void run() {
try {
String far = connection.acquireConnection();
count += printFileSizeNIO(String.valueOf(file));
connection.acquireConnection();
} catch (InterruptedException e) {
e.printStackTrace();
}
//System.out.println(printFileSizeNIO(String.valueOf(file)));
}
});
}
if (file.isDirectory()) {
count += countFilesInDirectory(file);
}
}
executor.shutdown();
//System.out.println(task1.getCount());
//return task1.getCount();
return count;
}
public static Long printFileSizeNIO(String fileName) {
Path path = Paths.get(fileName);
Long bytes = 0L;
try {
bytes = Files.size(path);
} catch (IOException e) {
e.printStackTrace();
}
return bytes;
}
}
This program gives results close to reality but can not calculate exactly. What do you think is the problem?
and with single thread this program work exactly!
Receives the current path and the number of fragments from the user and performs the calculation. Do you think there is another way besides the thread pool to write this program. Thank you, dear professors.
With executor service, your main thread wont wait for threads inside pool to finish.
See here:
Wait main thread until all the thread pools task complete of ExecutorService?
I have updated your code a little bit. It will give the same answer always.
public class TestSF
{
Semaphore semaphore = new Semaphore(1);
public void acquireConnection() throws InterruptedException
{
semaphore.acquire();
}
public void releaseConnection() throws InterruptedException
{
semaphore.release();
}
static AtomicLong count = new AtomicLong(0);
public static void main(String[] args)
{
long bytes = countFilesInDirectory(new File("C:\\Users\\ashish\\Desktop\\Loader"));
System.out.println(humanReadableByteCountBin(bytes));
}
public static Long countFilesInDirectory(File directory)
{
TestSF connection = new TestSF();
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(20);
for (File file : Objects.requireNonNull(directory.listFiles()))
{
executor.execute(() -> {
if (file.isFile())
{
try
{
connection.acquireConnection();
count.addAndGet(printFileSizeNIO(String.valueOf(file)));
connection.releaseConnection();
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
if (file.isDirectory())
{
countFilesInDirectory(file);
}
});
}
executor.shutdown();
try {
if (!executor.awaitTermination(60, TimeUnit.SECONDS)) {
executor.shutdownNow();
}
} catch (InterruptedException ex) {
executor.shutdownNow();
Thread.currentThread().interrupt();
}
return count.get();
}
public static Long printFileSizeNIO(String fileName)
{
Path path = Paths.get(fileName);
long bytes = 0L;
try
{
bytes = Files.size(path);
}
catch (IOException e)
{
e.printStackTrace();
}
return bytes;
}
public static String humanReadableByteCountBin(long bytes)
{
long absB = bytes == Long.MIN_VALUE ? Long.MAX_VALUE : Math.abs(bytes);
if (absB < 1024) {
return bytes + " B";
}
long value = absB;
CharacterIterator ci = new StringCharacterIterator("KMGTPE");
for (int i = 40; i >= 0 && absB > 0xfffccccccccccccL >> i; i -= 10)
{
value >>= 10;
ci.next();
}
value *= Long.signum(bytes);
return String.format("%.1f %ciB", value / 1024.0, ci.current());
}
}
Related
I'm trying to create a web crawler.
I've created a class to handle all URLs visited and to visit.
This class has to be accessed by multiple threads for retrieving and updating those lists.
The problem I'm facing, or at least I think, is in nextRandom() and probably also in next(). I think what is happening is the threads are interfering with each other since the function is somewhat synchronized but not atomic. Is there a way to make so this block of code is executed without any interruption by other threads?
The URL handler
import java.util.*;
import java.util.concurrent.ThreadLocalRandom;
public class UrlHandler {
private volatile Set<String> visited = new HashSet<String>();
private volatile List<String> toVisit = new ArrayList<String>();
public void addToVisit(String url) {
synchronized (this){
if (!visited.contains(url)) toVisit.add(url);
}
}
public void addToVisit(Collection<String> urls) {
synchronized (this){
for (String url : urls)
if (!visited.contains(url)) toVisit.add(url);
}
}
public void addVisited(String url){
synchronized (this){
visited.add(url);
}
}
public void addVisited(Collection<String> urls){
synchronized (this){
visited.addAll(urls);
}
}
public String next() {
while (toVisit.size() == 0) {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
synchronized (this){
String url = toVisit.get(0);
toVisit.remove(0);
return url;
}
}
public String nextRandom() {
synchronized (this){
int n = 0;
if (toVisit.size() > 1){
n = ThreadLocalRandom.current().nextInt(toVisit.size());
}
String url = toVisit.get(n);
toVisit.remove(n);
return url;
}
}
public List<String> getToVisit() {
synchronized (this){
return toVisit;
}
}
public Set<String> getVisited() {
synchronized (this){
return visited;
}
}
}
Web Crawler
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class WebCrawler {
private final ExecutorService executor;
public WebCrawler(int nOfThreads) {
this.executor = Executors.newFixedThreadPool(nOfThreads);
}
public void add(Runnable runnable) {
this.executor.execute(runnable);
}
//Used to shut down safely and wait also 5 of seconds for not finished tasks
public void shutdown() {
this.executor.shutdown();
try {
this.executor.awaitTermination(5, TimeUnit.SECONDS);
if (!this.executor.isTerminated()) {
System.err.println("Timed out waiting for executor to terminate cleanly. Shutting down.");
this.executor.shutdownNow();
}
} catch (final InterruptedException e) {
System.err.println("Interrupted while waiting for executor shutdown.");
Thread.currentThread().interrupt();
}
}
}
Failing test example
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class UrlHandlerTest {
List<String> testList = new ArrayList<>(List.of("test1", "test2", "test3", "test3"));
List<String> uniqueTestList = new ArrayList<>(List.of("test1", "test2", "test3"));
UrlHandler urlHandler = new UrlHandler();
#Test
public void concurrentAccess(){
urlHandler.addToVisit(testList);
WebCrawler webCrawler = new WebCrawler(10);
for (int i = 0; i < urlHandler.getToVisit().size(); i++) {
webCrawler.add(new Runnable() {
#Override
public void run() {
String url = urlHandler.nextRandom();
urlHandler.addVisited(url);
System.out.println("Here thread " + Thread.currentThread().getId() + " working on: " + url);
}
});
}
webCrawler.shutdown();
System.out.println(urlHandler.getVisited());
assertEquals(true, urlHandler.getVisited().containsAll(uniqueTestList));
}
}
In the next method this code is a problem:
while (toVisit.size() == 0) {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
The lock isn't held for this part, so size can be stale. Instead of this, try something like
while (toVisit.size() == 0)
wait();
Do this in a synchronized block so you have the lock held while checking the collection size. Code that adds to the collection should notify in order to wake up the waiting threads.
This piece of code is problematic:
for (int i = 0; i < urlHandler.getToVisit().size(); i++) {
webCrawler.add(new Runnable() {
// ...
});
}
The urlHandler.getToVisit().size() is always changing during the traversal, and there is uncertainty (because the size will be changed asynchronously).
Change to:
int size = urlHandler.getToVisit().size();
for (int i = 0; i < size; i++) {
webCrawler.add(new Runnable() {
// ...
});
}
I am reading two text files concurrently line by line.
What I am specifically want to do is when the lineCount on each thread are the same I want to take a look at the string that the scanner is currently reading.
I looked around for certain pattern I can implement like Compare and Swap and Slipped Condition but I cannot wrap my head around how it would help me achieve my goal. I am new to concurrency programming.
What I have managed so far is to synchronize the string reading and printing with counterSync method and I know that I have carry out my thread lock/pause operation there and take a look at the string.
public class concurrencyTest {
public static void main(String[] args) throws IOException {
String filePath1 = "path1.txt";
String filePath2 = "path2.txt";
reader reader = new reader();
MyThread source = new MyThread(reader, filePath1);
MyThread target = new MyThread(reader, filePath2);
source.start();
target.start();
}
static public class reader {
void read(String filePath) throws IOException {
readFile(filePath);
}
}
static synchronized void counterSync(String thread) {
System.out.println(thread);
}
static class MyThread extends Thread {
reader reader;
String filePath;
MyThread(reader reader, String filePath) {
this.reader = reader;
this.filePath = filePath;
}
#Override
public void run() {
try {
reader.read(filePath);
} catch (IOException e) {
e.printStackTrace();
}
}
}
static void readFile(String filePath) throws IOException {
FileInputStream inputStream = null;
Scanner sc = null;
int lineCount = 0;
try {
inputStream = new FileInputStream(filePath);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
lineCount++;
System.out.println(lineCount + "===" + sc.nextLine());
counterSync(sc.nextLine());
}
if (sc.ioException() != null) {
throw sc.ioException();
}
} finally {
if (inputStream != null) {
inputStream.close();
}
if (sc != null) {
sc.close();
}
}
}
}
Ok, what you are looking for is a little bit complex but still possible.
Your question lacks of some examples so correct me if I'm wrong in something.
You have 2 threads:
thread1
thread2
and 2 files:
file1
file2
Content of file1:
file1
file2
file3
file4
file5
file6
file7
file8
file9
Content of file2:
file11
file22
file33
file44
file55
file66
file77
file88
file99
You want to stop all threads on the same line numbers and do some oeration with the output.
This is the thread implementation for reading the files, we will instantiate 2 instance of it, each instance will manage a file.
static class ReaderThread extends Thread {
private File fileToRead;
public final Object lock = new Object();
private String currentLine;
private AtomicInteger lineCount = new AtomicInteger(0);
public ReaderThread(File fileToRead) {
this.fileToRead = fileToRead;
}
#Override
public void run() {
synchronized (lock) {
try {
Stream<String> lines = Files.lines(Path.of(fileToRead.getPath()));
lines.forEach(line -> {
currentLine = line;
// Here's your logic on different lines
if (lineCount.get() == 4 || lineCount.get() == 5 || lineCount.get() == 6) {
try {
lock.wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
lineCount.getAndIncrement();
});
} catch (IOException e) {
e.printStackTrace();
}
}
}
public String getCurrentLine() {
return currentLine;
}
public boolean isLocked() {
return getState().equals(State.WAITING);
}
}
Then we will use an helper thread to notify the reader threads when our elboration will be ok:
static class HelperThread extends Thread {
private List<ReaderThread> threads;
#Override
public void run() {
while (true) {
if (threads.stream().allMatch(ReaderThread::isLocked)) {
System.out.println("next line:");
threads.forEach(thread -> {
synchronized (thread.lock) {
System.out.println(thread.getCurrentLine());
thread.lock.notify();
}
});
System.out.println("\n");
}
}
}
public HelperThread(List<ReaderThread> threads) {
this.threads = threads;
}
}
Finally the main class for testing all:
public static void main(String[] args) {
File f1 = new File(Objects.requireNonNull(Main.class.getClassLoader().getResource("file1.txt")).getFile());
File f2 = new File(Objects.requireNonNull(Main.class.getClassLoader().getResource("file2.txt")).getFile());
ReaderThread t1 = new ReaderThread(f1);
ReaderThread t2 = new ReaderThread(f2);
HelperThread helperThread = new HelperThread(List.of(t1, t2));
helperThread.start();
t1.start();
t2.start();
}
Executing the program will result in this output:
next line:
file5
file55
next line:
file6
file66
next line:
file7
file77
Here's the complete list of imports:
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.Objects;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Stream;
Please note: this is a rude example, you need to manage with the correct shutdown of the threads, some modifiers are public so encapsulate it following the java guidelines, coorrectly manage all exceptions and do some general refactor.
If you want a more versatile implementation, to interpolate different lines, the following should be ok:
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.Objects;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Stream;
public class Main2 {
public static void main(String[] args) {
File f1 = new File(Objects.requireNonNull(Main2.class.getClassLoader().getResource("file1.txt")).getFile());
File f2 = new File(Objects.requireNonNull(Main2.class.getClassLoader().getResource("file2.txt")).getFile());
ReaderThread t1 = new ReaderThread(f1);
ReaderThread t2 = new ReaderThread(f2);
HelperThread helperThread = new HelperThread(List.of(t1, t2));
helperThread.start();
t1.setName("Reader1");
t1.setName("Reader2");
t1.start();
t2.start();
}
static class ReaderThread extends Thread {
private final File fileToRead;
private final Object lock = new Object();
private final AtomicInteger lineCount = new AtomicInteger(0);
private String currentLine;
public ReaderThread(File fileToRead) {
this.fileToRead = fileToRead;
}
#Override
public void run() {
synchronized (lock) {
try {
Stream<String> lines = Files.lines(Path.of(fileToRead.getPath()));
lines.forEach(line -> {
currentLine = line;
lineCount.getAndIncrement();
});
} catch (IOException e) {
e.printStackTrace();
}
}
}
public void lock() throws InterruptedException {
this.lock.wait();
}
public void unlock() {
this.lock.notify();
}
public boolean isLocked() {
return getState().equals(State.WAITING);
}
public Object getLock() {
return lock;
}
public AtomicInteger getLineCount() {
return lineCount;
}
public String getCurrentLine() {
return currentLine;
}
}
static class HelperThread extends Thread {
private List<ReaderThread> threads;
#Override
public void run() {
while (true) {
threads.forEach(t -> {
try {
if (t.getName().equals("Reader1") && t.getLineCount().get() == 3) t.lock();
if (t.getName().equals("Reader2") && t.getLineCount().get() == 4) t.lock();
} catch (InterruptedException e) {
e.printStackTrace();
}
});
if (threads.stream().allMatch(ReaderThread::isLocked)) {
System.out.println("next line:");
threads.forEach(t -> {
synchronized (t.getLock()) {
System.out.println(t.getCurrentLine());
t.unlock();
}
});
System.out.println("\n");
}
}
}
public HelperThread(List<ReaderThread> threads) {
this.threads = threads;
}
}
}
Be sure that the HelperThread starts before the other threads or it's possible to loose some data.
It seems that you didn't post a complete example. But, a few general comments:
You might be able to get away with using "compare-and-swap" logic for an integer, but you should not expect it to work for a more-sophisticated thing like a Java "String" or any sort of container.
You should simply use the synchronization-objects provided in the language. If you are going to update or even to examine a shared data structure, you must be holding the proper lock.
Of course, "thread-safe queues" are very helpful in many designs because they facilitate the most-common activity – message-passing – and allow the various threads to operate graciously at slightly-varying speeds. You still have to lock anything that's shared, but nonetheless it's a useful design that's really as old as the Unix® "pipe."
You can use
java.util.concurrent.CyclicBarrier
A synchronization aid that allows a set of threads to all wait for
each other to reach a common barrier point. CyclicBarriers are useful
in programs involving a fixed sized party of threads that must
occasionally wait for each other. The barrier is called cyclic because
it can be re-used after the waiting threads are released.
A CyclicBarrier supports an optional Runnable command that is run once
per barrier point, after the last thread in the party arrives, but
before any threads are released. This barrier action is useful for
updating shared-state before any of the parties continue.
Here is an example using this class:
public static void main(String[] args) throws IOException {
String filePath1 = "path1.txt";
String filePath2 = "path2.txt";
ReaderThread reader1 = new ReaderThread(filePath1);
ReaderThread reader2 = new ReaderThread(filePath2);
CyclicBarrier cyclicBarrier = new CyclicBarrier(2, () -> {
//processing when condition met in both thread
List<String> lines1 = reader1.getLines();
List<String> lines2 = reader2.getLines();
System.out.println(lines1.get(lines1.size() - 1) + " " + lines2.get(lines2.size()-1));
});
reader1.setCyclicBarrier(cyclicBarrier);
reader2.setCyclicBarrier(cyclicBarrier);
reader1.start();
reader2.start();
}
public static class ReaderThread extends Thread {
CyclicBarrier cyclicBarrier;
String file;
List<String> lines = new ArrayList<>();
public void setCyclicBarrier(CyclicBarrier cyclicBarrier) {
this.cyclicBarrier = cyclicBarrier;
}
public ReaderThread(String file) {
this.file = file;
}
#Override
public void run() {
try (BufferedReader reader = Files.newBufferedReader(Paths.get(file))) {
String line = null;
for (int i = 0; (line = reader.readLine()) != null; i++) {
lines.add(line);
//condition do something
if (i % 10 == 0) {
cyclicBarrier.await();
}
}
} catch (IOException | InterruptedException | BrokenBarrierException e) {
e.printStackTrace();
}
}
public List<String> getLines() {
return new ArrayList<>(lines);
}
}
And output:
this is from file1 1232123321312 this is from file 2 1232123321312
this is from file1 1232123321312 this is from file 2 1232123321312
this is from file1 1232123321312 this is from file 2 1232123321312
I have a simple code as below. This checks for alive status for a list of servers.
Could you please let me know how this can be done in parallel using threading or any other suitable solutions.
List<Host> hosts = this.getAllHosts();
List<Host> aliveHosts = new ArrayList<>();
if (hosts != null && hosts.size() > 0) {
for (Host host : hosts) {
try {
if(InetAddress.getByName(host.getIpaddress()).isReachable(TIMEOUT)) {
aliveHosts.add(host);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
return aliveHosts;
How can I call each getByName in a thread and execute this in parallel at the same time. Currently each of them is having a timeout of 3 seconds. If there are 10 items then the total time would be 30 seconds. Can anyone give a solution so that this can be done in 3-8 seconds overall.
With Java 8 streams:
List<Host> aliveHosts = hosts.stream()
.parallel()
.filter(h -> {
try {
return InetAddress.getByName(h.getIpaddress()).isReachable(TIMEOUT)
} catch(Exception e) {
return false;
}
})
.collect(Collectors.toList());
Let's consider this threading example:
public class SimpleThreads {
// Display a message, preceded by
// the name of the current thread
static void threadMessage(String message) {
String threadName =
Thread.currentThread().getName();
System.out.format("%s: %s%n",
threadName,
message);
}
private static class MessageLoop
implements Runnable {
public void run() {
String importantInfo[] = {
"Mares eat oats",
"Does eat oats",
"Little lambs eat ivy",
"A kid will eat ivy too"
};
try {
for (int i = 0;
i < importantInfo.length;
i++) {
// Pause for 4 seconds
Thread.sleep(4000);
// Print a message
threadMessage(importantInfo[i]);
}
} catch (InterruptedException e) {
threadMessage("I wasn't done!");
}
}
}
public static void main(String args[])
throws InterruptedException {
// Delay, in milliseconds before
// we interrupt MessageLoop
// thread (default one hour).
long patience = 1000 * 60 * 60;
// If command line argument
// present, gives patience
// in seconds.
if (args.length > 0) {
try {
patience = Long.parseLong(args[0]) * 1000;
} catch (NumberFormatException e) {
System.err.println("Argument must be an integer.");
System.exit(1);
}
}
threadMessage("Starting MessageLoop thread");
long startTime = System.currentTimeMillis();
Thread t = new Thread(new MessageLoop());
t.start();
threadMessage("Waiting for MessageLoop thread to finish");
// loop until MessageLoop
// thread exits
while (t.isAlive()) {
threadMessage("Still waiting...");
// Wait maximum of 1 second
// for MessageLoop thread
// to finish.
t.join(1000);
if (((System.currentTimeMillis() - startTime) > patience)
&& t.isAlive()) {
threadMessage("Tired of waiting!");
t.interrupt();
// Shouldn't be long now
// -- wait indefinitely
t.join();
}
}
threadMessage("Finally!");
}
}
Source.
In essence, you need a Runnable which is responsible for the way your threads will work. You will need to instantiate a Thread, passing an instance of the Runnable you have and then start your Thread. You will need to have all the Threads accessible and Join them. You can easily manage the timeout limits as well.
Non Java 8 way will look similar:
List<Host> hosts = this.getAllHosts();
Queue<Host> q = new ArrayBlockingQueue<>(hosts.size(), true, hosts);
ExecutorService ex = Executors.newFixedThreadPool(5);
List<Host> aliveHosts = Collections.synchronizedList(new ArrayList<>());
while(!q.isEmpty()){
ex.submit(new Runnable() {
#Override
public void run() {
Host host = q.poll();
try {
if(InetAddress.getByName(host.getIpaddress()).isReachable(TIMEOUT)) {
aliveHosts.add(host);
}
} catch (IOException e) {
e.printStackTrace();
}
}
});
}
ex.shutdown();
}
Java 8 and ExecutorService:
List<Host> hosts = this.getAllHosts();
List<Host> aliveHosts = Collections.synchronizedList(new ArrayList<Host>());
ExecutorService executorService = Executors.newFixedThreadPool(10);
if (hosts != null && hosts.size() > 0) {
for (Host host : hosts) {
executorService.submit(() -> {
try {
if (InetAddress.getByName(host.getIpaddress()).isReachable(TIMEOUT)) {
aliveHosts.add(host);
}
} catch (IOException e) {
// logger?
}
});
}
}
executorService.shutdown();
return aliveHosts;
In addition to the accepted Java8 answer you can actually control the level of concurrency quite easily by using a custom ForkJoinPool:
final Predicate<Host> isAlive = h -> {
try {
return InetAddress.getByName(h.getIpaddress()).isReachable(TIMEOUT);
} catch (Exception e) {
return false;
}
};
final Callable<List<Host>> collectAliveHosts = () ->
hosts.stream().parallel().filter(isAlive).collect(Collectors.toList());
final ForkJoinPool threadPool = new ForkJoinPool(4);
final List<Host> aliveHosts = threadPool.submit(collectAliveHosts).get();
If you don't use a custom pool, the common ForkJoinPool will be used, which is sized according to the number of cores/CPUs your current machine has. This pool is however used by the whole JVM. That is, if you submit long running tasks to the common pool, the whole application might suffer some performance degradation.
We can do parallelly using Future interface.
package test.basics;
import java.net.InetAddress;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class TestFutureTask {
private static final int TIMEOUT = 30000;
public static void main(String[] args) {
List<String> hosts = new ArrayList<String>();
hosts.add("127.0.0.1");
hosts.add("127.0.0.2");
hosts.add("127.0.0.3");
hosts.add("127.0.0.4");
hosts.add("127.0.0.5");
hosts.add("127.0.0.6");
List<String> aliveHosts = new ArrayList<>();
List<String> notAliveHosts = new ArrayList<>();
long stTime = System.currentTimeMillis();
System.out.println("Starting time " + stTime);
Map<String, Future> jobList = new HashMap<>();
ExecutorService newCachedThreadPool = Executors.newCachedThreadPool();
for (String host : hosts) {
Future f = newCachedThreadPool.submit(new Callable<Boolean>() {
private String host;
#Override
public Boolean call() throws Exception {
return InetAddress.getByName(host).isReachable(TIMEOUT);
}
public Callable<Boolean> init(String host) {
this.host = host;
return this;
}
}.init(host));
jobList.put(host, f);
}
for (String host : jobList.keySet()) {
try {
if ((boolean) jobList.get(host).get()) {
aliveHosts.add(host);
} else {
notAliveHosts.add(host);
}
} catch (InterruptedException | ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
long endTime = System.currentTimeMillis();
System.out.println("Ending time : " + endTime);
System.out.println("Time taken :" + (endTime - stTime));
System.out.println("Alive hosts: " + aliveHosts);
System.out.println("Not alive hosts: " + notAliveHosts);
}
}
Sample output:
Starting time 1500570979858
Ending time : 1500571009872
Time taken :30014
Alive hosts: [127.0.0.1]
Not alive hosts: [127.0.0.6, 127.0.0.5, 127.0.0.4, 127.0.0.3, 127.0.0.2]
I have a list of senders for them I have to parallely send mails individually.Currently I am iterating over the list construct the body (as it is different for different people) and then sending them. How can I use forkjoin for this. I tried using recusiveAction but I guess its only for recursive tasks.
All the examples available in internet are implemented with RecursiveAction. Is there any other class with which I can implement this.
ServiceExecutors work nicely for this. They come with Java.
import java.util.*;
import java.util.concurrent.*;
public class SendMailExample
{
public static void main(String[] args) throws Exception
{
ExecutorService executor = Executors.newFixedThreadPool(3);
Collection<Future> futures = new ArrayList<Future>();
futures.add(executor.submit(new Mailer("thread1")));
futures.add(executor.submit(new Mailer("thread2")));
futures.add(executor.submit(new Mailer("thread3")));
for (Future future : futures)
{
future.get();
}
executor.shutdown();
}
static class Mailer implements Runnable
{
private Object message;
public Mailer(Object message)
{
this.message = message;
}
public void run()
{
System.out.println("Sending message " + String.valueOf(message));
}
}
}
I browsed I got a better answer:
package Test1;
import java.util.*;
import java.util.concurrent.*;
import static java.util.Arrays.asList;
public class Sums
{
static class Sum implements Callable<Long>
{
private final long from;
private final long to;
Sum(long from, long to)
{
this.from = from;
this.to = to;
}
#Override
public Long call()
{
long acc = 0;
if(from == 0)
{
try
{
Thread.sleep(5000);
}
catch (InterruptedException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
System.out.println(from);
for (long i = from; i <= to; i++)
{
acc = acc + i;
}
return acc;
}
}
public static void main(String[] args) throws Exception
{
ExecutorService executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
List <Future<Long>> results = executor.invokeAll(asList(
new Sum(0, 10), new Sum(100, 1000), new Sum(10000, 1000000)
));
executor.shutdown();
for (Future<Long> result : results)
{
System.out.println(result.get());
}
}
}
With this code, you will be able to get the response and also any exceptions that are thrown.
What should I use to get semantics equivalent to AutoResetEvent in Java?
(See this question for ManualResetEvent).
#user249654's answer looked promising. I added some unit tests to verify it, and indeed it works as expected.
I also added an overload of waitOne that takes a timeout.
The code is here in case anyone else finds it useful:
Unit Test
import org.junit.Assert;
import org.junit.Test;
import static java.lang.System.currentTimeMillis;
/**
* #author Drew Noakes http://drewnoakes.com
*/
public class AutoResetEventTest
{
#Test
public void synchronisesProperly() throws InterruptedException
{
final AutoResetEvent event1 = new AutoResetEvent(false);
final AutoResetEvent event2 = new AutoResetEvent(false);
final int loopCount = 10;
final int sleepMillis = 50;
Thread thread1 = new Thread(new Runnable()
{
#Override
public void run()
{
try {
for (int i = 0; i < loopCount; i++)
{
long t = currentTimeMillis();
event1.waitOne();
Assert.assertTrue("Time to wait should be within 5ms of sleep time",
Math.abs(currentTimeMillis() - t - sleepMillis) < 5);
Thread.sleep(sleepMillis);
t = currentTimeMillis();
event2.set();
Assert.assertTrue("Time to set should be within 1ms", currentTimeMillis() - t <= 1);
}
} catch (InterruptedException e) {
Assert.fail();
}
}
});
Thread thread2 = new Thread(new Runnable()
{
#Override
public void run()
{
try {
for (int i = 0; i < loopCount; i++)
{
Thread.sleep(sleepMillis);
long t = currentTimeMillis();
event1.set();
Assert.assertTrue("Time to set should be within 1ms", currentTimeMillis() - t <= 1);
t = currentTimeMillis();
event2.waitOne();
Assert.assertTrue("Time to wait should be within 5ms of sleep time",
Math.abs(currentTimeMillis() - t - sleepMillis) < 5);
}
} catch (InterruptedException e) {
Assert.fail();
}
}
});
long t = currentTimeMillis();
thread1.start();
thread2.start();
int maxTimeMillis = loopCount * sleepMillis * 2 * 2;
thread1.join(maxTimeMillis);
thread2.join(maxTimeMillis);
Assert.assertTrue("Thread should not be blocked.", currentTimeMillis() - t < maxTimeMillis);
}
#Test
public void timeout() throws InterruptedException
{
AutoResetEvent event = new AutoResetEvent(false);
int timeoutMillis = 100;
long t = currentTimeMillis();
event.waitOne(timeoutMillis);
long took = currentTimeMillis() - t;
Assert.assertTrue("Timeout should have occurred, taking within 5ms of the timeout period, but took " + took,
Math.abs(took - timeoutMillis) < 5);
}
#Test
public void noBlockIfInitiallyOpen() throws InterruptedException
{
AutoResetEvent event = new AutoResetEvent(true);
long t = currentTimeMillis();
event.waitOne(200);
Assert.assertTrue("Should not have taken very long to wait when already open",
Math.abs(currentTimeMillis() - t) < 5);
}
}
AutoResetEvent with overload that accepts a timeout
public class AutoResetEvent
{
private final Object _monitor = new Object();
private volatile boolean _isOpen = false;
public AutoResetEvent(boolean open)
{
_isOpen = open;
}
public void waitOne() throws InterruptedException
{
synchronized (_monitor) {
while (!_isOpen) {
_monitor.wait();
}
_isOpen = false;
}
}
public void waitOne(long timeout) throws InterruptedException
{
synchronized (_monitor) {
long t = System.currentTimeMillis();
while (!_isOpen) {
_monitor.wait(timeout);
// Check for timeout
if (System.currentTimeMillis() - t >= timeout)
break;
}
_isOpen = false;
}
}
public void set()
{
synchronized (_monitor) {
_isOpen = true;
_monitor.notify();
}
}
public void reset()
{
_isOpen = false;
}
}
class AutoResetEvent {
private final Object monitor = new Object();
private volatile boolean open = false;
public AutoResetEvent(boolean open) {
this.open = open;
}
public void waitOne() throws InterruptedException {
synchronized (monitor) {
while (open == false) {
monitor.wait();
}
open = false; // close for other
}
}
public void set() {
synchronized (monitor) {
open = true;
monitor.notify(); // open one
}
}
public void reset() {//close stop
open = false;
}
}
I was able to get CyclicBarrier to work for my purposes.
Here is the C# code I was trying to reproduce in Java (it's just a demonstration program I wrote to isolate the paradigm, I now use it in C# programs I write to generate video in real time, to provide accurate control of the frame rate):
using System;
using System.Timers;
using System.Threading;
namespace TimerTest
{
class Program
{
static AutoResetEvent are = new AutoResetEvent(false);
static void Main(string[] args)
{
System.Timers.Timer t = new System.Timers.Timer(1000);
t.Elapsed += new ElapsedEventHandler(delegate { are.Set(); });
t.Enabled = true;
while (true)
{
are.WaitOne();
Console.WriteLine("main");
}
}
}
}
and here is the Java code I came up with to do the same thing (using the CyclicBarrier class as suggested in a previous answer):
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.CyclicBarrier;
public class TimerTest2 {
static CyclicBarrier cb;
static class MyTimerTask extends TimerTask {
private CyclicBarrier cb;
public MyTimerTask(CyclicBarrier c) { cb = c; }
public void run() {
try { cb.await(); }
catch (Exception e) { }
}
}
public static void main(String[] args) {
cb = new CyclicBarrier(2);
Timer t = new Timer();
t.schedule(new MyTimerTask(cb), 1000, 1000);
while (true) {
try { cb.await(); }
catch (Exception e) { }
System.out.println("main");
}
}
}
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.ReentrantLock;
public class AutoResetEvent {
private volatile boolean _signaled;
private ReentrantLock _lock;
private Condition _condition;
public AutoResetEvent(boolean initialState) {
_signaled = initialState;
_lock = new ReentrantLock();
_condition = _lock.newCondition();
}
public void waitOne(long miliSecond) throws InterruptedException {
_lock.lock();
try {
while (!_signaled)
_condition.await(miliSecond, TimeUnit.MILLISECONDS);
_signaled = false;
} finally {
_lock.unlock();
}
}
public void waitOne() throws InterruptedException {
_lock.lock();
try {
while (!_signaled)
_condition.await();
_signaled = false;
} finally {
_lock.unlock();
}
}
public void set() {
_lock.lock();
try {
_condition.signal();
_signaled = true;
} finally {
_lock.unlock();
}
}
public void reset() {
_lock.lock();
try {
_signaled = false;
} finally {
_lock.unlock();
}
}
}
One more extension to the solution from the accepted answer in case you would like to know whether your wait finished with timeout or with event set (which is exactly what .NET AutoResetEvent does).
public boolean waitOne(long timeout) throws InterruptedException {
synchronized (monitor) {
try {
long t = System.currentTimeMillis();
while (!isOpen) {
monitor.wait(timeout);
// Check for timeout
if (System.currentTimeMillis() - t >= timeout)
break;
}
return isOpen;
}
finally {
isOpen = false;
}
}
}
I believe what you're looking for is either a CyclicBarrier or a CountDownLatch.