Develop Reference

Create x amount of thread, and then wait for then to finish

I'm writing a console application to read json files and then do some processing with them. I have 200k json files to process, so I'm creating a thread per file. But I would like to have only 30 active threads running. I don't know how to control it in Java.
This is the piece of code I have so far:
for (String jsonFile : result) {
final String jsonFilePath = jsonFile;
Thread thread = new Thread(new Runnable() {
String filePath = jsonFilePath;
#Override
public void run() {
// Do stuff here
}
});
thread.start();
}
result is an array with the path of 200k files. From this point, I'm not sure how to control it. I thought about a List<Thread> and then in each thread implements a notifier and when they finish just remove from the list. But then I would have to make the main thread sleep and then wake-up. Which feels weird.
How can I achieve this?

I would suggest to not create one thread per file. Threads are limited resources. Creating too many can lead to starvation or even program abortion.
From what information was provided, I would use a ThreadPoolExecutor. Constructing such an Executor with a limited amount of threads is quite simple thanks to Executors::newFixedSizeThreadPool:
ExecutorService service = Executors.newFixedSizeThreadPool(30);
Looking at the ExecutorService-interface, method <T> Future<T> submit​(Callable<T> task) might be fitting.
For this, some changes will be necessary. The tasks (i.e. what is currently a Runnable in the given implementation) must be converted to a Callable<T>, where T should be substituted with the return-type. The Future<T> returned should then be collected into a list and waited upon on. When all Futures have completed, the result list can be constructed, e.g. through streaming.

With parallelStreams and ForkJoinPool maybe you can get a more straightforward code, plus, an easy way to collect the results of your files after processing. For parallel processing, I prefer to directly use Threads, as a last resort, only when parallelStream can't be used.
boolean doStuff( String file){
// do your magic here
System.out.println( "The file " + file + " has been processed." );
// return the status of the processed file
return true;
}
List<String> jsonFiles = new ArrayList<String>();
jsonFiles.add("file1");
jsonFiles.add("file2");
jsonFiles.add("file3");
...
jsonFiles.add("file200000");
ForkJoinPool forkJoinPool = null;
try {
final int parallelism = 30;
forkJoinPool = new ForkJoinPool(parallelism);
forkJoinPool.submit(() ->
jsonFiles.parallelStream()
.map( jsonFile -> doStuff( jsonFile) )
.collect(Collectors.toList()) // you can collect this to a List<Boolea> results
).get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
} finally {
if (forkJoinPool != null) {
forkJoinPool.shutdown();
}
}

Put your jobs (filenames) into a queue, start 30 threads to process them, then wait until all threads are done. For example:
static ConcurrentLinkedDeque<String> jobQueue = new ConcurrentLinkedDeque<String>();
private static class Worker implements Runnable {
int threadNumber;
public Worker(int threadNumber) {
this.threadNumber = threadNumber;
}
public void run() {
try {
System.out.println("Thread " + threadNumber + " started");
while (true) {
// get the next filename from job queue
String fileName;
try {
fileName = jobQueue.pop();
} catch (NoSuchElementException e) {
// The queue is empty, exit the loop
break;
}
System.out.println("Thread " + threadNumber + " processing file " + fileName);
Thread.sleep(1000); // so something useful here
System.out.println("Thread " + threadNumber + " finished file " + fileName);
}
System.out.println("Thread " + threadNumber + " finished");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public static void main(String[] args) throws InterruptedException {
// Create dummy filenames for testing:
for (int i = 1; i <= 200; i++) {
jobQueue.push("Testfile" + i + ".json");
}
System.out.println("Starting threads");
// Create 30 worker threads
List<Thread> workerThreads = new ArrayList<Thread>();
for (int i = 1; i <= 30; i++) {
Thread thread = new Thread(new Worker(i));
workerThreads.add(thread);
thread.start();
}
// Wait until the threads are all finished
for (Thread thread : workerThreads) {
thread.join();
}
System.out.println("Finished");
}
}

Thread - Can't exit while loop on ExecutorService.isShutDown

So my threads are working as expected, and I just wanted to add some extra sauce to my homework.
I made a while loop that checks uses the isShutdown which returns false unless shutdown(); has been called.
So i call shutdown at the end of my method, but it won't ever exit the while loop.
public void runParrallel() throws InterruptedException {
System.out.println("Submitting Task ...");
ExecutorService executor = Executors.newFixedThreadPool(5);
List<Future<TagCounter>> counters = new ArrayList();
counters.add(executor.submit(new TagCounterCallable("https//www.fck.dk")));
counters.add(executor.submit(new TagCounterCallable("https://www.google.com")));
counters.add(executor.submit(new TagCounterCallable("https://politiken.dk")));
counters.add(executor.submit(new TagCounterCallable("https://cphbusiness.dk")));
System.out.println("Task is submitted");
while (!executor.isShutdown()) {
System.out.println("Task is not completed yet....");
Thread.sleep(1000);
}
for (Future<TagCounter> future : counters) {
try {
TagCounter tc = future.get();
System.out.println("Title: " + tc.getTitle());
System.out.println("Div's: " + tc.getDivCount());
System.out.println("Body's: " + tc.getBodyCount());
System.out.println("----------------------------------");
} catch (ExecutionException ex) {
System.out.println("Exception: " + ex);
}
}
executor.shutdown();
}

The while-loop is before you ever call shutdown(). The condition cannot possibly evaluate to false, thus you are stuck with an infinite loop. I'd suggest moving the while loop to the point after you call shutdown().
See also this question on how to shut down an ExecutorService.

Correct me if I'm wrong, but it looks like you want to wait until all tasks that were submitted to your ExecutorService have finished. If you know that they're going to finish in a timely manner, then you can use ExecutorService#shutdown in conjunction with ExecutorService#awaitTermination to block the executing thread until all tasks are complete.
This can be done with the following:
public void runParrallel() throws InterruptedException {
// Same code to submit tasks.
System.out.println("Task is submitted");
executor.shutdown();
executor.awaitTermination(1, TimeUnit.DAYS);
// At this point, the ExecutorService has been shut down successfully
// and all tasks have finished.
for (Future<TagCounter> future : counters) {
try {
TagCounter tc = future.get();
System.out.println("Title: " + tc.getTitle());
System.out.println("Div's: " + tc.getDivCount());
System.out.println("Body's: " + tc.getBodyCount());
System.out.println("----------------------------------");
} catch (ExecutionException ex) {
System.out.println("Exception: " + ex);
}
}
}
With this solution, the while loop can be removed.

Your while-loop is running infinitely because there is nothing that activates the executor.shutdown() inside the while-loop. The code wont progress to the end where you call executor.shutdown() because the while-loop's condition returns back to the start of the while-loop.
Put an if-statement inside the while-loop. The if-statement checks if the task is submitted, and if it is, the executor.shutdown() will be called.
Following is just an example:
while (!executor.isShutdown()) {
System.out.println("Task is not completed yet....");
Thread.sleep(1000);
if(TaskIsCompleted){
executor.shutdown();
}
}

Receiving items emitted from an Observable

Here's my code:
public static void main(String[] args) {
Observable.just("747", "737", "777")
.flatMap(
a -> {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
return Observable.just(a).subscribeOn(Schedulers.newThread());
})
.subscribe(p -> System.out.println("Received " + p + " on thread " + Thread.currentThread().getName()));
}
As I have understood it, each of the items of the observable should will run in a separate thread (which does happen) and the results would be sent to the same thread that did the work (this happens as well). But what I cannot understand is that why is the main thread not exiting and waiting for the background threads to finish? The program continues for as long as each of the background threads is running.

If you look at a thread dump, you'll see that the main thread is actually stuck on the sleep statement. That's why it's not exiting.
This is because it is the thread that's executing the flatMap operator, so it's getting stuck. This also is why the code takes a long time to run. You can easily verify it by inserting a print statement just before sleep:
try {
System.out.println(Thread.currentThread().getName() + " is sleeping");
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
The output is something like this:
main is sleeping
main is sleeping
Received 747 on thread RxNewThreadScheduler-1
main is sleeping
Received 737 on thread RxNewThreadScheduler-2
Received 777 on thread RxNewThreadScheduler-3
I think you've meant to write something like this:
System.out.println(Thread.currentThread().getName() + " is creating the observable");
Observable.just("747", "737", "777")
.flatMap(a ->
Observable.fromCallable(() -> {
try {
System.out.println(Thread.currentThread().getName() + " is sleeping");
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
return a;
}).subscribeOn(Schedulers.newThread())
).subscribe(p -> System.out.println("Received " + p + " on thread " + Thread.currentThread().getName()));
System.out.println(Thread.currentThread().getName() + " is going to exit");
Output:
main is creating the observable
main is going to exit
RxNewThreadScheduler-3 is sleeping
RxNewThreadScheduler-2 is sleeping
RxNewThreadScheduler-1 is sleeping
Received 777 on thread RxNewThreadScheduler-3
Received 747 on thread RxNewThreadScheduler-1
Received 737 on thread RxNewThreadScheduler-1
In this version main is exiting right after the Observable is created.

Cleaning up after two threads finish execution

I have an application that is running jobs that require two threads for every job. The two threads normally do some work and finish shortly after each other. Then after the second thread finishes I need to do some cleanup but since the threads are doing some network IO, it is possible for one thread to get blocked for a long time. In that case, I want the cleanup to take place a few seconds after the first thread finishes.
I implemented this behaviour with the following piece of code in a callback class:
private boolean first = true;
public synchronized void done() throws InterruptedException {
if (first) {
first = false;
wait(3000);
// cleanup here, as soon as possible
}
else {
notify();
}
}
Both threads invoke the done() method when they finish. The first one will then block in the wait() for at most 3 seconds but will be notified immediately when the seconds thread invokes the done() method.
I have tested this implementation and it seems to work well but I'm am curious if there's a better way of doing this. Even though this implementation doesn't look too complicated, I'm afraid that my program will deadlock or have some unsuspected synchronization issue.

I hope I understood your need. You want to wait for thread-a to complete and then wait either 3 seconds or for the end of thread-b.
It is better to use the newer Concurrent tools instead of the old wait/notify as there are so many edge cases to them.
// Two threads running so count down from 2.
CountDownLatch wait = new CountDownLatch(2);
class TestRun implements Runnable {
private final long waitTime;
public TestRun(long waitTime) {
this.waitTime = waitTime;
}
#Override
public void run() {
try {
// Wait a few seconds.
Thread.sleep(waitTime);
// Finished! Count me down.
wait.countDown();
System.out.println(new Date() + ": " + Thread.currentThread().getName() + " - Finished");
} catch (InterruptedException ex) {
System.out.println(Thread.currentThread().getName() + " - Interrupted");
}
}
}
public void test() throws InterruptedException {
// ThreadA
Thread threadA = new Thread(new TestRun(10000), "Thread A");
// ThreadB
Thread threadB = new Thread(new TestRun(30000), "Thread B");
// Fire them up.
threadA.start();
threadB.start();
// Wait for all to finish but threadA must finish.
threadA.join();
// Wait up to 3 seconds for B.
wait.await(3, TimeUnit.SECONDS);
System.out.println(new Date() + ": Done");
threadB.join();
}
happily prints:
Tue Sep 15 16:59:37 BST 2015: Thread A - Finished
Tue Sep 15 16:59:40 BST 2015: Done
Tue Sep 15 16:59:57 BST 2015: Thread B - Finished
Added
With the new clarity - that the end of any thread starts the timer - we can use a third thread for the cleanup. Each thread must call a method when it finishes to trigger the cleanup mechanism.
// Two threads running so count down from 2.
CountDownLatch wait = new CountDownLatch(2);
class TestRun implements Runnable {
private final long waitTime;
public TestRun(long waitTime) {
this.waitTime = waitTime;
}
#Override
public void run() {
try {
// Wait a few seconds.
Thread.sleep(waitTime);
// Finished! Count me down.
wait.countDown();
System.out.println(new Date() + ": " + Thread.currentThread().getName() + " - Finished");
// Record that I've finished.
finished();
} catch (InterruptedException ex) {
System.out.println(Thread.currentThread().getName() + " - Interrupted");
}
}
}
Runnable cleanup = new Runnable() {
#Override
public void run() {
try {
// Wait up to 3 seconds for both threads to clear.
wait.await(3, TimeUnit.SECONDS);
// Do your cleanup stuff here.
// ...
System.out.println(new Date() + ": " + Thread.currentThread().getName() + " - Finished");
} catch (InterruptedException ex) {
System.out.println(Thread.currentThread().getName() + " - Interrupted");
}
}
};
final AtomicBoolean cleanupStarted = new AtomicBoolean(false);
private void finished() {
// Make sure I only start the cleanup once.
if (cleanupStarted.compareAndSet(false, true)) {
new Thread(cleanup, "Cleanup").start();
}
}
public void test() throws InterruptedException {
// ThreadA
Thread threadA = new Thread(new TestRun(10000), "Thread A");
// ThreadB
Thread threadB = new Thread(new TestRun(30000), "Thread B");
// Fire them up.
threadA.start();
threadB.start();
System.out.println(new Date() + ": Done");
}

As done method is synchronized, so only one thread can execute at a time, with this second will wait to send notify until first finishes its whole job, which might cause performance bottleneck.
I would rather design it with short synchronized block which would primarily update the boolean first.

Java multi threading file saving

I have an app that created multiple endless threads. Each thread reads some info and I created some tasks using thread pool (which is fine).
I have added additional functions that handle arrays, when it finishes, its send those ArrayLists to new thread that save those lists as files. I have implemented the saving in 3 ways and only one of which succeeds. I would like to know why the other 2 ways did not.
I created a thread (via new Thread(Runnable)) and gave it the array and name of the file. In the thread constructor I create the PrintWriter and saved the files. It ran without any problems. ( I have 1-10 file save threads runing in parallel).
If I place the save code outputStream.println(aLog); in the Run method, it never reaches it and after the constructor finishes the thread exit.
I place the created runnables (file save) in a thread pool (and code for saving is in the run() method). When I send just 1 task (1 file to save), all is fine. More than 1 task is being added to the pool (very quickly), exceptions is created (in debug time I can see that all needed info is available) and some of the files are not saved.
Can one explain the difference behavior?
Thanks
Please see code below. (starting with function that is being part of an endless thread class that also place some tasks in the pool), the pool created in the endless thread:
ExecutorService iPool = Executors.newCachedThreadPool();
private void logRate(double r1,int ind){
historicalData.clear();
for (int i = 499; i>0; i--){
// some Code
Data.add(0,array1[ind][i][0] + "," + array1[ind][i][1] + "," +
array1[ind][i][2] + "," + array1[ind][i][3] + "," +
array2[ind][i] + "\n" );
}
// first item
array1[ind][0][0] = r1;
array1[ind][0][1] = array1[ind][0][0] ;
array1[ind][0][2] = array1[ind][0][0] ;
array2[ind][0] = new SimpleDateFormat("HH:mm:ss yyyy_MM_dd").format(today);
Data.add(0,r1+","+r1+","+r1+","+r1+ "," + array2[ind][0] + '\n') ;
// save the log send it to the pool (this is case 3)
//iPool.submit(new FeedLogger(fName,Integer.toString(ind),Data));
// Case 1 and 2
Thread fl = new Thread(new FeedLogger(fName,Integer.toString(ind),Data)) ;
}
here is the FeedLogger class:
public class FeedLogger implements Runnable{
private List<String> fLog = new ArrayList<>() ;
PrintWriter outputStream = null;
String asName,asPathName;
public FeedLogger(String aName,String ind, List<String> fLog) {
this.fLog = fLog;
this.asName = aName;
try {
asPathName = System.getProperty("user.dir") + "\\AsLogs\\" + asName + "\\Feed" + ind
+ ".log" ;
outputStream = new PrintWriter(new FileWriter(asPathName));
outputStream.println(fLog); Case 1 all is fine
outputStream.flush(); // Case 1 all is fine
outputStream.close(); Case 1 all is fine
}
catch (Exception ex) {
JavaFXApplication2.logger.log(Level.SEVERE, null,asName + ex.getMessage());
}
}
#Override
public void run()
{
try{
outputStream.println(fLog); // Cas2 --> not reaching this code, Case3 (as task) create
exception when we have multiple tasks
outputStream.flush();
}
catch (Exception e) {
System.out.println("err in file save e=" + e.getMessage() + asPathName + " feed size=" +
fLog.size());
JavaFXApplication2.logger.log(Level.ALL, null,asName + e.getMessage());
}
finally {if (outputStream != null) {outputStream.close();}}
}
}

You need to call start() on a Thread instance to make it actually do something.