Develop Reference

Create x amount of thread, and then wait for then to finish

I'm writing a console application to read json files and then do some processing with them. I have 200k json files to process, so I'm creating a thread per file. But I would like to have only 30 active threads running. I don't know how to control it in Java.
This is the piece of code I have so far:
for (String jsonFile : result) {
final String jsonFilePath = jsonFile;
Thread thread = new Thread(new Runnable() {
String filePath = jsonFilePath;
#Override
public void run() {
// Do stuff here
}
});
thread.start();
}
result is an array with the path of 200k files. From this point, I'm not sure how to control it. I thought about a List<Thread> and then in each thread implements a notifier and when they finish just remove from the list. But then I would have to make the main thread sleep and then wake-up. Which feels weird.
How can I achieve this?

I would suggest to not create one thread per file. Threads are limited resources. Creating too many can lead to starvation or even program abortion.
From what information was provided, I would use a ThreadPoolExecutor. Constructing such an Executor with a limited amount of threads is quite simple thanks to Executors::newFixedSizeThreadPool:
ExecutorService service = Executors.newFixedSizeThreadPool(30);
Looking at the ExecutorService-interface, method <T> Future<T> submit​(Callable<T> task) might be fitting.
For this, some changes will be necessary. The tasks (i.e. what is currently a Runnable in the given implementation) must be converted to a Callable<T>, where T should be substituted with the return-type. The Future<T> returned should then be collected into a list and waited upon on. When all Futures have completed, the result list can be constructed, e.g. through streaming.

With parallelStreams and ForkJoinPool maybe you can get a more straightforward code, plus, an easy way to collect the results of your files after processing. For parallel processing, I prefer to directly use Threads, as a last resort, only when parallelStream can't be used.
boolean doStuff( String file){
// do your magic here
System.out.println( "The file " + file + " has been processed." );
// return the status of the processed file
return true;
}
List<String> jsonFiles = new ArrayList<String>();
jsonFiles.add("file1");
jsonFiles.add("file2");
jsonFiles.add("file3");
...
jsonFiles.add("file200000");
ForkJoinPool forkJoinPool = null;
try {
final int parallelism = 30;
forkJoinPool = new ForkJoinPool(parallelism);
forkJoinPool.submit(() ->
jsonFiles.parallelStream()
.map( jsonFile -> doStuff( jsonFile) )
.collect(Collectors.toList()) // you can collect this to a List<Boolea> results
).get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
} finally {
if (forkJoinPool != null) {
forkJoinPool.shutdown();
}
}

Put your jobs (filenames) into a queue, start 30 threads to process them, then wait until all threads are done. For example:
static ConcurrentLinkedDeque<String> jobQueue = new ConcurrentLinkedDeque<String>();
private static class Worker implements Runnable {
int threadNumber;
public Worker(int threadNumber) {
this.threadNumber = threadNumber;
}
public void run() {
try {
System.out.println("Thread " + threadNumber + " started");
while (true) {
// get the next filename from job queue
String fileName;
try {
fileName = jobQueue.pop();
} catch (NoSuchElementException e) {
// The queue is empty, exit the loop
break;
}
System.out.println("Thread " + threadNumber + " processing file " + fileName);
Thread.sleep(1000); // so something useful here
System.out.println("Thread " + threadNumber + " finished file " + fileName);
}
System.out.println("Thread " + threadNumber + " finished");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public static void main(String[] args) throws InterruptedException {
// Create dummy filenames for testing:
for (int i = 1; i <= 200; i++) {
jobQueue.push("Testfile" + i + ".json");
}
System.out.println("Starting threads");
// Create 30 worker threads
List<Thread> workerThreads = new ArrayList<Thread>();
for (int i = 1; i <= 30; i++) {
Thread thread = new Thread(new Worker(i));
workerThreads.add(thread);
thread.start();
}
// Wait until the threads are all finished
for (Thread thread : workerThreads) {
thread.join();
}
System.out.println("Finished");
}
}

Java-Cannot terminate main thread with return;

here is my code of main class, because only thread 3 will change the shutdown and main will not read return until thread 3 close so there no need to synchronize it. When I try to run it, the "shutdown" is printed but eclipse say that the program is still running
static Boolean shutdown = false;
public static void main(String[] args) throws InterruptedException, IOException{
System.out.println("Start Server");
final int SocketListSize = 100000;
final int AccListSize = 1000000;
ServerSocket serverSocket;
List<Socket> socketList = Collections.synchronizedList(new ArrayList<Socket>(SocketListSize));
List<Request> requestList = Collections.synchronizedList(new ArrayList<Request>(SocketListSize));
ArrayList<BankAccount> bankAccList = new ArrayList<BankAccount>(AccListSize);
serverSocket = new ServerSocket(1234);
serverSocket.setSoTimeout(50000);
Thread Thread_1 = new Thread(new scanSocketThread(serverSocket, socketList));
Thread Thread_2 = new Thread(new getRequestThread(socketList, requestList));
Thread Thread_3 = new Thread(new ServiceThread(requestList, bankAccList));
Thread_1.start();
System.out.println("thread 1 start");
Thread_2.start();
System.out.println("thread 2 start");
Thread_3.start();
System.out.println("thread 3 start");
Thread_3.join();
System.out.println("thread 3 close");
Thread_1.interrupt();
System.out.println("thread 1 close");
Thread_2.interrupt();
System.out.println("thread 2 close");
if(shutdown == true){
System.out.println("shutdown");
return;
}
Here is what I get
thread 3 close
thread 1 close
thread 2 close
shutdown

If you're looking to kill your program, you can use System.exit(0); instead of the return statement you have now.
More: What are the differences between calling System.exit(0) and Thread.currentThread().interrupt() in the main thread of a Java program?

Cleaning up after two threads finish execution

I have an application that is running jobs that require two threads for every job. The two threads normally do some work and finish shortly after each other. Then after the second thread finishes I need to do some cleanup but since the threads are doing some network IO, it is possible for one thread to get blocked for a long time. In that case, I want the cleanup to take place a few seconds after the first thread finishes.
I implemented this behaviour with the following piece of code in a callback class:
private boolean first = true;
public synchronized void done() throws InterruptedException {
if (first) {
first = false;
wait(3000);
// cleanup here, as soon as possible
}
else {
notify();
}
}
Both threads invoke the done() method when they finish. The first one will then block in the wait() for at most 3 seconds but will be notified immediately when the seconds thread invokes the done() method.
I have tested this implementation and it seems to work well but I'm am curious if there's a better way of doing this. Even though this implementation doesn't look too complicated, I'm afraid that my program will deadlock or have some unsuspected synchronization issue.

I hope I understood your need. You want to wait for thread-a to complete and then wait either 3 seconds or for the end of thread-b.
It is better to use the newer Concurrent tools instead of the old wait/notify as there are so many edge cases to them.
// Two threads running so count down from 2.
CountDownLatch wait = new CountDownLatch(2);
class TestRun implements Runnable {
private final long waitTime;
public TestRun(long waitTime) {
this.waitTime = waitTime;
}
#Override
public void run() {
try {
// Wait a few seconds.
Thread.sleep(waitTime);
// Finished! Count me down.
wait.countDown();
System.out.println(new Date() + ": " + Thread.currentThread().getName() + " - Finished");
} catch (InterruptedException ex) {
System.out.println(Thread.currentThread().getName() + " - Interrupted");
}
}
}
public void test() throws InterruptedException {
// ThreadA
Thread threadA = new Thread(new TestRun(10000), "Thread A");
// ThreadB
Thread threadB = new Thread(new TestRun(30000), "Thread B");
// Fire them up.
threadA.start();
threadB.start();
// Wait for all to finish but threadA must finish.
threadA.join();
// Wait up to 3 seconds for B.
wait.await(3, TimeUnit.SECONDS);
System.out.println(new Date() + ": Done");
threadB.join();
}
happily prints:
Tue Sep 15 16:59:37 BST 2015: Thread A - Finished
Tue Sep 15 16:59:40 BST 2015: Done
Tue Sep 15 16:59:57 BST 2015: Thread B - Finished
Added
With the new clarity - that the end of any thread starts the timer - we can use a third thread for the cleanup. Each thread must call a method when it finishes to trigger the cleanup mechanism.
// Two threads running so count down from 2.
CountDownLatch wait = new CountDownLatch(2);
class TestRun implements Runnable {
private final long waitTime;
public TestRun(long waitTime) {
this.waitTime = waitTime;
}
#Override
public void run() {
try {
// Wait a few seconds.
Thread.sleep(waitTime);
// Finished! Count me down.
wait.countDown();
System.out.println(new Date() + ": " + Thread.currentThread().getName() + " - Finished");
// Record that I've finished.
finished();
} catch (InterruptedException ex) {
System.out.println(Thread.currentThread().getName() + " - Interrupted");
}
}
}
Runnable cleanup = new Runnable() {
#Override
public void run() {
try {
// Wait up to 3 seconds for both threads to clear.
wait.await(3, TimeUnit.SECONDS);
// Do your cleanup stuff here.
// ...
System.out.println(new Date() + ": " + Thread.currentThread().getName() + " - Finished");
} catch (InterruptedException ex) {
System.out.println(Thread.currentThread().getName() + " - Interrupted");
}
}
};
final AtomicBoolean cleanupStarted = new AtomicBoolean(false);
private void finished() {
// Make sure I only start the cleanup once.
if (cleanupStarted.compareAndSet(false, true)) {
new Thread(cleanup, "Cleanup").start();
}
}
public void test() throws InterruptedException {
// ThreadA
Thread threadA = new Thread(new TestRun(10000), "Thread A");
// ThreadB
Thread threadB = new Thread(new TestRun(30000), "Thread B");
// Fire them up.
threadA.start();
threadB.start();
System.out.println(new Date() + ": Done");
}

As done method is synchronized, so only one thread can execute at a time, with this second will wait to send notify until first finishes its whole job, which might cause performance bottleneck.
I would rather design it with short synchronized block which would primarily update the boolean first.

Breaking out of a while loop if the process takes more than a specified amount of time in java

I'm reading a server log file after an event is performed on the UI. I have a while loop which waits for certain conditions to match and then returns that line of the log. Sometimes, however, there's a case where an event occurs before the code looks at the log and cannot get the new line. This causes the while loop to just hang and this hangs until another event occurs with the provided conditions. This is problematic for obvious reasons. Is there a way to break out of the while loop after a few seconds no matter what the case maybe? Following is my code
public String method(String, a, String b, String c) {
channel = session.openChannel("exec");
((ChannelExec) channel).setCommand(a + "\n" + b);
channel.connect();
fromServer = new BufferedReader (new InputStreamReader(channel.getInputStream()));
String time = methodToFireAnEventInUI();
Thread.sleep(2000);
String x = null;
while (true){
x = fromServer.readLine();
if(!x.equals(null) && x.contains(c) && x.contains(time)){
break;
}
}
msg = x.toString();
}catch (Exception e){
e.printStackTrace();
}
closeConnection();
return msg;
}
If you look at the above code, it hangs right at "x = fromServer.readline();" and just doesn't go anywhere, and that is where I want the logic for it to wait for an x amount of time and just abort the loop after that.
My attempt of "thread.sleep" ahead of the while loop doesn't work either.

You can put this logic in a separate thread and use a while like this:
class TestThread extends Thread {
#Override
public void run() {
while (!Thread.currentThread().isInterrupted()) {
method();
}
}
public void method() {
try {
// this method hangs. You can replace it with your method
while (true) {
sleep(100);
}
} catch (Exception e) {
System.out.println("Thread is interrupted");
}
}
}
After that you can interrupt this thread if it takes longer than some time frame like this:
public static void main(String[] args) throws Exception {
TestThread t1 = new TestThread();
long startTime = System.currentTimeMillis();
t1.start();
long currentTime = System.currentTimeMillis();
while (currentTime - startTime < 5000) { // you can decide the desired interval
sleep(1000); // sleep some time
currentTime = System.currentTimeMillis();
System.out.println(currentTime); //print this to ensure that the program is still running
}
t1.interrupt(); //interrupt the thread
}

How about simply:
long timeOut = System.currentTimeMillis() + 5000; // change the value to what ever(millis)
while(System.currentTimeMillis() < timeOut){
// do whatever
}

As your while loop blocks at "x = fromServer.readline();" you can just share the reader instance to another thread and make that thread close the reader after timeout. This will cause your readLine to throw exception which you can handle and proceed.

Find answer here:
How do I measure time elapsed in Java?
Try the approach below:
long startTime = System.nanoTime(); //fetch starting time
while(true ||(System.nanoTime()-startTime)<200000)
{
// do something
}

How to stop a task after it's timed out x times

I am trying execute a runnable a few times, and if it doesn't finished within x seconds 3 times, I will cancel it.
The code I'm using to simulate the situation where the task needs to be cancelled is as follows. From the output I can see that an InterruptedException was thrown and caught accordingly, but the task keeps running.
It seems that the first two times the task was run before the TimeoutException was thrown 3 times, those two runs kept on running until they are finished. I'm wondering if there is a way to stop those two runs from completing ?
public class SomeClass {
private static int c =0;
public static void main(String[] args){
Runnable dummyRunnable = new Runnable() {
#Override
public void run() {
System.out.println("Hello from dummyRunnable!");
for (int i =0; i< 10; i++){
try {
//simulate work here
if (!Thread.currentThread().isInterrupted()) Thread.sleep(5000);
System.out.println("thread sleeps for the " + i + " time!");
} catch (InterruptedException ie){
System.out.println("InterruptedException catched in dummyRunnable!");
//Thread.currentThread().interrupt(); //this has no effects
break;
}
}
}
};
BlockingQueue<Runnable> blockingQueue = new ArrayBlockingQueue<Runnable>(10 * 3, true);
ThreadPoolExecutor executor = new ThreadPoolExecutor(3, 3, Long.MAX_VALUE, TimeUnit.MILLISECONDS, blockingQueue);
for (int i =0; i< 5; i++){
Future<?> task = executor.submit(dummyRunnable);
try{
Thread.sleep(1000);
task.get(2000, TimeUnit.MILLISECONDS);
} catch (TimeoutException te){
c++;
System.out.println("TimeoutException from a task!");
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (c==3){
System.out.println("cancelling task...");
task.cancel(true);
break;
}
}
}
}
}

I don't get it what you are actually trying to simulate. I would expect a simulation like paying with card (60 secs time-out to finish a task) or perhaps a secretary in a doctor-patient situation.
The way it stand now you are creating the 5 objects in the Future.
If you want more control off your threads, you should think about using synchronized methods and a monitor that handles the threads for you.
Usually when starting a thread you should go with
new Thread(new Task(object or generics)).start();
Thread.sleep(2000); // calls this thread to wait 2 secs before doing other task(s)
Before doing some hardcore concurrency(multithreading), you should read some java tutorial to get some inspiration...
http://docs.oracle.com/javase/tutorial/essential/concurrency/index.html