There is declaring of host's features in platform.xml file:
<host id="Tier1_1" core="2" speed="100f"/>
The worker process lives in this host.
How can worker simultaneously receive and execute two tasks (in case of number of core is 2)?
Now I use such code, but it doesn't work in this case(this code can't simultaneously receive two task, only one);
while(true) {
commReceived = Task.irecv("Tier1_" + num);
commReceived.waitCompletion();
if (commReceived.test()){
task = commReceived.getTask();
commReceived = null;
Msg.info("Receive " + task.getName());
task.execute();
Msg.info("End to execute " + task.getName());
}
UPD:
Now I use this code. There are two processes with the same mailbox "Tier1_2". I send with isend to mailbox ("Tier1_2"):
for (int j=0; j<2; j++){
Process process = new Process(getHost().getName(), "Tier1_2_" + j) {
#Override
public void main(String[] strings) throws MsgException {
while (true){
commReceived = Task.irecv("Tier1_2");
commReceived.waitCompletion();
if (commReceived.test()){
task = commReceived.getTask();
commReceived = null;
Msg.info("Receive " + task.getName());
}
}
}
};process.start();
}
But it gives:
Exception in thread "Thread-5" java.lang.NullPointerException
at LHCb.Tier1$1.main(Tier1.java:46)
at org.simgrid.msg.Process.run(Process.java:338)
How correctly I should declare processes?
The idea is to have the worker process to spawn other processes that listen on different mailboxes. For instance something like (which I haven't tested)
for (int i = 0; i < 2; i++) {
Process p = new Process(getHost.getName(), "Tier1_" + i) {
public void main(String[] args) throws MsgException {
String mailbox = getName();
while(true) {
commReceived = Task.irecv(mailbox);
commReceived.waitCompletion();
if (commReceived.test()){
task = commReceived.getTask();
commReceived = null;
Msg.info("Receive " + task.getName());
task.execute();
Msg.info("End to execute " + task.getName());
}
}
});
p.start();
}
The new Process() method takes two arguments: the name of the host on which the process runs, and the name of the process itself. Here we declare a unique process name that will be used as the mailbox name (hence the mailbox = getName()).
Don't forget to kill these processes at some point, as they run forever. So you might want to put all the spawned processes in a vector to ease that.
Related
I'm writing a console application to read json files and then do some processing with them. I have 200k json files to process, so I'm creating a thread per file. But I would like to have only 30 active threads running. I don't know how to control it in Java.
This is the piece of code I have so far:
for (String jsonFile : result) {
final String jsonFilePath = jsonFile;
Thread thread = new Thread(new Runnable() {
String filePath = jsonFilePath;
#Override
public void run() {
// Do stuff here
}
});
thread.start();
}
result is an array with the path of 200k files. From this point, I'm not sure how to control it. I thought about a List<Thread> and then in each thread implements a notifier and when they finish just remove from the list. But then I would have to make the main thread sleep and then wake-up. Which feels weird.
How can I achieve this?
I would suggest to not create one thread per file. Threads are limited resources. Creating too many can lead to starvation or even program abortion.
From what information was provided, I would use a ThreadPoolExecutor. Constructing such an Executor with a limited amount of threads is quite simple thanks to Executors::newFixedSizeThreadPool:
ExecutorService service = Executors.newFixedSizeThreadPool(30);
Looking at the ExecutorService-interface, method <T> Future<T> submit​(Callable<T> task) might be fitting.
For this, some changes will be necessary. The tasks (i.e. what is currently a Runnable in the given implementation) must be converted to a Callable<T>, where T should be substituted with the return-type. The Future<T> returned should then be collected into a list and waited upon on. When all Futures have completed, the result list can be constructed, e.g. through streaming.
With parallelStreams and ForkJoinPool maybe you can get a more straightforward code, plus, an easy way to collect the results of your files after processing. For parallel processing, I prefer to directly use Threads, as a last resort, only when parallelStream can't be used.
boolean doStuff( String file){
// do your magic here
System.out.println( "The file " + file + " has been processed." );
// return the status of the processed file
return true;
}
List<String> jsonFiles = new ArrayList<String>();
jsonFiles.add("file1");
jsonFiles.add("file2");
jsonFiles.add("file3");
...
jsonFiles.add("file200000");
ForkJoinPool forkJoinPool = null;
try {
final int parallelism = 30;
forkJoinPool = new ForkJoinPool(parallelism);
forkJoinPool.submit(() ->
jsonFiles.parallelStream()
.map( jsonFile -> doStuff( jsonFile) )
.collect(Collectors.toList()) // you can collect this to a List<Boolea> results
).get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
} finally {
if (forkJoinPool != null) {
forkJoinPool.shutdown();
}
}
Put your jobs (filenames) into a queue, start 30 threads to process them, then wait until all threads are done. For example:
static ConcurrentLinkedDeque<String> jobQueue = new ConcurrentLinkedDeque<String>();
private static class Worker implements Runnable {
int threadNumber;
public Worker(int threadNumber) {
this.threadNumber = threadNumber;
}
public void run() {
try {
System.out.println("Thread " + threadNumber + " started");
while (true) {
// get the next filename from job queue
String fileName;
try {
fileName = jobQueue.pop();
} catch (NoSuchElementException e) {
// The queue is empty, exit the loop
break;
}
System.out.println("Thread " + threadNumber + " processing file " + fileName);
Thread.sleep(1000); // so something useful here
System.out.println("Thread " + threadNumber + " finished file " + fileName);
}
System.out.println("Thread " + threadNumber + " finished");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public static void main(String[] args) throws InterruptedException {
// Create dummy filenames for testing:
for (int i = 1; i <= 200; i++) {
jobQueue.push("Testfile" + i + ".json");
}
System.out.println("Starting threads");
// Create 30 worker threads
List<Thread> workerThreads = new ArrayList<Thread>();
for (int i = 1; i <= 30; i++) {
Thread thread = new Thread(new Worker(i));
workerThreads.add(thread);
thread.start();
}
// Wait until the threads are all finished
for (Thread thread : workerThreads) {
thread.join();
}
System.out.println("Finished");
}
}
I'm new to Akka toolkit. I need to run a process on multiple files that takes a considerable amount of time. So I created one actor per file and started the processing. I'm creating these actors in a POJO class as follows:
public class ProcessFiles {
private static final Logger logger = LoggerFactory.getLogger(ProcessFiles.class.getSimpleName());
public static void main(String[] args) throws IOException, InterruptedException {
long startTime = System.currentTimeMillis();
logger.info("Creating actor system");
ActorSystem system = ActorSystem.create("actor_system");
Set<String> files = new HashSet<>();
Stream<String> stringStream = Files.lines(Paths.get(fileName));
stringStream.forEach(line -> files.addAll(Arrays.asList(line.split(","))));
List<CompletableFuture<Object>> futureList = new ArrayList<>();
files.forEach((String file) -> {
ActorRef actorRef = system.actorOf(Props.create(ProcessFile.class, file));
futureList.add(PatternsCS.ask(actorRef, file, DEFAULT_TIMEOUT).toCompletableFuture());
});
boolean isDone;
do {
Thread.sleep(30000);
isDone = true;
int count = 0;
for (CompletableFuture<Object> future : futureList) {
isDone = isDone & (future.isDone() || future.isCompletedExceptionally() || future.isCancelled());
if (future.isDone() || future.isCompletedExceptionally() || future.isCancelled()) {
++count;
}
}
logger.info("Process is completed for " + count + " files out of " + files.size() + " files.");
} while (!isDone);
logger.info("Process is done in " + (System.currentTimeMillis() - startTime) + " ms");
system.terminate();
}
}
Here, ProcessFile is the actor class. After invoking all the actors in order to exit the program, the main process checks whether all the actors are finished or not in every 30 seconds. Is there any better way to implement this kind of functionality?
I would suggest to create one more actor that keeps tracks of termination of all the actors in system, and closing the actor system when all the actors are killed.
So in your application-
ProcessFile actor can send a poison pill to self, after processing the file.
WatcherActor will watch(context.watch(processFileActor)) the ProcessFileActor and maintain the count of all the ProcessFile actors registered.
On termination of the actors WatcherActor will receive the Terminated message.
It will decrease the count, and when the count reaches 0, close the ActorSystem.
Greets!
I am writing a simple server monitoring application in Java(JavaFX8). The current implementation is able to ping target machines one by one, and graph them onto a JavaFX LineChart. Each machine is a "Target" object, which is held in an ArrayList (Observable). My problem is the "one by one" part. The code to ping a target is a Callable that returns the ping. I, somehow, need to multithread the process so that I can ping the targets at least four at a time. Past attempts resulted in quirks such as four threads pinging the same target at the same time, resulting in a very pointless and processor intense redundancy. Heres my current loop...
public void beginPing() {
ExecutorService exec = Executors.newCachedThreadPool();
Runnable r = new Runnable() {
#Override
public void run() {
while (true) {
for (Target t : targets) {
String ping = null;
if (t.flagsProperty().get().contains("A")) {
try {
Callable c = new Pinger(t);
ping = c.call().toString();
switch (ping) {
case "TIME_OUT":
for (XYChart.Series s : lineChart.getData()) {
if (s.getName().equals(t.nameProperty().get())) {
addToChart(s, cycle, 00.00);
}
}
t.setStatus("TIME OUT");
t.setLastrtt("TIME_OUT");
t.setTimeouts(t.timeoutsProperty().get() + 1);
logUtil.log(LogUtil.INFO, t.nameProperty().get() + " - timed out!");
break;
case "UNKNOWN_HOST":
t.setStatus("ERROR");
t.setLastrtt("UNKNOWN HOST");
logUtil.log(LogUtil.WARNING, t.nameProperty().get() + " - unknown host!");
break;
case "UNREACHABLE":
t.setStatus("ERROR");
t.setLastrtt("UNREACHABLE HOST");
logUtil.log(LogUtil.WARNING, t.nameProperty().get() + " - is unreachable!");
break;
default:
t.setLastrtt(ping);
t.setStatus("ACTIVE");
for (XYChart.Series s : lineChart.getData()) {
if (s.getName().equals(t.nameProperty().get())) {
addToChart(s, cycle, Double.valueOf(ping));
}
}
break;
}
} catch (Exception e) {
logUtil.log(LogUtil.CRITICAL, e.getMessage() + ", "+ e.getCause());
e.printStackTrace();
}
}
}
cycle++;
rangeChart(cycle);
updateInfo();
}
}
};
exec.execute(r);
}
My impression is that you misuse your Callable class Pinger like a regular class, although it is only an interface that does not implement any multithreading services.
The thing you want to do should look more like this:
//init
Future<String> futures = new Future[targets.length];
String results = new String[targets.length];
ExecutorService service = Executors.newCachedThreadPool();
//start Threads
for (int i = 0; i<targets.length; i++){
Pinger pinger= new Pinger(targets[i]);
future[i] = service.submit(pinger);
}
//wait for Threads to finish and get results
for(int i = 0; i<futures.length; i++)
results[i] = futures[i].get()
Your Pinger should look like this:
public class Pinger implements Callable<String>{
Pinger(Target target){ ... }
public String call(){ ... }
}
Here you find a fully implemented Example for Callables. In your code you only submit one Runnable to the ExecutorService, so there will be only two threads (Main and your Runnable). You never call the method call(), this is done by the ExecutorService. Compare this to the Runnable Interface you have to execute the Thread calling start or submitting it to a ExecutorService instead of calling run(); You use the Future that is returned during the submit(). Just try to understand the concept of Callable and then you will be able to write everything you want. ;-)
So, heres the current working implementation...
public void beginPing() {
safeTargets = new ArrayList<>(); //thread-safe collection
for (Target t : targets) {
safeTargets.add(t);
}
safeTargets = Collections.synchronizedList(targets);
exec = Executors.newCachedThreadPool();
for (int i = 0; i < 4; i++) { //number of threads
exec.execute(new Runnable() {
#Override
public void run() {
while (true) {
for (Target t : safeTargets) {
String ping = null;
if (t.isActive() && !t.isIsBeingPinged()) { //checks if target is already being pinged by another thread and if it flagged as active and wishes to be pinged.
t.setIsBeingPinged(true);
t.setPinged(t.getPinged() + 1); //just to see how many times it has been pinged
t.setStatus("PINGING");
try {
Callable c = new Pinger(t);
ping = c.call().toString();
switch (ping) {
case "TIME_OUT":
t.setStatus("TIME OUT");
t.setLastrtt("TIME_OUT");
t.setTimeouts(t.timeoutsProperty().get() + 1);
logUtil.log(LogUtil.INFO, t.nameProperty().get() + " - timed out!");
t.setIsBeingPinged(false);
break;
case "UNKNOWN_HOST":
t.setStatus("ERROR");
t.setLastrtt("UNKNOWN HOST");
logUtil.log(LogUtil.WARNING, t.nameProperty().get() + " - unknown host!");
t.setIsBeingPinged(false);
break;
case "UNREACHABLE":
t.setStatus("ERROR");
t.setLastrtt("UNREACHABLE HOST");
logUtil.log(LogUtil.WARNING, t.nameProperty().get() + " - is unreachable!");
t.setIsBeingPinged(false);
break;
default:
t.setLastrtt(ping);
t.setStatus("ACTIVE");
t.setIsBeingPinged(false);
break;
}
System.out.println("C=" + t.getPinged() + " - " + t.nameProperty().get());
} catch (Exception e) {
logUtil.log(LogUtil.CRITICAL, e.getMessage() + ", " + e.getCause());
e.printStackTrace();
}
}
}
}
}
});
}
}
I had to get rid of the immediate addition to the chart after all.
The Target list objects get added to a thread-safe synchronizedList (as suggested by kleopatra).
A boolean variable was added to the Target model to determine if it is currently being pinged by one of the threads. (t.isIsBeingPinged())
The data gets added to the chart using a new Runnable in the same pool, which iterates the target list and adds the last RTT to the chart every second, to avoid Targets with higher pings from falling behind on the chart.
Thanks for the very quick responses!
I have an app that created multiple endless threads. Each thread reads some info and I created some tasks using thread pool (which is fine).
I have added additional functions that handle arrays, when it finishes, its send those ArrayLists to new thread that save those lists as files. I have implemented the saving in 3 ways and only one of which succeeds. I would like to know why the other 2 ways did not.
I created a thread (via new Thread(Runnable)) and gave it the array and name of the file. In the thread constructor I create the PrintWriter and saved the files. It ran without any problems. ( I have 1-10 file save threads runing in parallel).
If I place the save code outputStream.println(aLog); in the Run method, it never reaches it and after the constructor finishes the thread exit.
I place the created runnables (file save) in a thread pool (and code for saving is in the run() method). When I send just 1 task (1 file to save), all is fine. More than 1 task is being added to the pool (very quickly), exceptions is created (in debug time I can see that all needed info is available) and some of the files are not saved.
Can one explain the difference behavior?
Thanks
Please see code below. (starting with function that is being part of an endless thread class that also place some tasks in the pool), the pool created in the endless thread:
ExecutorService iPool = Executors.newCachedThreadPool();
private void logRate(double r1,int ind){
historicalData.clear();
for (int i = 499; i>0; i--){
// some Code
Data.add(0,array1[ind][i][0] + "," + array1[ind][i][1] + "," +
array1[ind][i][2] + "," + array1[ind][i][3] + "," +
array2[ind][i] + "\n" );
}
// first item
array1[ind][0][0] = r1;
array1[ind][0][1] = array1[ind][0][0] ;
array1[ind][0][2] = array1[ind][0][0] ;
array2[ind][0] = new SimpleDateFormat("HH:mm:ss yyyy_MM_dd").format(today);
Data.add(0,r1+","+r1+","+r1+","+r1+ "," + array2[ind][0] + '\n') ;
// save the log send it to the pool (this is case 3)
//iPool.submit(new FeedLogger(fName,Integer.toString(ind),Data));
// Case 1 and 2
Thread fl = new Thread(new FeedLogger(fName,Integer.toString(ind),Data)) ;
}
here is the FeedLogger class:
public class FeedLogger implements Runnable{
private List<String> fLog = new ArrayList<>() ;
PrintWriter outputStream = null;
String asName,asPathName;
public FeedLogger(String aName,String ind, List<String> fLog) {
this.fLog = fLog;
this.asName = aName;
try {
asPathName = System.getProperty("user.dir") + "\\AsLogs\\" + asName + "\\Feed" + ind
+ ".log" ;
outputStream = new PrintWriter(new FileWriter(asPathName));
outputStream.println(fLog); Case 1 all is fine
outputStream.flush(); // Case 1 all is fine
outputStream.close(); Case 1 all is fine
}
catch (Exception ex) {
JavaFXApplication2.logger.log(Level.SEVERE, null,asName + ex.getMessage());
}
}
#Override
public void run()
{
try{
outputStream.println(fLog); // Cas2 --> not reaching this code, Case3 (as task) create
exception when we have multiple tasks
outputStream.flush();
}
catch (Exception e) {
System.out.println("err in file save e=" + e.getMessage() + asPathName + " feed size=" +
fLog.size());
JavaFXApplication2.logger.log(Level.ALL, null,asName + e.getMessage());
}
finally {if (outputStream != null) {outputStream.close();}}
}
}
You need to call start() on a Thread instance to make it actually do something.
I am new to Multithreading and synchronization in java. I am trying to achieve a task in which i am given 5 files, each file will be read by one particular thread. Every thread should read one line from file then forward execution to next thread and so on. When all 5 threads read the first line, then again start from thread 1 running line no. 2 of file 1 and so on.
Thread ReadThread1 = new Thread(new ReadFile(0));
Thread ReadThread2 = new Thread(new ReadFile(1));
Thread ReadThread3 = new Thread(new ReadFile(2));
Thread ReadThread4 = new Thread(new ReadFile(3));
Thread ReadThread5 = new Thread(new ReadFile(4));
// starting all the threads
ReadThread1.start();
ReadThread2.start();
ReadThread3.start();
ReadThread4.start();
ReadThread5.start();
and in ReadFile (which implements Runnable, in the run method, i am trying to synchronize on bufferreader object.
BufferedReader br = null;
String sCurrentLine;
String filename="Source/"+files[fileno];
br = new BufferedReader(new FileReader(filename));
synchronized(br)
{
while ((sCurrentLine = br.readLine()) != null) {
int f=fileno+1;
System.out.print("File No."+f);
System.out.println("-->"+sCurrentLine);
br.notifyAll();
// some thing needs to be dine here i guess
}}
Need Help
Though this is not an ideal scenario for using multi-threading but as this is assignment I am putting one solution that works. The threads will execute sequentially and there are few point to note:
Current thread cannot move ahead to read the line in the file until and unless its immediately previous thread is done as they are supposed to read in round-robin fashion.
After current thread is done reading the line it must notify the other thread else that thread will wait forever.
I have tested this code with some files in temp package and it was able to read the lines in round robin fashion. I believe Phaser can also be used to solve this problem.
public class FileReaderRoundRobinNew {
public Object[] locks;
private static class LinePrinterJob implements Runnable {
private final Object currentLock;
private final Object nextLock;
BufferedReader bufferedReader = null;
public LinePrinterJob(String fileToRead, Object currentLock, Object nextLock) {
this.currentLock = currentLock;
this.nextLock = nextLock;
try {
this.bufferedReader = new BufferedReader(new FileReader(fileToRead));
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
#Override
public void run() {
/*
* Few points to be noted:
* 1. Current thread cannot move ahead to read the line in the file until and unless its immediately previous thread is done as they are supposed to read in round-robin fashion.
* 2. After current thread is done reading the line it must notify the other thread else that thread will wait forever.
* */
String currentLine;
synchronized(currentLock) {
try {
while ( (currentLine = bufferedReader.readLine()) != null) {
try {
currentLock.wait();
System.out.println(currentLine);
}
catch(InterruptedException e) {}
synchronized(nextLock) {
nextLock.notify();
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
synchronized(nextLock) {
nextLock.notify(); /// Ensures all threads exit at the end
}
}
}
public FileReaderRoundRobinNew(int numberOfFilesToRead) {
locks = new Object[numberOfFilesToRead];
int i;
String fileLocation = "src/temp/";
//Initialize lock instances in array.
for(i = 0; i < numberOfFilesToRead; ++i) locks[i] = new Object();
//Create threads
int j;
for(j=0; j<(numberOfFilesToRead-1); j++ ){
Thread linePrinterThread = new Thread(new LinePrinterJob(fileLocation + "Temp" + j,locks[j],locks[j+1]));
linePrinterThread.start();
}
Thread lastLinePrinterThread = new Thread(new LinePrinterJob(fileLocation + "Temp" + j,locks[numberOfFilesToRead-1],locks[0]));
lastLinePrinterThread.start();
}
public void startPrinting() {
synchronized (locks[0]) {
locks[0].notify();
}
}
public static void main(String[] args) {
FileReaderRoundRobinNew fileReaderRoundRobin = new FileReaderRoundRobinNew(4);
fileReaderRoundRobin.startPrinting();
}
}
If the only objective is to read the files in round-robin fashion and not strictly in same order then we can also use Phaser. In this case the order in which files are read is not always same, for example if we have four files (F1, F2, F3 and F4) then in first phase it can read them as F1-F2-F3-F4 but in next one it can read them as F2-F1-F4-F3. I am still putting this solution for sake of completion.
public class FileReaderRoundRobinUsingPhaser {
final List<Runnable> tasks = new ArrayList<>();
final int numberOfLinesToRead;
private static class LinePrinterJob implements Runnable {
private BufferedReader bufferedReader;
public LinePrinterJob(BufferedReader bufferedReader) {
this.bufferedReader = bufferedReader;
}
#Override
public void run() {
String currentLine;
try {
currentLine = bufferedReader.readLine();
System.out.println(currentLine);
} catch (IOException e) {
e.printStackTrace();
}
}
}
public FileReaderRoundRobinUsingPhaser(int numberOfFilesToRead, int numberOfLinesToRead) {
this.numberOfLinesToRead = numberOfLinesToRead;
String fileLocation = "src/temp/";
for(int j=0; j<(numberOfFilesToRead-1); j++ ){
try {
tasks.add(new LinePrinterJob(new BufferedReader(new FileReader(fileLocation + "Temp" + j))));
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}
public void startPrinting( ) {
final Phaser phaser = new Phaser(1){
#Override
protected boolean onAdvance(int phase, int registeredParties) {
System.out.println("Phase Number: " + phase +" Registeres parties: " + getRegisteredParties() + " Arrived: " + getArrivedParties());
return ( phase >= numberOfLinesToRead || registeredParties == 0);
}
};
for(Runnable task : tasks) {
phaser.register();
new Thread(() -> {
do {
phaser.arriveAndAwaitAdvance();
task.run();
} while(!phaser.isTerminated());
}).start();
}
phaser.arriveAndDeregister();
}
public static void main(String[] args) {
FileReaderRoundRobinUsingPhaser fileReaderRoundRobin = new FileReaderRoundRobinUsingPhaser(4, 4);
fileReaderRoundRobin.startPrinting();
// Files will be accessed in round robin fashion but not exactly in same order always. For example it can read 4 files as 1234 then 1342 or 1243 etc.
}
}
The above example can be modified as per exact requirement. Here the constructor of FileReaderRoundRobinUsingPhaser takes the number of files and number of lines to read from each file. Also the boundary conditions need to be taken into consideration.
You are missing many parts of the puzzle:
you attempt to synchronize on an object local to each thread. This can have no effect and the JVM may even remove the whole locking operation;
you execute notifyAll without a matching wait;
the missing wait must be at the top of the run method, not at the bottom as you indicate.
Altogether, I'm afraid that fixing your code at this point is beyond the scope of one StackOverflow answer. My suggestion is to first familiarize yourself with the core concepts: the semantics of locks in Java, how they interoperate with wait and notify, and the precise semantics of those methods. An Oracle tutorial on the subject would be a nice start.