I am using Fork join pool in java for multitasking. Now i came across a situation where, for every task, I need to hit a url then wait for 10 minutes and then again hit another url to read the data. Now the problem is that for those 10 minutes my CPU is idle and not starting another tasks ( more than those defined in fork join pool).
static ForkJoinPool pool = new ForkJoinPool(10);
public static void main(String[] args){
List<String> list = new ArrayList<>();
for(int i=1; i<=100; i++){
list.add("Str"+i);
}
final Tasker task = new Tasker(list);
pool.invoke(task);
public class Tasker extends RecursiveAction{
private static final long serialVersionUID = 1L;
List<String> myList;
public Tasker(List<String> checkersList) {
super();
this.myList = checkersList;
}
#Override
protected void compute() {
if(myList.size()==1){
System.out.println(myList.get(0) + "start");
//Date start = new Date();
try {
Thread.sleep(10*60*1000);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println(myList.get(0) + "Finished");
}
else{
List<String> temp = new ArrayList<>();
temp.add( myList.get( myList.size()-1 ) );
myList.remove( myList.size()-1 );
Tasker left = new Tasker(myList);
Tasker right = new Tasker(temp);
left.fork();
right.compute();
left.join();
}
}
Now What should I do so that CPU picks all the tasks and then wait parallaly for them.
Unfortunately, ForkJoinPool does not work well in the face of Thread.sleep(), because it designed for many short tasks that finish quickly, rather than tasks that block for a long time.
Instead, for what you are trying to accomplish, I would recommend using ScheduledThreadPoolExecutor and dividing your task into two parts.
import java.util.*;
import java.util.concurrent.*;
public class Main {
static ScheduledThreadPoolExecutor pool = new ScheduledThreadPoolExecutor(10);
public static void main(String[] args){
for(int i=1; i<=100; i++){
pool.schedule(new FirstHalf("Str"+i), 0, TimeUnit.NANOSECONDS);
}
}
static class FirstHalf implements Runnable {
String name;
public FirstHalf(String name) {
this.name = name;
}
public void run() {
System.out.println(name + "start");
pool.schedule(new SecondHalf(name), 10, TimeUnit.MINUTES);
}
}
static class SecondHalf implements Runnable {
String name;
public SecondHalf(String name) {
this.name = name;
}
public void run() {
System.out.println(name + "Finished");
}
}
}
If Java provides a thread pool which allows releasing the underlying resources (that is, the kernel thread participating in the thread pool) during a Thread.sleep(), you should use that instead, but I currently do not know of one.
According to docs forkJoin basic use section tells:
if (my portion of the work is small enough)
do the work directly
else
split my work into two pieces
invoke the two pieces and wait for the results
Hopefully this meets your need if you are using forkjoin
public class Tasker extends RecursiveAction {
static ForkJoinPool pool = new ForkJoinPool(10);
static int threshold = 10;
public static void main(String[] args){
List<String> list = new ArrayList<>();
for(int i=1; i<=100; i++){
list.add("Str"+i);
}
final Tasker task = new Tasker(list);
pool.invoke(task);
}
private static final long serialVersionUID = 1L;
List<String> myList;
public Tasker(List<String> checkersList) {
super();
this.myList = checkersList;
}
void computeDirectly() {
for(String url : myList){
System.out.println(url + " start");
}
//Date start = new Date();
try {
//keep hitting url
while (true) {
for(String url : myList) {
//url hitting code here
System.out.println(url + " hitting");
}
Thread.sleep(10 * 60 * 1000);
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
for(String url : myList){
System.out.println(url + " Finished");
}
}
#Override
protected void compute() {
if (myList.size() <= threshold) {
computeDirectly();
return;
}
//temp list have only one url
//List<String> temp = new ArrayList<>();
//temp.add( myList.get( myList.size()-1 ) );
//myList.remove( myList.size()-1 );
//Tasker left = new Tasker(myList);
//Tasker right = new Tasker(temp);
//left.fork();
//right.compute();
//left.join();
List<String> first = new ArrayList<>();
List<String> second = new ArrayList<>();
//divide list
int len = myList.size();
int smHalf = len / 2;//smaller half
first = myList.subList(0, smHalf);
second = myList.subList(smHalf + 1, len);
invokeAll(new Tasker(first), new Tasker(second));
}
}
Related
I am expecting 5 results i.e. 1, 2, 3, 4, 5 yet I am getting far more? can someone please help me understand semaphores? surely as each thread calls the run method using "start", it should add 1
to the int aNumber and then print aNumber.
I don't understand why I get more than five results.
// mainClass creates Semaphore, so Copier class no longer needed
import java.util.concurrent.Semaphore;
public class mainClass {
public static void main(String[] args) throws InterruptedException {
Semaphore cp1 = new Semaphore(1, true);
Worker[] workers = new Worker[5];
for (int x=0;x<5;x++) {
workers[x] = new Worker("w" + x, cp1);
}
for (int x=0;x<5;x++) {
workers[x].start();
}
for (int x=0;x<5;x++) {
workers[x].join();
}
}
}
import java.util.concurrent.Semaphore;
public class Worker extends Thread{
int aNumber = 0;
String myName;
Semaphore myCopier;
public Worker(String nameIn, Semaphore copierIn) {
myName = nameIn;
myCopier = copierIn;
}
public void run() {
for (int x=0;x<5;x++) {
try {
sleep((int) (Math.random() * 5000)); // do some filing
myCopier.acquire();
aNumber +=1;
//System.out.println(myName + " doing some copying");
System.out.println(aNumber);
sleep((int) (Math.random() * 1000)); // do some copying
myCopier.release();
} catch (InterruptedException e) { }
}
}
}
I could not clearly get your intention. Are you trying to use semaphores to print the nos ( 1 to 5) in sequence by each thread? In that case, you can try the below:
//mainClass creates Semaphore, so Copier class no longer needed
import java.util.concurrent.Semaphore;
public class mainClass {
public static void main(String[] args) throws InterruptedException {
Semaphore cp1 = new Semaphore(1, true);
Worker[] workers = new Worker[5];
for (int x=0;x<5;x++) {
workers[x] = new Worker("w" + x, cp1, x+1);
}
for (int x=0;x<5;x++) {
workers[x].start();
}
for (int x=0;x<5;x++) {
workers[x].join();
}
}
}
class Worker extends Thread{
int aNumber = 0;
String myName;
Semaphore myCopier;
public Worker(String nameIn, Semaphore copierIn, int no) {
myName = nameIn;
myCopier = copierIn;
aNumber = no;
}
public void run() {
/*for (int x=0;x<5;x++) {*/
try {
//sleep((int) (Math.random() * 5000)); // do some filing
myCopier.acquire();
//System.out.println(myName + " doing some copying");
System.out.println("Name of the thread:" + myName + " Printed No is:" + aNumber);
//sleep((int) (Math.random() * 1000)); // do some copying
myCopier.release();
} catch (InterruptedException e) { }
//}
}
}
I have a fairly typical producer-consumer scenario where I have 1 producer thread that executes queries and puts the results to a BlockingQueue and approx 7-8 consumers that pick these objects from the BlockingQueue and run a rather long lasting analysis on them. Once these analyses are completed, the resultant objects are placed in a HashMap with the original objects as keys, i.e. HashMap<AnalyzedObject, AnalysisResult>
Due to the nature of relationships in the underlying data model, I get a lot of duplicate tasks, which obviously do not need to be reprocessed. My current solution is essentially as follows:
public class AnalysisAction implements Runnable{
private Dataset data;
private DbManager dbManager;
private Path path;
private Set<Integer> identifiedElements;
private AnalysisResult res;
private Map<Path, AnalysisResult> analyzedPaths;
public static final AtomicInteger duplicates = new AtomicInteger(0);
public AnalysisAction(Path p, Dataset ds, DbManager dbm, Map<Path, AnalysisResult> paths){
this.data = ds;
this.path = p;
this.dbManager = dbm;
this.analyzedPaths = paths;
this.res = new AnalysisResult(path);
}
#Override
public void run() {
if(!analyzedPaths.containsKey(path)){
t0 = System.currentTimeMillis();
// 1. Check the coverage of the path
this.identifiedElements = getIdentifiedElements();
if(identifiedElements.size() != 0)
{
try{
// TIME CONSUMING STUFF...
analyzedPaths.put(path, res);
}
catch(Exception e){
// Exception handling...
}
}
t_end = System.currentTimeMillis();
DebugToolbox.submitProcTime(t_end - t0);
}
else {
duplicates.incrementAndGet();
logger.finer("Duplicate path encountered..." + System.lineSeparator());
}
}
// PRIVATE METHODS THAT CARRY OUT THE TIME CONSUMING STUFF...
}
Then within the class that controls the multithreading I have the following solution:
public class ConcurrencyService {
private final ThreadPoolExecutor pool;
private final int poolSize;
private final int qCapacity = 1 << 7;
private final long timeout = 3;
private final Path tainedPath =
new Path(Long.MIN_VALUE, "LAST_PATH_IN_QUEUE", "N/A", "N/A");
private BlockingQueue<PathwayImpl> bq;
private DbManager dbMan;
private Dataset ds;
private Map<Path,AnalysisResult> analyzedPaths;
private volatile boolean started;
public ConcurrencyService(Dataset data, DbManager db){
this.ds = data;
this.bq = new LinkedBlockingQueue<Path>(qCapacity);
this.dbMan = db;
this.analyzedPaths = new ConcurrentHashMap<Path,AnalysisResult>(1<<15);
this.started = false;
poolSize = Runtime.getRuntime().availableProcessors();
pool = (ThreadPoolExecutor) Executors.newFixedThreadPool(poolSize, new FThreadFactory(-1));
}
public void serve() throws InterruptedException {
try {
ds.finalize();
started = true;
Thread producerThread = new Thread(new QueryingAction(), "f-query-thread");
producerThread.start();
Thread loggerThread = new Thread(new PeriodicLogAction(null), "f-logger-thread");
loggerThread.start();
while((producerThread.getState() != Thread.State.TERMINATED) || !bq.isEmpty()){
Path p = bq.poll(timeout, TimeUnit.MINUTES);
if(p != null){
if (p.equals(tainedPath)) break;
pool.submit(new AnalysisAction(p, ds, dbMan, analyzedPaths));
}else
logger.warning("Timed out while waiting for a path...");
}
} catch (Exception ex) {
// Exception handling...
} finally{
pool.shutdown();
long totalTasks = pool.getTaskCount(),
compTasks = pool.getCompletedTaskCount(),
tasksRemaining = totalTasks - compTasks,
timeout = 10 * tasksRemaining / poolSize;
pool.awaitTermination(timeout, TimeUnit.SECONDS);
logger.info(
"A total of " + DebugToolbox.getNbrProcTimes()
+ " tasks analyzed. Mean process time is: "
+ DebugToolbox.getMeanProcTimeAsString()
+ " milliseconds." + System.lineSeparator());
}
public boolean isDone(){
if(this.started)
return pool.isTerminated();
else
return false;
}
}
protected class QueryingAction implements Runnable {
// Use this to limit the number of paths to be analyzed
// private final int debugLimiter = 1500;
private final int debugLimiter = Integer.MAX_VALUE;
public void run() {
try {
int i = 0;
outer: for(String el : ds.getElements()){
inner: for(Path path : dbMan.getAllPathsWithElement(el)){
if(i++ > debugLimiter)
break outer;
else
bq.put(path);
}
}
logger.info("Total number of queried paths: " + i);
} catch (SQLException e) {
// Exception handling...
} catch (InterruptedException e) {
// Exception handling...
}
bq.offer(tainedPath);
}
}
protected class PeriodicLogAction implements Runnable {
private final PrintStream ps;
private final long period;
private final static long DEF_PERIOD = 30000;
private final String nL = System.getProperty("line.separator");
private volatile boolean loop;
private int counter = 0;
private ConcurrencyService cs;
private int inQueryQueue, inPoolQueue,
completedTasks, inProccessedSet,duplicates;
boolean sanityCheck;
StringBuffer sb;
PeriodicLogAction(PrintStream ps, long timePeriod) {
this.ps = ps;
this.period = timePeriod;
this.loop = true;
this.cs = ConcurrencyService.this;
}
// Alternative constructors
#SuppressWarnings("rawtypes")
public void run() {
logger.config("PeriodicLogAction started on thread: " +
Thread.currentThread().getName() +
System.lineSeparator());
while(loop){
// log # of paths created, analyzed and are in queue
outputLogInfo();
// wait designated time period
try {
Thread.sleep(period);
} catch (InterruptedException e) {}
if(cs.isDone()){
this.loop = false;
outputLogInfo();
}
}
}
private void outputLogInfo(){
synchronized (pool) {
Queue queryQueue = cs.bq,
poolQueue = cs.pool.getQueue();
Map<PathwayImpl,AnalysisResult> processedSet = cs.analyzedPaths;
inQueryQueue = queryQueue.size();
inPoolQueue = poolQueue.size();
completedTasks = (int) pool.getCompletedTaskCount();
inProccessedSet = processedSet.size();
duplicates = AnalysisAction.duplicates.get();
sanityCheck = (completedTasks == inProccessedSet + duplicates);
}
sb = new StringBuffer();
sb.append("Checkpoint ").append(++counter).append(": ")
.append("QQ: ").append(inQueryQueue).append("\t")
.append("PQ: ").append(inPoolQueue).append("\t")
.append("CT: ").append(completedTasks).append("\t")
.append("AP: ").append(inProccessedSet).append("\t")
.append("DP: ").append(duplicates).append("\t")
.append("Sanity: ").append(sanityCheck);
if(ps == null)
logger.info(sb.toString() + nL);
else
ps.println(sb.toString());
}
}
}
Here's what I see on the logs:
Sep 17, 2014 5:30:00 PM main.ConcurrencyService$QueryingAction run
INFO: Total number of queried paths: 81128
Sep 17, 2014 5:30:00 PM main.ConcurrencyService serve
INFO: All paths are queried and queued...
Initiating a timely shutdown of the pool..
...
Sep 17, 2014 5:49:49 PM main.ConcurrencyService serve
INFO: A total of 8620 tasks analyzed. Mean process time is: 1108.208 milliseconds.
...
Sep 17, 2014 5:49:54 PM main.ConcurrencyService$PeriodicLogAction outputLogInfo
INFO: Checkpoint 41: QQ: 0 PQ: 0 CT: 81128 AP: 8565 DP: 72508 Sanity: false
... which indicate:
The number of completed tasks is in agreement with the number of objects that were queried and queued. So nothing is missed..
The number of analyzed paths (thus results) and the number of duplicates do not add up to the number of completed tasks: 81128 - (8565 + 72508) = 55
The number of results accumulated does not match with reported process times from AnalysisAction class: 8565 vs 8620 (i.e. there are 55 results missing)
Not sure what could be the cause of this discrepancy, or where to begin debugging. I can't obviously step through 81128 tasks to investigate which 55 are missing, and why..
Any suggestions?
EDIT: here are some clarifications addressing the question in the comments
DebugToolbox.submitProcTimes(long t) is a synchronized static method, it simply adds t to an ArrayList.
isDone() is a method in ConcurrencyService, I had accidentally removed it as I was trying to shorten the code I post here. I've edited the code to reflect how the method is implemented.
You check the map for presence of the key, then spend time generating the value, then put the value into the map.
While you're generating the value, another thread can process the same key. Since it's not been added yet, you now have two threads generating the same value. So the count of values generated is larger than the final size of the map.
The solution is to add the result (may be a placeholder) and check for the key's presence atomically using putIfAbsent().
This is my first posting here, hope I won't seem too desperate with my question.
I have a work task which involves comparing two large set of names to see if matching exists between them (regardless of order of words in the names).
I've tried both a regular, more straightforward approach and also one using Regex.
Standard approach:
public static boolean isMatch(String terroristName, String clientName) {
String[] terroristArray = terroristName.split(" ");
String[] clientArray = clientName.split(" ");
int size = clientArray.length;
int ctrl = 0;
boolean alreadyFound = false;
for (String client : clientArray) {
for (String terrorist : terroristArray) {
//if already found a match, stop comparing with rest of the words from terrorist name
if (!alreadyFound)
if (client.compareTo(terrorist) == 0) {
alreadyFound = true;
ctrl++;
break;
}
}
alreadyFound = false;
if (ctrl == 0 && !alreadyFound) {
//if first word of client is not found in whole terrorist name
//then exit loop, no match possible
break;
}
}
if (ctrl == size)
return true;
else
return false;
}
Regex approach:
public static boolean isRegexMatch(String terroristName, String clientName) {
boolean result = false;
String[] clientNameArray = clientName.split(" ");
String myPattern = "^";
//build pattern using client name
for (String cname : clientNameArray) {
myPattern += "(?=.*\\b" + cname + "\\b)";
}
myPattern += ".*$";
Pattern pattern = Pattern.compile(myPattern);
Matcher matcher = pattern.matcher(terroristName);
// check all occurance
while (matcher.find()) {
result = true;
}
return result;
}
Loop comparing the 2 lists of names:
for (Person terrorist : terrorists) {
System.setOut(matchPrintStream);
for (Person client : clients) {
if (Util.isRegexMatch(terrorist.getNoDuplicatesName(), client.getName())) {
System.out.println(client.getId() + ";" + client.getName() + ";" + terrorist.getId() + ";" +
terrorist.getName());
}
}
}
The two sets have the following sizes:
terrorist = aprox 16000
clients = aprox 3.4 million
Runtime of both methods is quite slow:
ps -ef | grep 42810
42810 41919 99 17:47 pts/0 00:52:23 java -Xms1024M -Xmx1024M -classpath ojdbc6.jar:TerroristBuster.jar ro.btrl.mis.mihai.Main
By the time above of 00:52:23 runtime, it had processed about 170 entries, meaning it would need several days to complete. I know it has a large complexity, unsure how to lower it. What do you think of maybe using something other than List? I figured it would the most fast using foreach since of the random access.
Can this code be improved/changed in any way to improve the runtime, or am i just dealing with a too large set of data?
If you can use Java 8 this should be very easy to parallelise.
First, you don't have that many clients, so preprocess those:
final Collection<Collection<String>> processedClients = clients.parallelStream().
map(c -> c.split("\\s+")).
map(Arrays::asList).
collect(toList());
This takes each client name, splits it into the parts, and then uses the asList wrapper to turn it into a List. This is done parallelised, so should be fast.
Next we need to loop over all the terrorists:
terrorists.parallelStream().
map(t -> t.split("\\s+")).
map(t -> Stream.of(t).collect(toSet())).
forEach(t -> {
processedClients.parallelStream().forEach(c -> {
if (t.containsAll(c)) {
System.out.println("Match found t:" + t + ", c:" + c);
}
});
});
Here, for each terrorist, we split their name, but this time we turn it into a Set because Set has O(1) contains() - this means checking whether a whole client name is contained in a whole terrorist name will only take time proportional to the size of the client name.
We then use forEach to loop over the terrorists and another forEach to loop over the clients, we check is the terrorists name Set containsAll the client name.
Again this is in parallel.
In theory it shouldn't take long at all. Storing the processed client names in memory might require a bit of RAM, but it shouldn't be too much - about 1GB.
EDIT
Here is a rewrite to an earlier version (1.7, but if you remove the diamond notation it should work on 1.5)
First you need two processing classes, these are submitted to individual work threads:
final class NameProcessor implements Callable<Collection<String>> {
private final String name;
public NameProcessor(final String name) {
this.name = name;
}
#Override
public Collection<String> call() throws Exception {
return Arrays.asList(name.split("\\s+"));
}
}
final class TerroristProcessor implements Runnable {
private final String name;
public TerroristProcessor(final String name) {
this.name = name;
}
#Override
public void run() {
final Set<String> splitName = new HashSet<>(Arrays.asList(name.split("\\s+")));
for (final Collection<String> client : proccessedClients) {
if (splitName.containsAll(client)) {
System.out.println("Match found t:" + name + ", c:" + client);
}
}
}
}
Now you need to ExecutorService and an ExecutorCompletionService:
final ExecutorService es = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
final ExecutorCompletionService<Collection<String>> cs = new ExecutorCompletionService<>(es);
Now you first need to process your clients, as before:
for (final String name : clients) {
cs.submit(new NameProcessor(name));
}
final Collection<Collection<String>> proccessedClients = new LinkedList<>();
for (int i = 0; i < clients.size(); ++i) {
try {
proccessedClients.add(cs.take().get());
} catch (InterruptedException ex) {
return;
} catch (ExecutionException ex) {
throw new RuntimeException(ex);
}
}
And then process the terrorists:
final Collection<Future<?>> futures = new LinkedList<>();
for (final String terrorist : terrorists) {
futures.add(es.submit(new TerroristProcessor(terrorist)));
}
es.shutdown();
es.awaitTermination(1, TimeUnit.DAYS);
for (final Future<?> f : futures) {
try {
f.get();
} catch (ExecutionException ex) {
throw new RuntimeException(ex);
}
}
The loop over the futures is to check for processing errors.
EDIT
The OP wants to process custom objects rather than collections of String.
I would assume you have some sort of Person class like so:
final class Person {
private final int id;
private final String name;
//constructor
//getters and setters
}
Then you can simply create a wrapper class like so:
final class PersonWrapper {
private final Person person;
private final Collection<String> processedName;
//constructor
//getters and setters
}
And create a result class like so:
final class ProblemClient {
private final Person client;
private final Person terrorist;
//constructor
//getters and setters
}
And simply rewrite the code appropriately:
final class NameProcessor implements Callable<PersonWrapper> {
private final Person person;
public NameProcessor(final Person person) {
this.person = person;
}
#Override
public PersonWrapper call() throws Exception {
return new PersonWrapper(person, Arrays.asList(person.getName().split("\\s+")));
}
}
final ExecutorService es = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
final ExecutorCompletionService<PersonWrapper> cs = new ExecutorCompletionService<>(es);
for (final Person client : clients) {
cs.submit(new NameProcessor(client));
}
final Collection<PersonWrapper> proccessedClients = new LinkedList<>();
for (int i = 0; i < clients.size(); ++i) {
try {
proccessedClients.add(cs.take().get());
} catch (InterruptedException ex) {
return;
} catch (ExecutionException ex) {
throw new RuntimeException(ex);
}
}
final class TerroristProcessor implements Runnable {
private final Person person;
private final Collection<ProblemClient> results;
public TerroristProcessor(final Person person, final Collection<ProblemClient> results) {
this.person = person;
this.results = results;
}
#Override
public void run() {
final Set<String> splitName = new HashSet<>(Arrays.asList(person.getName().split("\\s+")));
for (final PersonWrapper client : proccessedClients) {
if (splitName.containsAll(client.getProcessedName())) {
results.add(new ProblemClient(client.getPerson(), person));
}
}
}
}
final Collection<ProblemClient> results = new ConcurrentLinkedQueue<>();
final Collection<Future<?>> futures = new LinkedList<>();
for (final Person terrorist : terrorists) {
futures.add(es.submit(new TerroristProcessor(terrorist, results)));
}
es.shutdown();
es.awaitTermination(1, TimeUnit.DAYS);
for (final Future<?> f : futures) {
try {
f.get();
} catch (ExecutionException ex) {
throw new RuntimeException(ex);
}
}
//process results
for (final ProblemClient problemClient : results) {
//whatever.
}
As I said, it might be informative to see what the outcome of preprocessing terrorists first and then looping over clients is too:
final class TerroristPreprocessor implements Callable<PersonWrapper> {
private final Person person;
public TerroristPreprocessor(final Person person) {
this.person = person;
}
#Override
public PersonWrapper call() throws Exception {
final Set<String> splitName = new HashSet<>(Arrays.asList(person.getName().split("\\s+")));
return new PersonWrapper(person, splitName);
}
}
final ExecutorService es = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
final ExecutorCompletionService<PersonWrapper> cs = new ExecutorCompletionService<>(es);
for (final Person terrorist : terrorists) {
cs.submit(new TerroristPreprocessor(terrorist));
}
final Collection<PersonWrapper> proccessedTerrorists = new LinkedList<>();
for (int i = 0; i < terrorists.size(); ++i) {
try {
proccessedTerrorists.add(cs.take().get());
} catch (InterruptedException ex) {
return;
} catch (ExecutionException ex) {
throw new RuntimeException(ex);
}
}
final class ProblemClientFinder implements Runnable {
private final Person client;
private final Collection<ProblemClient> results;
public ProblemClientFinder(final Person client, final Collection<ProblemClient> results) {
this.client = client;
this.results = results;
}
#Override
public void run() {
final Collection<String> splitName = Arrays.asList(client.getName().split("\\s+"));
for (final PersonWrapper terrorist : proccessedTerrorists) {
if (terrorist.getProcessedName().containsAll(splitName)) {
results.add(new ProblemClient(client, terrorist.getPerson()));
}
}
}
}
final Collection<ProblemClient> results = new ConcurrentLinkedQueue<>();
final Collection<Future<?>> futures = new LinkedList<>();
for (final Person client : clients) {
futures.add(es.submit(new ProblemClientFinder(client, results)));
}
es.shutdown();
es.awaitTermination(1, TimeUnit.DAYS);
for (final Future<?> f : futures) {
try {
f.get();
} catch (ExecutionException ex) {
throw new RuntimeException(ex);
}
}
//process results
for (final ProblemClient problemClient : results) {
//whatever.
}
I have an pool of objects in a blockingQueue. Now i want to assign objects from the queue to a thread and use it inside the run method.
What is the best way of doing it?
Here is a sample code i am using to build the pool:
public class ConsumerPool {
private static Logger log;
//building consumer Pool
#SuppressWarnings("finally")
public BlockingQueue<OAuthConsumer> buildConsumerPool() {
BlockingQueue<OAuthConsumer> consumerObjectsQueue = null;
try {
//setting the config path
PropertyHandler.setConfigPath(propertiesMain);
String twitterPath = PropertyHandler.getProperty("twitterPath");
//setting config for twitter
PropertyHandler.setConfigPath(twitterPath);
//Blocking Linked Queue
consumerObjectsQueue = new LinkedBlockingQueue<OAuthConsumer>();
//fetching required tokens for all apps
String consumerKeySet = PropertyHandler.getProperty("consumerKey");
String consumerSecretSet = PropertyHandler.getProperty("consumerSecret");
String accessTokenSet = PropertyHandler.getProperty("accessToken");
String tokenSecretSet = PropertyHandler.getProperty("tokenSecret");
String[] splitconsumerKeys = consumerKeySet.split(",");
String[] splitconsumerSecret = consumerSecretSet.split(".");
String[] splitaccessToken = accessTokenSet.split(",");
String[] splittokenSecret = tokenSecretSet.split(".");
//creating consumer objects for each app
for (int i= 0; i< splitconsumerKeys.length; i++) {
log.info("constructing consumer object for twitter api " +i);
String consumerKey = splitconsumerKeys[i];
String consumerSecret = splitconsumerSecret[i];
String accessToken = splitaccessToken[i];
String tokenSecret = splittokenSecret[i];
OAuthConsumer consumer = new CommonsHttpOAuthConsumer(consumerKey, consumerSecret);
consumer.setTokenWithSecret(accessToken, tokenSecret);
consumerObjectsQueue.put(consumer);
log.info("added the consumer object to que pool");
}
} catch (Exception e) {
e.printStackTrace();
} finally {
return consumerObjectsQueue;
}
}
That is used to build the object pool.
Here is the way i want to create threads.
public class MrRunnable implements Runnable {
private String toFireUrl;
MrRunnable(String url){
}
#Override
public void run() {
// do some function here
}
}
public class Main {
public static void main(String[] args) {
// We will create 500 threads
for (int i = 0; i < 500; i++) {
Runnable task = new MrRunnable("some new url");
Thread worker = new Thread(task);
//start the thread
worker.start();
}
}
}
Now i want to access the objects in the pool via a thread. In the main program should i pass the object from the consumer pool to runnable class during the creation of MrRunnable Object or is there any other way i can do it ?
the constructor of MrRunnable should get a reference to the queue
I have a arraylist which consists of 5000 IP Addresses. For each IP Address, I want to execute a SNMPGet request and a FTPDownload command. I want to implement it in a fashion, wherein at a time 2 different threads run simultaneously for the first five IP Addresses. After the execution of these IP Addresses, next 2 IP Addresses will be executed on these threads. Can anyone help how to do it?
Here, connection is a class which extends the thread and the work to be implemented is written in its run() method. Please help.
Connection newConnection =new Connection(0);
Connection newConnection1 =new Connection(1);
for(int i = 0; i < NE_list.getRowCount(); i=i+2)
{
if(NE_list.getValueAt(i, 0).toString().equals("true")) //Some condition here for the IP Address
{
newConnection.i=i;
newConnection1.i=i+1;
newConnection.runprogram();
newConnection1.runprogram();
}
}
class Connection extends Thread{
int i;
Connection(int val){
i=val;
}
void runprogram(){
start();
}
public void run(){
//SNMP and FTP Code here for IP Address in index i of NE_list
}
}
Executor Framework will be best suit for your solution. I have created one example here. You can increase the number of threads as per your requirement.
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
class SomeRunnable implements Runnable {
int threadNo = -1 ;
List<String> list = new ArrayList<String>();
public SomeRunnable(List list, int threadNo ) {
this.list.addAll(list);
this.threadNo =threadNo;
}
#Override
public void run() {
for (String element : list) {
System.out.println("By Thread:" + threadNo+", Processed Element:" +element);
}
}
}
public class ExecutorDemo {
public static void main(String[] args) {
List<String> list = new ArrayList<String>();
for (int i = 0; i < 100; i++) {
list.add("Elem:"+i);
}
// Divide list
int divideIndex = list.size()/2;
//Create objects of Runnable
SomeRunnable obj1 = new SomeRunnable(list.subList(0, divideIndex),1);
SomeRunnable obj2 = new SomeRunnable(list.subList(divideIndex,list.size()),2);
//Create fixed Thread pool, here pool of 2 thread will created
ExecutorService pool = Executors.newFixedThreadPool(2);
pool.execute(obj1);
pool.execute(obj2);
pool.shutdown();
}
}
Herewith adding working example having 5 threads. Just put the test.txt in your CLASS_PATH of the application.
class MyRunnable implements Runnable {
List<List<String>> records;
MyRunnable(List<List<String>> records){
this.records = records;
}
public void run(){
for(List list : records){
System.out.println(Thread.currentThread().getName() + " : "+list.toString());
}
}}
Main Class -
public class FileProcessThreads {
public List<List<String>> process(String fileName) throws IOException {
List<List<String>> records = new ArrayList<>();
BufferedReader br = new BufferedReader(new FileReader(fileName));
String line = null;
while((line = br.readLine()) != null){
List<String> listValues = Arrays.asList(line.split(" "));
records.add(listValues);
}
System.out.println(records.size());
return records;
}
public static void main(String[] args) throws IOException {
FileProcessThreads fp = new FileProcessThreads();
List<List<String>> records = fp.process("test.txt");
ExecutorService es = Executors.newFixedThreadPool(5);
int recordsInEachThread = (int) (records.size()/5);
System.out.println(recordsInEachThread);
MyRunnable my1 = new MyRunnable(records.subList(0, recordsInEachThread));
MyRunnable my2 = new MyRunnable(records.subList(recordsInEachThread+1, recordsInEachThread*2));
MyRunnable my3 = new MyRunnable(records.subList(recordsInEachThread*2 + 1, recordsInEachThread*3));
MyRunnable my4 = new MyRunnable(records.subList(recordsInEachThread*3 + 1, recordsInEachThread*4));
MyRunnable my5 = new MyRunnable(records.subList(recordsInEachThread*4 + 1, records.size() - 1));
es.execute(my1);
es.execute(my2);
es.execute(my3);
es.execute(my4);
es.execute(my5);
es.shutdown();
}}