I am using a simple thread pool to read in a large dictionary I have written a for loop that loops through the whole dictionary (10000 words), I have tried to get it so it stores every 500 into a thread assigning that sub arraylist to a single thread to handle.
When there are 500 words within the arraylist it then stores it within an instance of a class 'Words'. This simply stores and allows access to an arrayList (the arrayList assigned to that thread).
This doesnt appear to be working as there are duplicates and most the time the last 500 words in the dictionary are what all the threads end up using which I find strange. In addition I also noticed when I add in a simple timeout for 3 seconds at the end of the inside of that for loop shown below it works however this seems like a horrible fix and I want this program to be as efficent and fast as possible.
// Executor Program
ExecutorService executor = Executors.newFixedThreadPool(cores);
ArrayList<String> words123 = new ArrayList<String>();
for (int i = 0; i < dictionary.size(); i++) {
words123.add(dictionary.get(i));
if(words123.size() == 1000) {
Words wordsList = new Words(words123);
Runnable worker = new WorkerThread(wordsList, passwords, stp);
executor.execute(worker);
words123 = new ArrayList<String>();
}
}
executor.shutdown();
//wait for all jobs to complete
while (!executor.isTerminated()) {
}
System.out.println("Finished all threads");
// WORD OBJECT ------------------
public class Words {
public static ArrayList<String> words = new ArrayList<String>();
public Words(ArrayList words) {
this.words = words;
}
public int getSize() {
return words.size();
}
public String getWord(int i) {
return words.get(i);
}
}
//WORKER THREAD ----------------
public static Words wordList;
public static int cmd;
public static HashMap<String, String> passwords = new HashMap<String, String>();
public static SimpleThreadPool stp;
/**
* Constructor
* #param s
*/
public WorkerThread(Words word, HashMap passwords, SimpleThreadPool stp, int cmd){
this.wordList = word;
//System.out.println("in " + words);
//Read in hashes using readFromFile method
this.passwords = passwords;
this.stp = stp;
this.cmd = cmd;
}
/**
* For a thread pool to function, ensure that the run() method terminates
* This method prints out the command, calls a function, then prints end and terminates
*/
#Override
public void run() {
//System.out.println(Thread.currentThread().getName()+" Start.");
//System.out.println("WOOOO " + wordList.getWords() + cmd);
for(int i = 0; i < wordList.getSize(); i++){
Password pass = new Password(wordList.getWord(i), hashPassword(wordList.getWord(i)));
//System.out.println(pass.getOriginalPass());
//checkHash(pass);
// Check password with letter-number edits (e.g. a-->#)
letterSymbolEdit(pass);
// Check password with capital letter edits
//capitalsEdit(pass);
// Reverse password
reverseEdit(pass);
// Concatenate all words in dictionary
//concatEdit(pass);
printPermutations(pass);
// All possible numbers generated and appended to string
for(int j = 0; j < 4; j++){
numberBuilder("", 0, j, pass);
}
}
//System.out.println(Thread.currentThread().getName()+" End.");
}
The issue is that the 'words' variable in the Words class is static, that means that every instance of that class is using the same list.
Also, since you're trying to get efficient, I would use a different approach. Instead of
ExecutorService executor = Executors.newFixedThreadPool(cores);
use
LinkedBlockingQueue<Runnable> workQueue = new LinkedBlockingQueue<>();
ThreadPoolExecutor executor = new ThreadPoolExecutor(cores, cores, 0L, TimeUnit.MILLISECONDS, workQueue);
executor.prestartAllCoreThreads();
and then add Runnable instances directly to the workQueue. In this way you don't have to wait to divide words among threads yourself: threads will fetch them as soon as they complete a task.
Related
There is one ArrayList with 1 million element and we are using two threads to read from this ArrayList. The first thread will read first half of the list and second thread will read the second half of list and I am using two threads to achieve this, but I don't see any difference in performance between using one thread and two threads.
I have written below program to achieve this, but I am not sure If this is the right way to implement and achieve this.
Can someone check if my code is correct or how I can fix the multithreading?
import java.util.ArrayList;
import java.util.List;
public class ThreadTask {
public static void main(String[] args) throws InterruptedException {
// TODO Auto-generated method stub
List<Integer> list = new ArrayList<>();
for(int i = 0; i <=1000000; i++){
list.add(i);
}
Thread t1 = new Thread(new PrintList(list));
Thread t2 = new Thread(new PrintList(list));
t1.setName("thread1");
t2.setName("thread2");
long starttime = System.currentTimeMillis();
System.out.println(starttime);
t1.start();
t2.start();
t1.join();
t2.join();
long endtime = System.currentTimeMillis();
System.out.println(endtime);
System.out.println("Total time "+(endtime - starttime));
}
}
class PrintList implements Runnable{
private List list = new ArrayList();
public PrintList(List list){
this.list = list;
}
#Override
public void run() {
if(Thread.currentThread().getName() != null && Thread.currentThread().getName().equalsIgnoreCase("thread1")){
for(int i = 0; i< list.size()/2;i++){
// System.out.println("Thread 1 "+list.get(i));
}
}else if(Thread.currentThread().getName() != null && Thread.currentThread().getName().equalsIgnoreCase("thread2")){
for(int i = list.size()/2; i<list.size(); i++){
//System.out.println("Thread 2 "+list.get(i));
}
}
}
}
Also, If someone can help me on how can we implement it to make it generic to use more then to thread.
System.out.println is synchronized internally (in order that you don't get mixing between the messages printed by multiple threads), so only one thread is actually printing at once.
Basically, it behaves like a single thread.
Even if in reality System.out is synchronized, still you dont want to have manually initialized threads reading from your ArrayList. Plus I doubt that your end goal is the System.out. You should use a higher abstraction. Such abstraction can easily be present either through Java8 Stream API either by ExecutorServices.
Here is one example of paralelism with Java 8 api.
Arraylist toprint;
toPrint.parallelstream().forEach(DoSometing);
This will work in parallel threads.
If you use ExecutorService You can slice your Arraylist and pass each slice to a Callable to perform the work for you in a separate thread.
class Task implements Callable {
List sublist;
public Task(List sublist) {
this.sublist = sublist;
}
public void call() {
// do something
}
}
ArrayList listToSlice;
List<List> slicedList;
ExecutorService executor = Executors.newFixedThreadPool(2);
for (List sublist:slicedList) {
Future<Integer> future = executor.submit(new Task(sublist));
......
.......s on
}
I'm having a difficult time understanding how to synchronise an ArrayList over two threads. Basically, I want one thread appending objects to the list and the other one reading from that list at the same time.
Here is the class that deploys the threads:
public class Main {
public static ArrayList<Good> goodList = new ArrayList();
public static void main(String[] args) {
Thread thread1 = new Thread(new GoodCreator());
Thread thread2 = new Thread(new WeightCounter());
thread1.start();
thread2.start();
}
}
Then the two Runnable classes:
This one reads lines of two values from a text file and appends new objects.
public class GoodCreator implements Runnable{
private ArrayList<Good> goodList = Main.goodList;
private static Scanner scan;
#Override
public void run() {
System.out.println("Thread 1 started");
int objCount = 0;
try {
scan = new Scanner(new File(System.getProperty("user.home") + "//Goods.txt"));
} catch (FileNotFoundException e) {
System.out.println("File not found!");
e.printStackTrace();
}
while(scan.hasNextLine()){
String line = scan.nextLine();
String[] words = line.split("\\s+");
synchronized(goodList){
goodList.add(new Good(Integer.parseInt(words[0]), Integer.parseInt(words[1])));
objCount++;
}
if(objCount % 200 == 0) System.out.println("created " + objCount + " objects");
}
}
}
This iterates over the arraylist and is supposed to sum up one of the fields.
public class WeightCounter implements Runnable{
private ArrayList<Good> goodList = Main.goodList;
#Override
public void run() {
System.out.println("Thread 2 started");
int weightSum = 0;
synchronized(goodList){
for(Good g : goodList){
weightSum += g.getWeight();
}
}
System.out.println(weightSum);
}
}
No matter the input, weightSum never gets incremented and stays 0
Thread 1 started
Thread 2 started
0
Any help is much appreciated
You are running two independently running threads. These thread can run in any order and if one stop e.g. to read from a file, the other thread doesn't assume it has to wait for it.
In short, your second thread completes before the first thread has added anything to the list.
There is no good fix as this is not a good example of why you would use multiple threads, however to get an outcome what you can do is this.
public class WeightCounter implements Runnable{
private ArrayList<Good> goodList = Main.goodList;
#Override
public void run() {
System.out.println("Thread 2 started");
for(int i = 0; i < 10; i++) {
try {
Thread.sleep(100);
} catch (InterruptedException ie) {
throw AssertionError(ie);
}
int weightSum = 0;
synchronized(goodList){
for (Good g : goodList)
weightSum += g.getWeight();
}
System.out.println(weightSum);
}
}
}
This will print the sum 10 times, 0.1 seconds apart. Depending on how long your file takes to load you will be able to see the sum for what has loaded so far.
This is something called a producer-consumer task. You can do it with arraylist, but it's honestly just not the right way to approach this problem.
Luckily, Java provides us with some collections, the BlockingQueue collections, which are designed specifically for this reason;
//the collection with the stuff in it
static BlockingQueue<Object> items = new BlockingQueue<Object>();
//(there are a few different types of blocking queues, check javadocs.
//you would want Linked or Array blocking queue
//what happens on the reader thread
public void producer()
{
//read the data into the collection
for (all the data in the file)
{
//add the next item
items.put(/* next item from file or w/e */);
//stop if necessary
if (atEndOfFile) stillReadingData = false;
//etc
}
}
Now you need to read the data out of the queue - luckily this is easy enough;
//what happens on the other threads
public void consumer()
{
//keep this thread alive so long as there is data to process
//or so long as there might be more data to process
while (stillReadingData || !items.isEmpty())
{
//get the next item from the list
//while the list is empty, we basically sleep for "timeout" timeunits,
//then the while-loop would repeat, and so on
Object o = items.poll(long timeout, int units);
if (o != null) //process it
}
}
In this way, you can continuously add items to the queue with the producer thread, and the items will be processed as soon as a consumer thread is free (this approach scales well with lots of consumer threads). If you still need a collection for the items, then you should make a second collection and add them to that after they have been processed.
As a side note, you may still need to synchronize oprations which occur while processing the items. For example, you would need to synchronize increments on "weightSum" (or alternately use AtomicInteger).
Try this change in the WeightCounter class.
public class WeightCounter implements Runnable{
private ArrayList<Good> goodList = Main.goodList;
#Override
public void run() {
System.out.println("Thread 2 started");
int weightSum = 0;
while(goodList.isEmpty()) {
Thread.sleep(1000);
}
synchronized(goodList){
for(Good g : goodList){
weightSum += g.getWeight();
}
}
System.out.println(weightSum);
}
}
This change will cause the WeightCounter thread to wait for the other thread to finish populating the goodList with data before attempting to read from it.
I have searched the web for a while now trying to resolve this issue, but have had no success.
In my application, I have a large set of messages that I am attempting to encrypt using a basic commutative encryption scheme. Since the sets are large numbers of BigIntegers, I am attempting to multithread the encryptions to increase performance.
Basically, I take the large set of messages and split it up into subsets that are passed to an encryption thread to do a subset of the encryptions. Then I attempt to extract each subset and aggregate them into the original large set after the threads have all done their parts.
When I iterate over the threads and pull out each of their encryptions, the error is occurring when I attempt to actually addAll of the encryptions to the list of all encryptions and the error it throws is the java.util.ConcurrentModificationException error.
I have attempted to use synchronization, but it isn't helping.
Here is the function call:
protected Set<BigInteger> multiEncrypt(BigInteger key, HashSet<BigInteger> messageSet) {
ArrayList<BigInteger> messages = new ArrayList<BigInteger>(messageSet);
Set<BigInteger> encryptions = Collections.synchronizedSet(new HashSet<BigInteger>());
int cores = Runtime.getRuntime().availableProcessors();
int numMessages = messages.size();
int stride = numMessages/cores;
//create all the threads and run them
ArrayList<EncryptThread> threads = new ArrayList<EncryptThread>();
for (int thread = 0; thread < cores; thread++) {
int start = thread*stride;
//don't want to go over the end
int stop = ((thread+1)*stride >= messages.size()) ? messages.size()-1 : (thread+1)*stride;
List<BigInteger> subList = messages.subList(start, stop);
EncryptThread t = new EncryptThread(encryptionScheme.getPrime(), key, subList);
t.start();
threads.add(t);
}
//pull out the encryptions
synchronized(encryptions){
for (int i=0; i < threads.size()-1; i++) {
EncryptThread thread = threads.get(i);
ArrayList<BigInteger> these = thread.getEncryptions();
encryptions.addAll(these); //<-- Erroring Here
thread.finish();
}
}
And here are the relevant parts of the EncryptThread class I wrote to do the encryptions:
/**
* Constructor
*/
public EncryptThread(BigInteger prime, BigInteger key, List<BigInteger> messages) {
//need a new encryption scheme object for each thread
encryptionScheme = new EncryptionScheme(prime);
encryptions = new ArrayList<BigInteger>();
this.key = key;
this.messages = messages;
wait = true;
}
#Override
public void run() {
encryptMessages(key, messages);
while(wait);
}
/**
* Used to encrypt a set of messages
* #param key
* #param messages
* #return
*/
public void encryptMessages(BigInteger key, List<BigInteger> messages) {
System.out.println("Encrypting stuff");
for (BigInteger m : messages) {
BigInteger em = encryptionScheme.encrypt(key, m);
encryptions.add(m);
}
}
public ArrayList<BigInteger> getEncryptions() {
return encryptions;
}
//call this after encryptions have been pulled to let the thread finish
public void finish() {
wait = false;
}
}
I am not new to Java, but I am new to multi threading in java and so I would appreciate any and all advice. Thanks in advance!
EDIT: As per the suggestions, I added a simple locking mechanism to the EncryptThread class, which makes the thread wait to return the encryptions until they are all done and it works now.
public void encryptMessages(BigInteger key, List<BigInteger> messages) {
System.out.println("Encrypting stuff");
this.lock = true;
for (BigInteger m : messages) {
BigInteger em = encryptionScheme.encrypt(key, m);
//deals with when we have to mark chaff at S2
if (shift) {
em.shiftLeft(1);
if(shiftVal != 0) em.add(BigInteger.ONE);
}
encryptions.add(m);
}
this.lock = false;
}
public ArrayList<BigInteger> getEncryptions() {
while(lock);
return encryptions;
}
EDIT #2 So I ended up using a solution which was suggested to me by someone from my lab. I got rid of the lock and wait booleans, and the finish() function in the EncryptThread class, and instead added a simple thread.join() loop between the start and getEncryption loops:
//create all the threads
ArrayList<EncryptThread> threads = new ArrayList<EncryptThread>();
for (int thread = 0; thread < cores; thread++) {
int start = thread*stride;
//don't want to go over the end
int stop = ((thread+1)*stride >= messages.size()) ? messages.size()-1 : (thread+1)*stride;
List<BigInteger> subList = messages.subList(start, stop);
EncryptThread t = new EncryptThread(encryptionScheme.getPrime(), key, subList, shiftVal);
t.start();
threads.add(t);
}
//wait for them to finish
for( EncryptThread thread: threads) {
try {
thread.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
//pull out the encryptions
for (int i=0; i < threads.size()-1; i++) {
EncryptThread thread = threads.get(i);
encryptions.addAll(thread.getEncryptions());
}
I think my main confusion was that I thought a thread class couldn't have its methods called on it after it had finished running. But the above works fine.
ConcurrentModificationException happens when you modify a Collection while you're iterating over it. It has very little to do with multi threading, since you can easily create a single threaded example:
ArrayList<String> myStrings = new ArrayList<>();
myStrings.add("foo");
myStrings.add("bar");
for(String s : myStrings) {
myStrings.add("Hello ConcurrentModificationException!");
If you look at the documentation on List's addAll, it says the following:
Appends all of the elements in the specified collection to the end of this list, in the order that they are returned by the specified collection's iterator (optional operation). The behavior of this operation is undefined if the specified collection is modified while the operation is in progress. (Note that this will occur if the specified collection is this list, and it's nonempty.)
You can see your List being modified while addAll is using it's iterator in your encryptMessages method that one of your threads you spawned is currently executing.
for (BigInteger m : messages) {
BigInteger em = encryptionScheme.encrypt(key, m);
encryptions.add(m); // <-- here
}
I didn't look through all of your code fully, but some of the stuff here is not thread safe. You might do well using a CopyOnWriteArrayList instead of a regular ArrayList to avoid the ConcurrentModificationException, that's if, you are okay with not having everything added to the list in the addAll call, if you aren't, you also then need to be waiting for the threads to finish. You probably want to instead just use tasks with an ExecutorService. There's other improvements to make as well probably.
In additional, the goto book everyone mentions to learn how to write thread safe programs in Java is Concurrency in Practice, I'd recommend that if you are new to concurrency in Java.
you are starting your threads here.
for (int thread = 0; thread < cores; thread++) {
int start = thread*stride;
//don't want to go over the end
int stop = ((thread+1)*stride >= messages.size()) ? messages.size()-1 : (thread+1)*stride;
List<BigInteger> subList = messages.subList(start, stop);
EncryptThread t = new EncryptThread(encryptionScheme.getPrime(), key, subList);
t.start();
threads.add(t);
}
Well. Then you have to wait for all threads to get complete , before start aggregate in this block.
//pull out the encryptions
synchronized(encryptions){
for (int i=0; i < threads.size()-1; i++) {
EncryptThread thread = threads.get(i);
ArrayList<BigInteger> these = thread.getEncryptions();
encryptions.addAll(these); //<-- Erroring Here
thread.finish();
}
}
you are blocking threads which are accessing encryptions only. but the thread you have created is not accessing the set . mean time it will keep on add to its own array List these . So when you call encryptions.addAll(these); these is accessed by two threads ( thread owning encryptions and the thread owning these
And the other answers provided detail about why Concurrent exception in addAll.
You have to wait until all the threads get complete thier work.
You can do this using ExecutorService
Change your starting thread as
ExecutorService es = Executors.newFixedThreadPool(cores);
for(int i=0;i<5;i++)
es.execute(new Runnable() { /* your task */ }); //EncryptThread instance
es.shutdown();
boolean finshed = es.awaitTermination(1, TimeUnit.MINUTES);
Then process your adding back process.
ExecutorService es = Executors.newFixedThreadPool(cores);
for (int thread = 0; thread < cores; thread++) {
int start = thread*stride;
//don't want to go over the end
int stop = ((thread+1)*stride >= messages.size()) ? messages.size()-1 : (thread+1)*stride;
List<BigInteger> subList = messages.subList(start, stop);
EncryptThread t = new EncryptThread(encryptionScheme.getPrime(), key, subList);
es.execute(t);
threads.add(t);
}
es.shutdown();
boolean finshed = es.awaitTermination(1, TimeUnit.MINUTES);
//pull out the encryptions
synchronized(encryptions){
for (int i=0; i < threads.size()-1; i++) {
EncryptThread thread = threads.get(i);
ArrayList<BigInteger> these = thread.getEncryptions();
encryptions.addAll(these); //<-- Erroring Here
thread.finish();
}
}
Assumed, your EncryptThread is Thread right now. you might need to change to implements Runnable. and no other change in getEncryptions
I have a class that basically has two methods the first one takes a String (name of a file) and a thread:
public static void readFile(String s, Thread t){
Runnable read = new Runnable() {
public void run() {
//SOME CODE
}
t = new Thread(read);
t.start();
}
The second method is a main method that asks the user for input and then uses that input to set a few things (like if the number of threads is just one or if it is equal to the number of objects in a list).
public static void main(String[] args){
//SOME CODE
for(Object x: ListOfObjects){
//t1 is the same thread each time if one thread requested, otherwise t1 is a different thread each time
readFromFile(textFileString, t1);
//SOME CODE
}
If the user were to request 5 threads (for 5 items in a list), how could the above be modified for that? Currently, my main method has a loop (for the number of items in the list) and then the first method is called for each iteration in the loop. Is there a way to take the number of threads requested by the user, and initiate/start them in the first method all at once instead of one at a time and calling the method?
Implement the Runnable interface. I tried
this and it seems to work:
class StringThread implements Runnable
{
private String str;
private int num;
StringThread(String s, int n)
{
str = new String (s);
num =n;
}
public void run ( )
{
for (int i=1; i<=num; i++)
System.out.print ("THREAD NAMED: " + str+" I IS: " +
i + "\n");
}
}
//IN the main program:
StringThread t1 = new StringThread ("THR-1",100);
new Thread(t1). start ( );
StringThread t2 = new StringThread ("THR-2",200);
new Thread(t2). start ( );
I have encountered a problem when I want to do a time counting.
Basicly the problem is like this: there is a class A, which initiates a private thread in itself, and I have a instant of A in my class B, and in the main method of B I invoked some methods of A and want to test the time to run these methods.
A a = new A();
//start time counter
for (int i = 0; i < 10; i++){ invoke a.method() that takes some time}
//end time counter and prints the time elapsed
but by doing so the method in the for loop will running in a seperate thread in A and the prints method in the last line would probably be executed before the loop ends. So I want to access the thead in a and invokes a join() to wait until all stuff in the for loop get finished. Could you help me figure how to achieve this? Any ideas would be greatly appreciated.
List All Threads and their Groups
public class Main
{
public static void visit(final ThreadGroup group, final int level)
{
final Thread[] threads = new Thread[group.activeCount() * 2];
final int numThreads = group.enumerate(threads, false);
for (int i = 0; i < numThreads; i++)
{
Thread thread = threads[i];
System.out.format("%s:%s\n", group.getName(), thread.getName());
}
final ThreadGroup[] groups = new ThreadGroup[group.activeGroupCount() * 2];
final int numGroups = group.enumerate(groups, false);
for (int i = 0; i < numGroups; i++)
{
visit(groups[i], level + 1);
}
}
public static void main(final String[] args)
{
ThreadGroup root = Thread.currentThread().getThreadGroup().getParent();
while (root.getParent() != null)
{
root = root.getParent();
}
visit(root, 0);
}
}
Based on your edits, you might can find out what group and name the thread is and get a reference to it that way and do what you need to do.
For your own code in the future
You want to look at ExecutorCompletionService and the other thread management facilities in java.util.concurrent. You should not be managing threads manually in Java anymore, pretty much every case you can imagine is handled one or more of the ExecutorService implementations.