public class TestConcurrentForList {
List<Integer> mainList = new ArrayList<Integer>();
ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(1);
Random r = new Random();
public void start() throws InterruptedException {
Runnable cmd = new Runnable() {
#Override
public void run() {
List<Integer> tempList = mainList;
mainList = new ArrayList<Integer>();
for (Integer i: tempList) {
System.out.println("subThread:" + i);
}
}
};
scheduledExecutorService.scheduleAtFixedRate(cmd, 1, 1, TimeUnit.MILLISECONDS);
while (true) {
mainList.add(r.nextInt(200));
Thread.sleep(100);
}
}
public static void main(String[] args) {
TestConcurrentForList tester = new TestConcurrentForList();
try {
tester.start();
} catch (Exception e) {
e.printStackTrace();
System.err.println(e.getMessage());
}
}
}
Part of our product code likes this, the main thread and subthread share the mainList. I run the test serval times but never reproduce the ConcurrentModificationException.
update：
thanks for all your replying，this code is actually a brief abstraction of our production code. What I wanna do actually is very simply:
the main thread hold a list to receive data from some source, when the list reaches a certain size the main thread pass the list to a sub thread which stores the data to a data base.
Maybe a more safer way to accomplish this is to extract the
List<Integer> tempList = mainList;
mainList = new ArrayList<Integer>();
to the main thread, and pass the templist to sub thread. The code I list before is a legacy code, and I want to fix this code.
As David Wallace points out, you need to at least declare mainList as volatile.
However, that alone does not actually make the code thread-safe. Even though you're switching the reference in the cmd thread, the main thread may have already fetched the reference before that happens and can then proceed to work on it at the same time as the cmd thread reads from it.
For example, this is a possible sequence of events:
cmd thread fetches mainList reference and gets list A
Main thread fetches mainList reference and also gets list A
cmd thread creates the new list, B, and assigns it to mainList
Main thread starts calculating a random number
cmd thread starts iterating over list A
Main thread adds its random number to list A
cmd thread continues to iterate over the modified list, now in an inconsistent state due to concurrent modification
EDIT: At this point, I was planning to edit in a suggestion to do what you wanted, but I realized there could be several quite different things that you wanted to do with this code, so a single suggestion would just be a guess anyway. If you want a solution, I suggest you start a new question describing your goal in more detail.
No it's not thread safe. You are not using any synchronization facilities around mainList. The fact that the code didn't throw ConcurrentModificationException does not imply that the code is thread-safe. It merely means you might have a race condition if it's thrown.
No, I do not think the code is thread safe because the main thread could call List.add while the pool thread is assigning a new value to mainList. If mainList where a primative, it might be sufficient to make it 'volatile'. But I don't think you can use 'volatile' with object references.
To make the assignment safe, you would need to synchronize on something while you make the assignment and then also wherever you try to touch mainList like:
Object lock = new Object();
...
synchronized (lock) {
mainList = new ArrayList<Integer>();
}
...
synchronized (lock) {
mainList.add(r.nextInt(200));
}
This would ensure that the pool thread could not reassign mainList while the main thread was in the process of calling add().
But I'm not sure if you can get a ConcurrentModificationException if only the main thread modifies the list and the pool thread is only iterating through elements. And even if the pool thread did modify the list I'm still not sure if you could get a CME if the pool thread modified a new list that has not yet been assigned to mainList.
So if you are seeing a CME, I suspect your test does not really represent what is happening in production.
Related
Sorry if the question is unclear
I am making a simple multithread program that has a linked list to store all thread created except the main thread. Then I want to send some signal to terminate the main thread but only when all other threads have closed and I intend to do this by making that when the thread close, it will remove itself from linked list then the main thread will check if that list size == null or not
here is my code
public class MainProgram {
//some global static variable
static List<Thread> threadList = new LinkedList<Thread>();
public void main() throws IOException {
ServerSocket serverSocket;
serverSocket = new ServerSocket(1234);
while(true){
if(Shutdown_Handler.shutdown==true){
//wait for all other thread close/terminate
return
}
Socket s = serverSocket.accept();
ClientThread x = new ClientThread(s);
Thread y = new Thread(x);
threadList.add(y);
y.start();
}
}
}
when Shutdown_Handler.shutdown==true the main will check the threadList if it is null. The problem is I don't know how to make the thread remove itself from the list. As what I have searched, for normal object, I can create method like this
public class someObject {
public static void delete(){
if(list.size = null) return;
list.remove(this);
}
}
However, in case of thread, the Class implement Runnable so this reference is to the object but not the thread stored in the list
I would recommend using a HashMap instead of a List. The keys can be the Thread Name (e.g. Thread.getName()) and the values will be the Threads.
Map<String, Thread> threadMap = new HashMap<String, Thread>();
You should also create this Map as a synchronizedMap (using Collections.synchronizedMap(...))
Map<String, Thread> synchronizedMap = Collections.synchronizedMap(threadMap);
Now, whenever you construct a Thread, you pass this HashMap into its constructor and the Thread can hold a reference to it. Therefore, when the Thread is about to terminate it can remove itself from the HashMap by using its own Thread name as the key to remove.
Assuming that ClientThread is a Runnable, the basic code is:
public class ClientThread implements Runnable {
public void run() {
// do stuff
MainProgram.threadList.remove(Thread.currentThread());
}
}
However this has a couple of problems:
There are going to be multiple threads performing operations on a list without proper synchronization. That is incorrect, and you are liable to get intermittent failures if you do this.
Unless run() removes the thread from the list in a finally block, a thread that terminates abnormally is liable to not get removed.
It is bad design to use a global static. And worse design to expose it as a bare (non-private) variable.
A HashSet<Thread> would be more efficient if the number of threads is liable to be large.
I'm trying to figure out the best way to have multiple threads working from the same list of strings. For example, say I have a list of words, and I want multiple threads to work on printing out each word on this list.
Here is what I came up with. The thread uses a while loop, and while the iterator has next, it prints out and removes it from the list.
import java.util.*;
public class ThreadsExample {
static Iterator it;
public static void main(String[] args) throws Exception {
ArrayList<String> list = new ArrayList<>();
list.add("comet");
list.add("planet");
list.add("moon");
list.add("star");
list.add("asteroid");
list.add("rocket");
list.add("spaceship");
list.add("solar");
list.add("quasar");
list.add("blackhole");
it = list.iterator();
//launch three threads
RunIt rit = new RunIt();
rit.runit();
rit.runit();
rit.runit();
}
}
class RunIt implements Runnable {
public void run()
{
while (ThreadsExample.it.hasNext()) {
//Print out and remove string from the list
System.out.println(ThreadsExample.it.next());
ThreadsExample.it.remove();
}
}
public void runit() {
Thread thread = new Thread(new RunIt());
thread.start();
}
}
This seems to work, although I get some Exception in thread "Thread-2" Exception in thread "Thread-0" java.lang.IllegalStateException errors during the run:
Exception in thread "Thread-1" Exception in thread "Thread-0"
java.lang.IllegalStateException at
java.util.ArrayList$Itr.remove(ArrayList.java:864) at
RunIt.run(ThreadsExample.java:44) at
java.lang.Thread.run(Thread.java:745) java.lang.IllegalStateException
at java.util.ArrayList$Itr.remove(ArrayList.java:864) at
RunIt.run(ThreadsExample.java:44) at
java.lang.Thread.run(Thread.java:745)
Am I doing this correctly or is there a better way to have multiple threads working on the same pool of strings?
A better way to do this is to use a concurrent queue. The Queue interface is designed to hold elements in a structure prior to processing them.
final Queue<String> queue = new ConcurrentLinkedQueue<String>();
queue.offer("asteroid");
ExecutorService executorService = Executors.newFixedThreadPool(4);
executorService.execute(new Runnable() {
public void run() {
System.out.println(queue.poll());
}
});
executorService.shutdown();
Try creating the list as a synchronized list using List.synchronizedList
Update your code like this:
ArrayList<String> list = Collections.synchronizedList(new ArrayList<>());
Am I doing this correctly or is there a better way to have multiple threads working on the same pool of strings?
You are not doing it correctly. Your code is not properly synchronized, and therefore its behavior is not well defined. There are a great number of ways you could approach the general problem you present, but one way the issues in your particular code could be fixed would be to change RunIt.run() to properly synchronize:
public void run()
{
while (true) {
synchronized(ThreadsExample.it) {
if (ThreadsExample.it.hasNext()) {
//Print out and remove string from the list
System.out.println(ThreadsExample.it.next());
ThreadsExample.it.remove();
} else {
break;
}
}
}
}
Note here that the hasNext() check, retrieval of the next element, and removal of that element are all handled within the same synchronized block to ensure mutual consistency of these operations. On the other hand, the scope of that block is contained within the loop, so that different threads executing the loop concurrently each get a chance to execute.
Note, too, that although in this case all the threads synchronize on the Iterator object, that's basically just a convenience (for me). As long as they all synchronize on the same object, it doesn't matter so much which object that is.
I have done some research and could not find a solution to this problem.
From this topic Synchronization, When to or not to use? i understand i could use synchronized, but doing so doesn't solve the problem.
The case is that i have a method in which a Thread is used to create an ArrayList. In that same Thread another method is called after a BufferedReader has finished reading a file and the lines are being added to the first List.
In the second method the first list is being used to create the second List. When all that is done, the first method uses the second list.
This is somewhat the code i use, if there is something not clear please ask and i will try to provide the info needed.
public synchronized void theBaseList() {
Thread t = new Thread() {
#override
public void run() {
try(
while((line = br.readLine()) != null) {
firstList.add(line):
}
}
nextMethod();
currentObject = (Object[]) secondList.get(0); // throws an exception
}
}
};
t.start();
public synchronized void nextMethod() {
Thread t1 = new Thread(){
double objectListSize = calculateObjectListLength(firstList.size());
#override
public void run() {
try {
// create Objects
secondList.add(objects);
}
}
};
t1.start();
}
When i use a Thread in nextMethod() to create a new list of Objects from the items in the first list, i get an ArrayIndexOutOfBoundsException saying
Exception in thread "Thread-4" java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
I avoided this by not using a Thread in the second method and all works fine.
If i do use 2 Threads and make both methods synchronized, it still throws the exception.
Is it possible or should i just settle by not using a Thread in the second method? I thought synchronized was for dealing with this sort of problems. I don't understand why it doesn't work.
Let's say your methods are defined in a class named Sample and you've created an instance mySample. This appears to be what your code is doing:
main thread calls mySample.theBaseList() and synchronizes by locking on mySample.
theBaseList() defines thread1 and starts it.
theBaseList() exits scope, thus unlocking on mySample.
thread1 reads in the lines of a file and adds them to list1 (these operations are not synchronized)
thread1 calls mySample.nextMethod()
mySample.nextMethod() synchronizes by locking on mySample
nextMethod() defines thread2 and starts it.
nextMethod() exits scope, thus unlocking on mySample.
* thread2 sets up list2 (these operations are not synchronized)
* thread1, having returned from nextMethod() reads from list2 (these operations are not synchronized)
The last two operations are the cause of your race condition.
In your case, using synchronized methods is perhaps too coarse grained. A better option may be to synchronize on the object on which both threads operate, secondList.
nextMethod();
synchronized(secondList) {
currentObject = (Object[]) secondList.get(0); // should no longer throw an exception
}
synchronized(secondList) {
// create Objects
secondList.add(objects);
}
EDIT:
synchronized(secondList) {
nextMethod();
secondList.wait();
currentObject = (Object[]) secondList.get(0); // should no longer throw an exception
}
synchronized(secondList) {
// create Objects
secondList.add(objects);
secondList.notifyAll();
}
I'm using multithreading to process a List of Strings in batches, however I'm getting this error when the Runnable task is iterating over the List to process each String.
For example the code roughly follows this structure:
public class RunnableTask implements Runnable {
private List<String> batch;
RunnableTask(List<String> batch){
this.batch = batch;
}
#Override
public void run() {
for(String record : batch){
entry = record.split(",");
m = regex.matcher(entry[7]);
if (m.find() && m.group(3) != null){
currentKey = m.group(3).trim();
currentValue = Integer.parseInt(entry[4]);
if ( resultMap.get(currentKey) == null ){
resultMap.put(currentKey, currentValue);
} else {
resultMap.put(currentKey, resultMap.get(currentKey) + currentValue);
}
}
}
}
}
Where the thread that is passing these batches for processing never modifies "batch" and NO CHANGES to batch are made inside the for loop. I understand that this exception ConcurrentModificationException is due to modifying the List during iteration but as far as I can tell that isn't happening. Is there something I'm missing?
Any help is appreciated,
Thankyou!
UPDATE1: It seems instance-variables aren't thread safe. I attempted to use CopyOnWriteArrayList in place of the ArrayList but I received inconsistent results - suggesting that the full iteration doesn't complete before the list is modified in some way and not every element is being processed.
UPDATE2: Locking on the loop with sychronized and/or a reentrantlock both still give the same exception.
I need a way to pass Lists to Runnable tasks and iterate over those lists without new threads causing concurrency issues with that list.
I understand that this exception ConcurrentModificationException is due to modifying the List during iteration but as far as I can tell that isn't happening
Ok, consider what happens when you create a new thread, passing a reference to RunnableTask instance, initialized with a different list as constructor parameter? You just changed the list reference to point to different list. And consider what happens when at the same time, a different thread inside the run() method, is changing the list, at any point. This will at some point of time, throw ConcurrentModificationException.
Instance Variables are not Thread-Safe.
Try this in your code:
public void run() {
for(String record : new ArrayList(batch)){
//do processing with record
}
}
There is a sort of problem with all your threads processing the list (is the list modified during the process?) but is difficult to tell with the code you're providing
Problem is due to multiple thread concurrently modifying the the source List structure. What I would suggest you should devide the source list to new sublist(according to size) and pass that list to threads.
Say your source List have 100 elements. and you are running 5 concurrent thread.
int index = 0;
List<TObject> tempList = new ArrayList<>();
for(TObject obj:srcList){
if(i==(srcList.size()/numberOfthread)){
RunnableTask task = new RunnableTask(tempList);
tempList = new ArrayList<>();
}else
tempList.add(obj);
}
In this case your original list would not be modified.
you need to lock the list before accessing its elements. because List is not thread safe. Try this
public void run() {
synchronizd(batch){
for(String record : batch){//do processing with record}
}
}
yes you are getting ConcurrentModificationException because your List is getting modified during iteration. If performance is not a critical issue I suggest use synchronization.
public class RunnableTask implements Runnable {
private List<String> batch = new ArrayList<String>();
RunnableTask(List<String> batch){
this.batch = batch;
}
public void run() {
synchronized (batch) {
for(String record : batch){//do processing with record}
}
}
}
}
or even better use ReentrantLock.
Your followups indicate that you are trying to reuse the same List multiple times. Your caller must create a new List for each Runnable.
Obviously someone else is changing the content of the list, which is out of picture of the code you mentioned. (If you are sure that the ConcurrentModificationException is complaining for the batch list, but not resultMap, and you are actually showing all code in RunnableTask)
Try to search in your code, for places that is updating the content of the list, check if it is possible concurrently with your RunnableTask.
Simply synchronizing in the RunnableTask is not going to help, you need to synchronize all access to the list, which is obviously happening somewhere else.
If performance is an issue to you so that you cannot synchronize on the batch list (which prohibit multiple RunnableTask to execute concurrently), consider making use of ReaderWriterLock: RunnableTask acquires read lock, while the list update logic acquire the write lock.
I have a simple utility which pings a set of nodes and returns an ArrayList of strings to a future object to be outputted to a file. The program should run until terminated by the user.
It doesn't appear that the future receives the results (or at least passes them to the method to output to the file). No matter the number of threads I have concurrently running (always less than 100, determined by an input file), I am only outputting the results from the first and last initialized threads.
As a sanity check, I created a global variable in which each thread will send its results before closing and returning its results to the Future object. This variable is correctly updated by all threads.
Does anyone have any ideas why Future doesn't seem to be receiving all my results from the threads?
public class PingUtility{
public static ExecutorService pool = Executors.newFixedThreadPool(100);
static Future<ArrayList<String>> future;
public static void main(String[] args) throws Exception {
Timer timer = new Timer();
TimerTask task = new TimerTask(){
public void run(){
//Creates a pool of threads to be executed
ArrayList<String[]> nodes = new ArrayList<String[]>()
future = pool.submit(new PingNode(nodes));
}
}
};
timer.scheduleAtFixedRate(task, 0, interval);
while(true){
try{
ArrayList<String[]> tempOutputArray = future.get();
Iterator<String[]> it = tempOutputArray.iterator();
while(it.hasNext()) appendFile(it.next());
tempOutputArray.clear();
}catch(Exception nullException){
//Do nothing
}
}
}
Your problem is that you are modifying the future static field without synchronization in your timer-task thread(s) and reading it in the main thread. You need to either synchronize on it when you modify and read it or use another mechanism to share information between the threads.
I'd recommend switching from a static field to a LinkedBlockingQueue as a better way to send information from the PingNode call to the appendFile(...) method. This saves from needing to do the synchronization yourself and protects against the race conditions where multiple timer-tasks will start and overwrite the future before the consumer can get() from them. Maybe something like:
BlockingQueue<String[]> queue = new LinkedBlockingQueue<String[]>();
...
// inside of run, producer passes the queue into the PingNode
public void run() {
pool.submit(new PingNode(queue));
}
// consumer
while (true) {
String[] array = queue.take();
...
}
This doesn't take into effect how you are going to stop the threads when you are done. If the timer task is killed the entity could add to the queue a termination object to stop the main loop.
A Future object is not a bin, like an ArrayList, it merely points to a single computational result. Because you only have one static pointer to this Future, what I imagine is happening is this:
future = null
nullException
nullException
nullException
nullException
...
First thread finally sets future = Future<ArrayList<String>>
Call to future.get() blocks...
Meanwhile, all other threads get scheduled, and they reassign future
The last thread will obviously get the last say in what future points to
Data is gathered, written to file, loop continues
future now points to the Future from the last thread
Results from last thread get printed