It's plenty of questions regarding ConcurrentModificationException for ArrayList objects, but I could not find yet an answer to my problem.
In my servlet I have an ArrayList as a member object:
List myList<Object> = new ArrayList<Object> (...);
The list must be shared among users and sessions.
In one of the servlet's methods, method1, I need to iterate over the ArrayList items, and eventually add clear the list after the iteration. Here a snippet:
for (Object o : myList) {
// read item o
}
myList.clear();
In another method, method2, I simply add a new Item to the list.
Most of the times the method ends its job without errors. Sometimes, probably due to the concurrent invocation of this method by different users, I get the famous java util.ConcurrentModificationException exception.
Should I define my List as:
List myList = Collections.synchronizedList(new ArrayList(...));
Would this be enough or am I missing something? What's behind the scenes? When there is a possible concurrency, is the second thread held in standby by the container?
EDIT: I have added the answers to some comments.
Using a synchronized list will not solve your problem. The core of the problem is that you are iterating over a list and modifying it at the same time. You need to use mutual exclusion mechanisms (synchronized blocks, locks etc) to ensure that they do not happen at the same time. To elaborate, if you start with:
methodA() {
iterate over list {
}
edit list;
}
methodB() {
edit list;
}
If you use a synchronized list, what you essentially get is:
methodA() {
iterate over list {
}
synchronized {
edit list;
}
}
methodB() {
synchronized {
edit list;
}
}
but what you actually want is:
methodA() {
synchronized {
iterate over list {
}
edit list;
}
}
methodB() {
synchronized {
edit list;
}
}
Just using synchronizedList makes all methods thread safe EXCEPT Iterators.
I would use CopyOnWriteArrayList. It is thread safe and doesn't produce ConcurrentModificationException.
ConcurrentModificaitonException occurs when you attempt to modify a collection while you're iterating through it. I imagine that the error only gets thrown when you perform some conditional operation.
I'd suggest pushing the values you want to add/remove into a separate list and performing the add /remove after you're done iterating.
You need to lock not just over the method accesses but over your use of the list.
So if you allocate a paired Object like:
Object myList_LOCK = new Object();
then you can lock that object whenever you are accessing the List, like this:
synchronized(myList_LOCK) {
//Iterate through list AND modify all within the same lock
}
at the moment the only locking you're doing is within the individual methods of the List, which isn't enough in your case because you need atomicity over the entire sequence of iteration and modification.
You could use the actual object (myList) to lock rather than a paired object but in my experience you are better off using another dedicated object as it avoids unexpected deadlock conditions that can arise as a result of the code internal to the object locking on the object itself.
This is kind of an add onto Peter Lawery's answer. But since copying wouldn't effect you too negatively you can do a mixture of copying with synchronization.
private final List<Object> myList = new ArrayList<Object>();
public void iterateAndClear(){
List<Object> local = null;
synchronized(myList){
local = new ArrayList<Object>(myList);
myList.clear();
}
for(Object o : local){
//read o
}
}
public void add(Object o){
synchronized(myList){
myList.add(o);
}
}
Here you can iterate over o elements without fear of comodifications (and outside of any type of synchronization), all while myList is safely cleared and added to.
Related
I am new to Java and concurrency stuff.
The purpose of the assignment was to learn concurrency.
- So when answering this question please keep in mind that I am supposed to use only Hashmap (which is not synchronized by nature) and synchronize it myself. If you provide more knowledge its appreciated but not required.
I declared a hashmap like this:
private HashMap<String, Flight> flights = new HashMap<>();
recordID is the key of the flight to be deleted.
Flight flightObj = flights.get(recordID);
synchronized(flightObj){
Flight deletedFlight = flights.remove(recordID);
editResponse = "Flight with flight ID " + deletedFlight.getFlightID() +" deleted successfully";
return editResponse;
}
Now my doubt: Is it fine to synch on the basis of flightObj?
Doubt 2:
Flight newFlight = new Flight(FlightServerImpl.createFlightID());
flights.put(newFlight.getFlightID(),newFlight);
If I create flightts by using above code and if more than 1 thread try execute this code will there be any data consistency issues ? Why or why not?
Thanks in advance.
To quickly answer you questions:
Both are not okay - you can't remove two different objects in parallel, and you can't add two different objects in parallel.
From java documentation:
If multiple threads access a hash map concurrently, and at least one of the threads modifies the map structurally, it must be synchronized externally. (A structural modification is any operation that adds or deletes one or more mappings; merely changing the value associated with a key that an instance already contains is not a structural modification.) This is typically accomplished by synchronizing on some object that naturally encapsulates the map. If no such object exists, the map should be "wrapped" using the Collections.synchronizedMap method. This is best done at creation time, to prevent accidental unsynchronized access to the map:
So, it's okay for many threads to use get concurrently and even put that replaces an object.
But if you remove or add a new object - you need to synchronize before calling any hashmap function.
In that case you can either do what's suggested in the documentation and use a global lock. But, it seems that since some limited concurrency is still allowed, you could get that concurrency it by using a read/write lock.
You can do the following
class MySynchronizedHashMap<E> implements Collection<E>, Serializable {
private static final long serialVersionUID = 3053995032091335093L;
final Collection<E> c; // Backing Collection
final Object mutex; // Object on which to synchronize
SynchronizedCollection(Collection<E> c) {
this.c = Objects.requireNonNull(c);
mutex = this;
}
public boolean add(E e) {
synchronized (mutex) {return c.add(e);}
}
public boolean remove(Object o) {
synchronized (mutex) {return c.remove(o);}
}
}
MySynchronizedHashMap mshm = new MySynchronizedHashMap<>(new HashMap<String, Flight>());
mshm.add(new Flight());
I'm using multithreading to process a List of Strings in batches, however I'm getting this error when the Runnable task is iterating over the List to process each String.
For example the code roughly follows this structure:
public class RunnableTask implements Runnable {
private List<String> batch;
RunnableTask(List<String> batch){
this.batch = batch;
}
#Override
public void run() {
for(String record : batch){
entry = record.split(",");
m = regex.matcher(entry[7]);
if (m.find() && m.group(3) != null){
currentKey = m.group(3).trim();
currentValue = Integer.parseInt(entry[4]);
if ( resultMap.get(currentKey) == null ){
resultMap.put(currentKey, currentValue);
} else {
resultMap.put(currentKey, resultMap.get(currentKey) + currentValue);
}
}
}
}
}
Where the thread that is passing these batches for processing never modifies "batch" and NO CHANGES to batch are made inside the for loop. I understand that this exception ConcurrentModificationException is due to modifying the List during iteration but as far as I can tell that isn't happening. Is there something I'm missing?
Any help is appreciated,
Thankyou!
UPDATE1: It seems instance-variables aren't thread safe. I attempted to use CopyOnWriteArrayList in place of the ArrayList but I received inconsistent results - suggesting that the full iteration doesn't complete before the list is modified in some way and not every element is being processed.
UPDATE2: Locking on the loop with sychronized and/or a reentrantlock both still give the same exception.
I need a way to pass Lists to Runnable tasks and iterate over those lists without new threads causing concurrency issues with that list.
I understand that this exception ConcurrentModificationException is due to modifying the List during iteration but as far as I can tell that isn't happening
Ok, consider what happens when you create a new thread, passing a reference to RunnableTask instance, initialized with a different list as constructor parameter? You just changed the list reference to point to different list. And consider what happens when at the same time, a different thread inside the run() method, is changing the list, at any point. This will at some point of time, throw ConcurrentModificationException.
Instance Variables are not Thread-Safe.
Try this in your code:
public void run() {
for(String record : new ArrayList(batch)){
//do processing with record
}
}
There is a sort of problem with all your threads processing the list (is the list modified during the process?) but is difficult to tell with the code you're providing
Problem is due to multiple thread concurrently modifying the the source List structure. What I would suggest you should devide the source list to new sublist(according to size) and pass that list to threads.
Say your source List have 100 elements. and you are running 5 concurrent thread.
int index = 0;
List<TObject> tempList = new ArrayList<>();
for(TObject obj:srcList){
if(i==(srcList.size()/numberOfthread)){
RunnableTask task = new RunnableTask(tempList);
tempList = new ArrayList<>();
}else
tempList.add(obj);
}
In this case your original list would not be modified.
you need to lock the list before accessing its elements. because List is not thread safe. Try this
public void run() {
synchronizd(batch){
for(String record : batch){//do processing with record}
}
}
yes you are getting ConcurrentModificationException because your List is getting modified during iteration. If performance is not a critical issue I suggest use synchronization.
public class RunnableTask implements Runnable {
private List<String> batch = new ArrayList<String>();
RunnableTask(List<String> batch){
this.batch = batch;
}
public void run() {
synchronized (batch) {
for(String record : batch){//do processing with record}
}
}
}
}
or even better use ReentrantLock.
Your followups indicate that you are trying to reuse the same List multiple times. Your caller must create a new List for each Runnable.
Obviously someone else is changing the content of the list, which is out of picture of the code you mentioned. (If you are sure that the ConcurrentModificationException is complaining for the batch list, but not resultMap, and you are actually showing all code in RunnableTask)
Try to search in your code, for places that is updating the content of the list, check if it is possible concurrently with your RunnableTask.
Simply synchronizing in the RunnableTask is not going to help, you need to synchronize all access to the list, which is obviously happening somewhere else.
If performance is an issue to you so that you cannot synchronize on the batch list (which prohibit multiple RunnableTask to execute concurrently), consider making use of ReaderWriterLock: RunnableTask acquires read lock, while the list update logic acquire the write lock.
While using Java Threading Primitives to construct a thread safe bounded queue - whats the difference between these 2 constructs
Creating an explicit lock object.
Using the list as the lock and waiting on it.
Example of 1
private final Object lock = new Object();
private ArrayList<String> list = new ArrayList<String>();
public String dequeue() {
synchronized (lock) {
while (list.size() == 0) {
lock.wait();
}
String value = list.remove(0);
lock.notifyAll();
return value;
}
}
public void enqueue(String value) {
synchronized (lock) {
while (list.size() == maxSize) {
lock.wait();
}
list.add(value);
lock.notifyAll();
}
}
Example of 2
private ArrayList<String> list = new ArrayList<String>();
public String dequeue() {
synchronized (list) { // lock on list
while (list.size() == 0) {
list.wait(); // wait on list
}
String value = list.remove(0);
list.notifyAll();
return value;
}
}
public void enqueue(String value) {
synchronized (list) { // lock on list
while (list.size() == maxSize) {
list.wait(); // wait on list
}
list.add(value);
list.notifyAll();
}
}
Note
This is a bounded list
No other operation is being performed apart from enqueue and dequeue.
I could use a blocking queue, but this question is more for improving my limited knowledge of threading.
If this question is repeated please let me know.
The short answer is, no, there is no functional difference, other than the extra memory overhead of maintaining that extra lock object. However, there are a couple of semantics-related items I would consider before making a final decision.
Will I ever need to perform synchronized operations on more than just my internal list?
Let's say you wanted to maintain a parallel data structure to your ArrayList, such that all operations on the list and that parallel data structure needed to be synchronized. In this case, it might be best to use the external lock, as locking on either the list or the structure might be confusing to future development efforts on this class.
Will I ever give access to my list outside of my queue class?
Let's say you wanted to provide an accessor method for your list, or make it visible to extensions of your Queue class. If you were using an external lock object, classes that retrieved references to the list would never be able to perform thread-safe operations on that list. In that case, it'd be better to synchronize on the list and make it clear in the API that external accesses/modifications to the list must also synchronize on that list.
I'm sure there are more reasons why you might choose one over the other, but these are the two big ones I can think of.
Consider you have a shared memory (List) which will serve as the "critic section".
Now, consider you that you always have items in the list for these scenarios and you want that your system will behave this way:
Thread1 get some item from the list, in the very same time Thread2 wants to add item to the list. Allow this scenario( in assumption I will take first item from begining and insert the new item in the end of the list - in the SAME TIME!).
Thread1 wants to get an item and in the same time Thread2 wants to get an item too.
This should fail.
THANKS
One possibility is to wrap your List in a class that proxies or overrides the get and add methods.
That way, you can use an explicit Lock on the add method, so that only one thread can add at any given time.
See for instance:
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/ReentrantLock.html
You could do this either by extending a List implementation, and overriding the add and get methods (or all relevant methods), or by using composition instead of inheritance, having a proxy class that forwards the calls to the list, but decorates the add and get with the explicit obtaining of the Lock.
A very simple example would be something like:
public class SharedMemory<K> {
private final List<K> memoryList;
private static final ReentrantLock lock = new ReentrantLock();
public SharedMemory() {
memoryList = new ArrayList<>();
}
public void storeItem(K item) {
memoryList.add(item);
}
public K getItem(int pos){
lock.lock();
try{
return memoryList.get(pos);
} finally {
lock.unlock();
}
}
}
public final class ClientGateway {
private static ClientGateway instance;
private static List<NetworkClientListener> listeners = Collections.synchronizedList(new ArrayList<NetworkClientListener>());
private static final Object listenersMutex = new Object();
protected EventHandler eventHandler;
private ClientGateway() {
eventHandler = new EventHandler();
}
public static synchronized ClientGateway getInstance() {
if (instance == null)
instance = new ClientGateway();
return instance;
}
public void addNetworkListener(NetworkClientListener listener) {
synchronized (listenersMutex) {
listeners.add(listener);
}
}
class EventHandler {
public void onLogin(final boolean isAdviceGiver) {
new Thread() {
public void run() {
synchronized (listenersMutex) {
for (NetworkClientListener nl : listeners)
nl.onLogin(isAdviceGiver);
}
}
}.start();
}
}
}
This code throws a ConcurrentModificationException
But I thought if they are both synchronized on the listenersMutex then they should be executed in serial? All code within functions that operate on the listeners list operate within syncrhonized blocks that are synchronized on the Mutex. The only code that modifies the list are addNetworkListener(...) and removeNetworkListener(...) but removeNetworkListener is never called at the moment.
What appears to be happening with the error is that a NetworkClientListener is still being added while the onLogin function/thread is iterating the listeners.
Thank you for your insight!
EDIT: NetworkClientListener is an interface and leaves the implementation of "onLogin" up to the coder implementing the function, but their implementation of the function does not have access to the listeners List.
Also, I just completely rechecked and there is no modification of the list outside of the addNetworkListener() and removeNetworkListener() functions, the other functions only iterate the list. Changing the code from:
for (NetworkClientListener nl : listeners)
nl.onLogin(isAdviceGiver);
To:
for(int i = 0; i < listeners.size(); i++)
nl.onLogin(isAdviceGiver);
Appears to solve the concurrency issue, but I already knew this and would like to know what's causing it in the first place.
Thanks again for your continuing help!
Exception:
Exception in thread "Thread-5" java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:782)
at java.util.ArrayList$Itr.next(ArrayList.java:754)
at chapchat.client.networkcommunication.ClientGateway$EventHandler$5.run(ClientGateway.java:283)
EDIT Okay, I feel a little dumb. But thank you for all your help! Particularly MJB & jprete!
Answer: Someone's implementation of onLogin() added a new listener to the gateway. Therefore(since java's synchronization is based on Threads and is reentrant, so that a Thread may not lock on itself) when onLogin() was called we in his implementation, we were iterating through the listeners and in the middle of doing so, adding a new listener.
Solution: MJB's suggestion to use CopyOnWriteArrayList instead of synchronized lists
Mutexes only guard from access from multiple threads. If nl.onLogin() happens to have logic that adds a listener to the listeners list, then a ConcurrentModificationException may be thrown, because it's being accessed (by the iterator) and changed (by the add) simultaneously.
EDIT: Some more information would probably help. As I recall, Java collections check for concurrent modifications by keeping a modification count for each collection. Every time you do an operation that changes the collection, the count gets incremented. In order to check the integrity of operations, the count is checked at the beginning and end of the operation; if the count changed, then the collection throws a ConcurrentModificationException at the point of access, not at the point of modification. For iterators, it checks the counter after every call to next(), so on the next iteration of the loop through listeners, you should see the exception.
I must admit that I don't see it either - if indeed removeListeners is not called.
What is the logic of the nl.onLogin bit? If it modified stuff, it could cause the exception.
A tip btw if you expect listeners to be moderately rare in being added, you could make the list CopyOnWriteArrayList type -- in which case you don't need your mutexes at all - CopyOnWriteArrayList is totally thread safe, and returns a weakly consistent iterator that will never throw CME (except where I just said, in nl.onLogin).
Instead of ArrayList , use can use thread-safe class CopyOnWriteArrayList which does not throw ConcurrentModificationException even if it is modified while iterating. While iterating if it is attempted to modify(add,update) then it makes a copy of the list, but iterater will continue working on original one.
Its a bit slower than ArrayList . It is useful in cases where you do not want to syncronise the iterations.