I have an ArrayList with two accessor methods and a notifier. My list:
private final List<WeakReference<LockListener>> listeners = new ArrayList<>();
All subscribe operations use this:
public void subscribe(#NonNull LockListener listener) {
for (Iterator<WeakReference<LockListener>> it = listeners.iterator(); it.hasNext(); ) {
// has this one already subscribed?
if (listener.equals(it.next().get())) {
return;
}
}
listeners.add(new WeakReference<>(listener));
}
All unsubscribe operations use this:
public void unsubscribe(#NonNull LockListener listener) {
if (listeners.isEmpty()) {
return;
}
for (Iterator<WeakReference<LockListener>> it = listeners.iterator(); it.hasNext(); ) {
WeakReference<LockListener> ref = it.next();
if (ref == null || ref.get() == null || listener.equals(ref.get())) {
it.remove();
}
}
}
And the notifier:
private void notifyListeners() {
if (listeners.isEmpty()) {
return;
}
Iterator<WeakReference<LockListener>> it = listeners.iterator();
while (it.hasNext()) {
WeakReference<LockListener> ref = it.next();
if (ref == null || ref.get() == null) {
it.remove();
} else {
ref.get().onLocked();
}
}
}
What I'm seeing in my testing is that it.next() in notifyListeners() occasionally throws a ConcurrentModificationException. My guess is that this is due to listeners.add() in the subscriber method.
I guess I had a misunderstanding of the iterator here. I was under the assumption that iterating over the list protected me from concurrency issues caused by add/remove operations.
Apparently I'm wrong here. Is it that the iterator is only a protection from ConcurrentModificationException while changing the collection you're iterating? For example, calling remove() on your list while iterating would throw an error, but calling it.remove() is safe.
In my case, subscribing calls add() on the same list as it is being iterated. Is my understanding here correct?
If I read your last sentence correctly, the three methods in your example are called concurrently from several threads. If this is indeed the case, then this is your problem.
ArrayList is not thread-safe. Modifying it concurrently without additional synchronization results in undefined behavior, no matter if you modify it directly or using an iterator.
You could either synchronize access to the list (e.g. making the three methods synchronized), or use a thread-safe collection class like ConcurrentLinkedDeque. In case of the latter, make sure to read the JavaDoc (especially the part about iterators being weekly consistent) to understand what is guaranteed and what is not.
Related
I want to implement a util getting an Enum object by its string value. Here is my implementation.
IStringEnum.java
public interface IStringEnum {
String getValue();
}
StringEnumUtil.java
public class StringEnumUtil {
private volatile static Map<String, Map<String, Enum>> stringEnumMap = new HashMap<>();
private StringEnumUtil() {}
public static <T extends Enum<T>> Enum fromString(Class<T> enumClass, String symbol) {
final String enumClassName = enumClass.getName();
if (!stringEnumMap.containsKey(enumClassName)) {
synchronized (enumClass) {
if (!stringEnumMap.containsKey(enumClassName)) {
System.out.println("aaa:" + stringEnumMap.get(enumClassName));
Map<String, Enum> innerMap = new HashMap<>();
EnumSet<T> set = EnumSet.allOf(enumClass);
for (Enum e: set) {
if (e instanceof IStringEnum) {
innerMap.put(((IStringEnum) e).getValue(), e);
}
}
stringEnumMap.put(enumClassName, innerMap);
}
}
}
return stringEnumMap.get(enumClassName).get(symbol);
}
}
I wrote a unit test in order to test whether it works in multi-thread case.
StringEnumUtilTest.java
public class StringEnumUtilTest {
enum TestEnum implements IStringEnum {
ONE("one");
TestEnum(String value) {
this.value = value;
}
#Override
public String getValue() {
return this.value;
}
private String value;
}
#Test
public void testFromStringMultiThreadShouldOk() {
final int numThread = 100;
CountDownLatch startLatch = new CountDownLatch(1);
CountDownLatch doneLatch = new CountDownLatch(numThread);
List<Boolean> resultList = new LinkedList<>();
for (int i = 0; i < numThread; ++i) {
new Thread(() -> {
try {
startLatch.await();
} catch (Exception e) {
e.printStackTrace();
}
resultList.add(StringEnumUtil.fromString(TestEnum.class, "one") != null);
doneLatch.countDown();
}).start();
}
startLatch.countDown();
try {
doneLatch.await();
} catch (Exception e) {
e.printStackTrace();
}
assertEquals(numThread, resultList.stream().filter(item -> item.booleanValue()).count());
}
}
The testing result is:
aaa:null
java.lang.AssertionError:
Expected :100
Actual :98
It denotes that only one thread execute this line of code:
System.out.println("aaa:" + stringEnumMap.get(enumClassName));
So the initialization codes should be executed by only one thread.
The strange thing is, the result of some thread will be null after executing this line of code:
return stringEnumMap.get(enumClassName).get(symbol);
Since there is no NullPointerException, stringEnumMap.get(enumClassName) must return the reference of innerMap. But why it will get null after calling get(symbol) of innerMap?
Please help, it drive me crazy the whole day!
The problem is due to the line
List<Boolean> resultList = new LinkedList<>();
From JavaDoc of LinkedList:
Note that this implementation is not synchronized.If multiple threads access a linked list concurrently, and at least one of the threads modifies the list structurally, it must be synchronized externally. (A structural modification is any operation that adds or deletes one or more elements; merely setting the value of an element is not a structural modification.) This is typically accomplished by synchronizing on some object that naturally encapsulates the list.If no such object exists, the list should be "wrapped" using the Collections.synchronizedListmethod. This is best done at creation time, to prevent accidental unsynchronized access to the list:
List list = Collections.synchronizedList(new LinkedList(...));
As LinkedList is not thread safe, and unexpected behavior may happens during the add operation.
Which cause the resultList size less than the thread count, and hence the expected count is less than the result count.
To get correct result, add Collections.synchronizedList as suggested.
Although you implementation is fine, I suggest you to follow Matt Timmermans answer for simpler and robust solution.
stringEnumMap should be a ConcurrentHashMap<String, Map<String,Enum>>, and use computeIfAbsent to do the lazy initialization.
ConcurrentMap interface
As others noted, if manipulating a Map across threads you must account for concurrency.
You could handle concurrent access yourself. But there is no need. Java comes with two implementations of Map that are built to internally handle concurrency. These implementations implement the ConcurrentMap interface.
ConcurrentSkipListMap
ConcurrentHashMap
The first maintains the keys in sorted order, implementing the NavigableMap interface.
Here is a table I authored to show the characteristics of all the implementations of Map bundled with Java 11.
You might find other third-party implementations of the ConcurrentMap interface.
try moving
if (!stringEnumMap.containsKey(enumClassName))
and the
return stringEnumMap.get(enumClassName).get(symbol);
into the synchronized block.
I'm in this situation: if I find a specific value in a HashSet, I have to update a field, clear the set and return the field.
Here one example:
static Set<Integer> testSet = new HashSet<>();
static Integer myField = null; // the field could be already != null
public static int testClearSet()
{
for (int i = 0; i < 100; i++) { // this is just for the test
testSet.add(i);
}
for (Integer n : testSet) {
if (n == 50) {
myField = n;
testSet.clear();
return myField;
}
}
return -1;
}
I'm wondering if doing this to the set it's safe, considering the fact that later on I should reuse the set.
I'm asking this, because I knew that to make changes over a Collection while iterating, is not a "good practice", but this case I think is a little bit different.
A possible solution would be:
boolean clear = false;
for (Integer n : testSet) {
if (n == 50) {
myField = n;
clear = true;
break;
}
}
if (clear) {
testSet.clear();
return myField;
}
So, which one is the right way?
It should be safe to remove elements from a set when using an explicit iterator. Hence the following should be safe:
Iterator<Integer> iterator = testSet.iterator();
while (iterator.hasNext()) {
Integer element = iterator.next();
if (element.intValue() == 50) {
testSet.clear();
break;
}
}
A ConcurrentModificationException is only thrown if you continue iterating after changing it manually.
What you do is change it and abort iterating, so it should be 100% safe (regardless of the for-each implementation).
The real issue is, the readability of the code. A piece of code should ideally do one job, and if this job is complicated, split it up. In particular, your code has two parts, a condition and an action:
if (some condition) do some action
So:
public static int testClearSet() {
if (setConatins(50)) {
myField = 50;
testSet.clear();
return myField;
}
return -1;
}
private static boolean setConatins(int searchFor) {
for (Integer n : testSet) {
if (n == searchFor) {
return true;
}
}
return false;
}
The latter method can be replaced with a single API call, for you to figure out.
If you know that your Set changing only in one thread, so you can clean it like in first example.
Method clear() does not throw ConcurrentModificationException.
Both your code will work.
There is indeed a restriction in modifying the collection when u iterate using fail fast iterators. That means, iterating using fail fast iterator will fail if there is any modification in the collection after the iterator was created. All the default iterators that is returned by java collection classes are fail-fast iterators.
private void removeDataTest (Collection<String> c, String item) {
Iterator<String> iter = c.iterator(); //Iterator is created.
while (iter.hasNext()) {
String data = iter.next();
if (data.equals(item)) {
//c.remove(data); //Problem. Directly modifying collection after this iterator is created. In the next iteration it will throw concurrent modification exception.
iter.remove(); //This is fine. Modify collection through iterator.
//c.clear(); break; //This is also should be okay. Modifying the collection directly, but after that it is breaking out and not using the iterator.
}
}
}
In your code, u don't continue iteration after the set is modified. So should be fine.
I have set of connection objects (library code I cannot change) that have a send method. If the sending fails, they call back a generic onClosed listener which I implement that calls removeConnection() in my code, which will remove the connection from the collection.
The onClosed callback is generic and can be called at any time. It is called when the peer closes the connection, for example, and not just when a write fails.
However, if I have some code that loops over my connections and sends, then the onClosed callback will attempt to modify a collection during iteration.
My current code creates a copy of the connections list before each iteration over it; however, in profiling this has shown to be very expensive.
Set<Connection> connections = new ....;
public void addConnection(Connection conn) {
connections.add(conn);
conn.addClosedListener(this);
}
#Override void onClosed(Connection conn) {
connections.remove(conn);
}
void send(Message msg) {
// how to make this so that the onClosed callback can be safely invoked, and efficient?
for(Connection conn: connections)
conn.send(msg);
}
How can I efficiently cope with modifying collections during iteration?
To iterate a collection with the concurrent modification without any exceptions use List Iterator.
http://www.mkyong.com/java/how-do-loop-iterate-a-list-in-java/ - example
If you use simple for or foreach loops, you will receive ConcurrentModificationException during the element removing - be careful on that.
As an addition, you could override the List Iterator with your own one and add the needed logic. Just implement the java.util.Iterator interface.
A ConcurrentSkipListSet is probably what you want.
You could also use a CopyOnWriteArraySet. This of course will still make a copy, however, it will only do so when the set is modified. So as long as Connection objects are not added or removed regularly, this would be more efficient.
You can also use ConcurrentHashMap.
ConcurrentHashMap is thread-safe, so you don't need to make a copy in order to be able to iterate.
Take a look at this implementation.. http://www.java2s.com/Tutorial/Java/0140__Collections/Concurrentset.htm
I would write a collection wrapper that:
Keeps a set of objects that are to be removed. If the iteration across the underlying collection comes across one of these it is skipped.
On completion of iteration, takes a second pass across the list to remove all of the gathered objects.
Perhaps something like this:
class ModifiableIterator<T> implements Iterator<T> {
// My iterable.
final Iterable<T> it;
// The Iterator we are walking.
final Iterator<T> i;
// The removed objects.
Set<T> removed = new HashSet<T>();
// The next actual one to return.
T next = null;
public ModifiableIterator(Iterable<T> it) {
this.it = it;
i = it.iterator();
}
#Override
public boolean hasNext() {
while ( next == null && i.hasNext() ) {
// Pull a new one.
next = i.next();
if ( removed.contains(next)) {
// Not that one.
next = null;
}
}
if ( next == null ) {
// Finished! Close.
close();
}
return next != null;
}
#Override
public T next() {
T n = next;
next = null;
return n;
}
// Close down - remove all removed.
public void close () {
if ( !removed.isEmpty() ) {
Iterator<T> i = it.iterator();
while ( i.hasNext() ) {
if ( removed.contains(i.next())) {
i.remove();
}
}
// Clear down.
removed.clear();
}
}
#Override
public void remove() {
throw new UnsupportedOperationException("Not supported.");
}
public void remove(T t) {
removed.add(t);
}
}
public void test() {
List<String> test = new ArrayList(Arrays.asList("A","B","C","D","E"));
ModifiableIterator i = new ModifiableIterator(test);
i.remove("A");
i.remove("E");
System.out.println(test);
while ( i.hasNext() ) {
System.out.println(i.next());
}
System.out.println(test);
}
You may need to consider whether your list could contain null values, in which case you will need to tweak it somewhat.
Please remember to close the iterator if you abandon the iteration before it completes.
I have found myself using two slightly different approaches for getting/creating items from/in a ConcurrentHashMap and I wonder which one is better.
The first way:
public class Item {
private boolean m_constructed;
...
public void construct() {
if (m_constructed) {
return;
}
synchronized (this) {
if (m_constructed) {
return;
}
// Some heavy construction
m_constructed = true;
}
}
}
ConcurrentHashMap<String, Item> m_items = new ConcurrentHashMap<String, Item>();
...
// The following code is executed concurrently by multiple threads:
public Item getOrCreateItem(String key) {
Item newItem = new Item(); // The constructor is empty
Item item = m_items.putIfAbsent(key, newItem);
if (item == null) {
item = newItem;
}
item.construct(); // This is the real construction
return item;
}
Please, do not comment on using this in synchronize (this). I am aware of the crappiness of using this as the lock object, but I am fine with it in this particular example.
The second way:
public class Item {
private boolean m_constructed;
...
public void construct() {
// Some heavy construction
m_constructed = true;
}
public void waitForConstruction() throws InterruptedException {
while (!m_constructed) {
Thread.sleep(50);
}
}
}
ConcurrentHashMap<String, Item> m_items = new ConcurrentHashMap<String, Item>();
...
// The following code is executed concurrently by multiple threads:
public Item getOrCreateItem(String key) {
Item newItem = new Item(); // The constructor is empty
Item item = m_items.putIfAbsent(key, newItem);
if (item == null) {
item.construct(); // This is the real construction
item = newItem;
}
item.waitForConstruction();
return item;
}
I wonder if one way is more superior to the other. Any ideas?
EDIT
A few words on the context. The Item map is populated concurrently by multiple threads, all of which execute getOrCreateItem. No code tries to access the map in any other way. Once the population is over, the map is never modified and becomes open to read-only access. Hence no one can get a partially constructed Item instance outside the getOrCreateItem method.
EDIT2
Thanks for all the answers. I have adopted the first approach with the suggested fixes:
public class Item {
private volatile boolean m_constructed; // !!! using volatile
...
public void construct() {
if (m_constructed) {
return;
}
synchronized (this) {
if (m_constructed) {
return;
}
// Some heavy construction
m_constructed = true;
}
}
}
ConcurrentHashMap<String, Item> m_items = new ConcurrentHashMap<String, Item>();
...
// The following code is executed concurrently by multiple threads:
public Item getOrCreateItem(String key) {
Item item = m_items.get(key); // !!! Checking the mainstream case first
if (item == null) {
Item newItem = new Item(); // The constructor is empty
item = m_items.putIfAbsent(key, newItem);
if (item == null) {
item = newItem;
}
}
item.construct(); // This is the real construction
return item;
}
Of course, I am acting under the assumption that no code accesses the map of items using any other way but the getOrCreateItem method. Which is true in my code.
First, note that none of your solutions is properly synchronized. The m_constructed flag must be made volatile for this to work. Otherwise you may encounter thread visibility problems, since your read access to this member isn't protected by the lock.
Anyway, the item class is too similar to the concept of Future. If you store Future implementation as map values (e.g. FutureTask), your problem is solved.
I think the first solution is not so bad. The m_constructed variable must be volatile for it to work correctly, and as John Vint suggested, calling ConcurrentMap.get() before doing putIfAbsent() is recommended.
I suspect that the most important call path is the steady state access (threads accessing the item that is already added to the map and is already constructed). In that case, with the first solution, you would do a ConcurrentHashMap.get() call and a volatile read (on m_constructed), which is not so bad.
The second solution is a poor one as it involves unnecessary busy spin loop. When it is converted to using a CountDownLatch per John Vint's suggestion, the steady state performance would be similar to the above: a ConcurrentHashMap.get() call and CountDownLatch.await() which should be similar to a uncontended volatile read. However, the only downside is that it adds more memory to Item.
I have LinkedList of objects and an iterator. I know that this ConcurrentModificationException is thrown when you try to modify the list while running the iterator. But in my case, I don't understand where this modification is being done.
The iterator looks like this :
private static void insertTasks(Task t) {
if(eventQueue.size() == 0) {
eventQueue.addFirst(tsk);
return;
}
int pos = 0;
while (itr.hasNext()){
//The line below throws the exception
if (t.getArrivalTime() <= itr.next().getArrivalTime() )
{
break;
}
pos++;
}
}
I am calling this insertTasks method from another method as shown below :
tsk = null;
tsk = new Task(1,"P1",1,4.0f,1.5f,0.0f,8.0f);
insertTasks(tsk);
tsk = null;
tsk = new Task(0,"P0",2,5.0f,2.5f,1.0f,10.0f);
insertTasks(tsk);
The getArrivalTime in the Task objects looks like :
public float getArrivalTime() { return arrivalTime; }
My question is, where am I doing this modification ? The while loop where I run this iterator is not doing any modification. Does it ?
Am I missing something ?
I reckon the problem is that itr is a static field in your class and that's creating the issue, as you're adding an element to eventQueue in your second call to insertTasks().
Avoid static fields... program yourself to fear them and avoid them as much as you can :). They evil, and OO unfriendly.