I stumbled over this odd bug. Seems like Collections.sort() does not modify the sorted list in a way that enables a detection of concurrent modifications when also iterating over the same list. Example code:
List<Integer> my_list = new ArrayList<Integer>();
my_list.add(2);
my_list.add(1);
for (Integer num : my_list) {
/*
* print list
*/
StringBuilder sb = new StringBuilder();
for (Integer i : my_list)
sb.append(i).append(",");
System.out.println("List: " + sb.toString());
/*
* sort list
*/
System.out.println("CurrentElement: " + num);
Collections.sort(my_list);
}
outputs
List: 2,1,
CurrentElement: 2
List: 1,2,
CurrentElement: 2
One would expect a ConcurrentModificationException, but it is not being raised and the code works although it shouldn't.
Why would it throw ConcurrentModificationException when you are not adding/removing elements from your collection while iterating?
Note that ConcurrentModificationException would only occur when a new element is added in to your collection or remove from your collection while iterating. i.e., when your Collection is Structurally modified.
(Structural modifications are those that change the size of this list,
or otherwise perturb it in such a fashion that iterations in progress
may yield incorrect results.)
sort wouldn't structurally modify your Collection, all it does is modify the order.
Below code would throw ConcurrentModificationException as it add's an extra element into the collection while iterating.
for(Integer num : my_list) {
my_list.add(12);
}
If you look at the source of sort method in Collections class, its not throwing ConcurrentModificationException.
This implementation dumps the specified list into an array, sorts the
array, and iterates over the list resetting each element from the
corresponding position in the array. This avoids the n2 log(n)
performance that would result from attempting to sort a linked list in
place.
public static <T extends Comparable<? super T>> void sort(List<T> list) {
Object[] a = list.toArray();
Arrays.sort(a);
ListIterator<T> i = list.listIterator();
for (int j=0; j<a.length; j++) {
i.next();
i.set((T)a[j]);
}
}
Extract from the book java Generics and Collections:
The policy of the iterators for the Java 2 collections is to fail
fast, as described in Section 11.1: every time they access the backing
collection, they check it for structural modification (which, in
general, means that elements have been added or removed from the
collection). If they detect structural modification, they fail
immediately, throwing ConcurrentModificationException rather than
continuing to attempt to iterate over the modified collection with
unpredictable results.
Speaking of functionality I don't see why it should not throw ConcurrentModificationException. But according to documentation the iterator throws the exception when it notices structural modification and structural modification is defined as:
Structural modifications are those that change the size of the list,
or otherwise perturb it in such a fashion that iterations in progress
may yield incorrect results.
I think there is an argument for claiming that sort rearranging the elements causes the iterator to yield wrong results, but I haven't checked what are right results for iterator defined to be.
Speaking of implementation, it is easy to see why it does not: See the source for ArrayList and Collections:
ArrayList.modCount changes with the so called structural modifications
ListItr methods make a copy of its value in init and check that it hasn't changed in its methods
Collections.sort calls ListItr.set which calls ArratList.set. This last method does not increment modCount
So ListItr.next() sees the same modCount and no exception is thrown.
For Android, it depends on API versions. From API 26, Collections#sort(List<T>, Comparator<? super T>) actually calls List#sort(Comparator<? super E>). So, if you sort ArrayList, you can get ConcurrentModificationException depending on whether you've modified the list in another thread. Here's the source code from java/util/ArrayList.java that throws the exception:
public void sort(Comparator<? super E> c) {
final int expectedModCount = modCount;
Arrays.sort((E[]) elementData, 0, size, c);
if (modCount != expectedModCount) {
throw new ConcurrentModificationException();
}
modCount++;
}
Related
This code:
Collection<String> col = new ArrayList<String>();
col.add("a");
col.add("b");
col.add("c");
for(String s: col){
if(s.equals("b"))
col.remove(1);
System.out.print(s);
}
prints: abc
Meanwhile this one:
ArrayList<String> col = new ArrayList<String>();
col.add("a");
col.add("b");
col.add("c");
for(String s: col){
if(s.equals("b"))
col.remove(1);
System.out.print(s);
}
prints: ab
However it should print the same result...
What's the problem?
Collection has only boolean remove(Object o) method, which removes the passed object if found.
ArrayList also has public E remove(int index), which can remove an element by its index.
Your first snippet calls boolean remove(Object o), which doesn't remove anything, since your ArrayList doesn't contain 1. Your second snippet calls public E remove(int index) and removes the element whose index was 1 (i.e. it removes "b").
The different behavior results from the fact that method overload resolution occurs at compile time and depends on the compile time type of the variable for which you are calling the method. When the type of col is Collection, only remove methods of the Collection interface (and methods inherited by that interface) are considered for overloading resolution.
If you replace col.remove(1) with col.remove("b"), both snippets would behave the same.
As Tamoghna Chowdhury commented, boolean remove(Object o) can accept a primitive argument - int in your case - due to auto-boxing of the int to an Integer instance. For the second snippet, the reason public E remove(int index) is chosen over boolean remove(Object o) is that the method overloading resolution process first attempts to find a matching method without doing auto-boxing/unboxing conversions, so it only considers public E remove(int index).
To safely remove from a Collection while iterating over it, you should use an Iterator.
ArrayList<String> col = new ArrayList<String>();
col.add("a");
col.add("b");
col.add("c");
Iterator<String> i = col.iterator();
while (i.hasNext()) {
String s = i.next(); // must be called before you can call remove
if(s.equals("b"))
i.remove();
System.out.print(s);
}
Regarding, the reason why removal from collection is not working for you while the ArrayList worked is because of the following:
The java.util.ArrayList.remove(int index) method removes the element at the specified position in this list. Shifts any subsequent elements to the left (subtracts one from their indices). Hence, this one worked for you.
The java.util.Collection.remove(Object o) method removes a single instance of the specified element from this collection, if it is present (it is an optional operation). More formally, removes an element e such that (o==null ? e==null : o.equals(e)), if this collection contains one or more such elements. Returns true if this collection contained the specified element (or equivalently, if this collection changed as a result of the call).
Hope, this helps.
Both snippets are broken in different ways!
Case 1 (with Collection<String> col):
Since a Collection is unindexed, the only remove method its interface exposes is Collection.remove(Object o), which removes the specified equal object. Doing col.remove(1); first calls Integer.valueOf(1) to get an Integer object, then asks the list to remove that object. Since the list does not contain such any Integer objects, nothing is removed. Iteration continues normally through the list and abc is printed out.
Case 2 (with ArrayList<String> col):
When col's compile-time type is ArrayList, calling col.remove(1); instead invokes the method ArrayList.remove(int index) to remove the element at the specified position, thus removing b.
Now, why isn't c printed out? In order to loop over a collection with the for (X : Y) syntax, it behind the scenes calls the collection to get an Iterator object. For the Iterator returned by an ArrayList (and most collections) it is not safe to perform structural modifications to the list during iteration – unless you modify it through the methods of the Iterator itself – because the Iterator will become confused and lose track of which element to return next. That can result in elements being iterated multiple times, elements being skipped, or other errors. That's what happens here: element c is present in the list but never printed out because you confused the Iterator.
When an Iterator can detect this problem has happened it will warn you by throwing a ConcurrentModificationException. However, the check that an Iterator does for the problem is optimized for speed, not 100% correctness, and it doesn't always detect the problem. In your code if you change s.equals("b") to s.equals("a") or s.equals("c"), it does throw the exception (although this may be dependent on the particular Java version). From the ArrayList documentation:
The iterators returned by this class's iterator and listIterator methods are fail-fast: if the list is structurally modified at any time after the iterator is created, in any way except through the iterator's own remove or add methods, the iterator will throw a ConcurrentModificationException. Thus, in the face of concurrent modification, the iterator fails quickly and cleanly, rather than risking arbitrary, non-deterministic behavior at an undetermined time in the future.
Note that the fail-fast behavior of an iterator cannot be guaranteed as it is, generally speaking, impossible to make any hard guarantees in the presence of unsynchronized concurrent modification. Fail-fast iterators throw ConcurrentModificationException on a best-effort basis.
To remove elements during iteration, you must change the for (X : Y)-style of loop into a manual loop over an explicit Iterator, using its remove method:
for (Iterator<String> it = col.iterator(); it.hasNext();) {
String s = it.next();
if (s.equals("b"))
it.remove();
System.out.print(s);
}
This is now completely safe. It will iterate all elements exactly once (printing abc), while element b will be removed.
If you want, you can achieve the same effect without an Iterator using an int i-style loop, if you carefully adjust the index after any removals:
for (int i = 0; i < col.size(); i++) {
String s = col.get(i);
if (s.equals("b")) {
col.remove(i);
i--;
}
System.out.print(s);
}
I have a very basic question.
I have created simple ArrayList and I am removing the item while iterating using for-each loop. It gives me java.util.ConcurrentModificationException because I can't remove an item while iterating but when I un-comment the if condition it works fine.
Please can anybody explain me how for-each works in this way.
ArrayList<String> list1 = new ArrayList<String>();
list1.add("Hello");
list1.add("World");
list1.add("Good Evening");
for (String s : list1) {
//if (s.equals("World")) {
list1.remove(1);
//}
}
If I change it to list1.remove(2); or list1.remove(0); then also its working fine.
Note: This is sample code and I know it will work fine using Iterator. My sole purpose of this question is to know how method remove() works perfectly if condition is un-commented no matter what index you are removing from the list.
The list has a variable called modCount, which means "modification count". Whenever you call remove (or perform other structural modifications), it increments the modCount.
The iterator can't keep track of its position in the list if you are adding or removing elements without telling the iterator. So as a safety check, at the start of iteration, the iterator makes a note of the modCount, saving it as expectedModCount. When each item is read from the iterator, the iterator checks to make sure the modCount still equals the expected value, and throws an exception if it doesn't.
Usually, this will successfully cause the exception to be thrown if the list is unsafely modified during iteration. However, it's not sufficient in this case when the if statement is enabled. After your code has read "World", that item is removed, and so the list now contains ["Hello", Good Evening"]. The iterator is still at position 1 (which now contains "Good Evening") and when it tries to read the next item, it finds it has now reached the end of the list, so it doesn't bother to check the modCount. Hence, no exception.
Note the caveat in the ConcurrentModificationException documentation: "It is, generally speaking, impossible to make any hard guarantees in the presence of unsynchronized concurrent modification. Fail-fast operations throw ConcurrentModificationException on a best-effort basis."
Even if it doesn't happen to throw the exception in this case, the code is still wrong. To remove an element while iterating, you must use the iterator's own remove method:
for (Iterator<String> it = list1.iterator(); it.hasNext();) {
String s = it.next();
if (s.equals("World")) {
it.remove();
}
}
That way, the iterator knows that the list has changed and can still iterate correctly.
Alternatively, you can iterate from a temporary copy of the list:
for (String s : new ArrayList<>(list1)) {
if (s.equals("World")) {
list1.remove(...);
}
}
Although, in this simple case, you don't even need to do that; you can just write:
list1.remove("World");
You can also use an index-based removal. The drawback of this solution is that the list1.size() gets evaluated during every loop iteration. The positive thing is that removing an item from a List by its index is faster.
for (int i = 0; i < list1.size(); /* i incremented in loop body */) {
if ("World".equals(list1.get(i))) {
list1.remove(i);
}
else {
i++;
}
}
Use an Iterator and call remove():
Iterator<String> iter = list1.iterator();
while (iter.hasNext()) {
String str = iter.next();
if (someCondition)
iter.remove();
}
There's this strange code in AbstractCollection:
public Object[] toArray() {
// Estimate size of array; be prepared to see more or fewer elements
Object[] r = new Object[size()];
Iterator<E> it = iterator();
for (int i = 0; i < r.length; i++) {
if (! it.hasNext()) // fewer elements than expected
return Arrays.copyOf(r, i);
r[i] = it.next();
}
return it.hasNext() ? finishToArray(r, it) : r;
}
The part "be prepared to see more or fewer elements" is IMHO a pure non-sense:
In case the collection changes in the meantime, the iterator throw a ConcurrentModification exception anyway.
I haven't found any non-concurrent subclass supporting this, especially
the ArrayList uses Arrays.copyOf(elementData, size) which could (due to visibility issues) copy a bunch of nulls instead of the data in case of resizing,
the LinkedList throws an ArrayIndexOutOfBoundsException if you're lucky enough.
Am I overlooking something?
Would you support this feature in your collection (meant for general use)?
From the JAVA DOC of toArray()
This implementation returns an array containing all the elements
returned by this collection's iterator, in the same order, stored in
consecutive elements of the array, starting with index 0. The length
of the returned array is equal to the number of elements returned by
the iterator, even if the size of this collection changes during
iteration, as might happen if the collection permits concurrent
modification during iteration.The size method is called only as an
optimization hint; the correct result is returned even if the iterator
returns a different number of elements.
Not sure what is triggering a java.util.ConcurrentModificationException when I iterate over the LinkedHashMap structure in the code below. Using the Map.Entry approach works fine. Did not get a good explanation on what is triggering this from the previous posts.
Any help would be appreciated.
import java.util.LinkedHashMap;
import java.util.Map;
public class LRU {
// private Map<String,Integer> m = new HashMap<String,Integer>();
// private SortedMap<String,Integer> lru_cache = Collections.synchronizedSortedMap(new TreeMap<String, Integer>());
private static final int MAX_SIZE = 3;
private LinkedHashMap<String,Integer> lru_cache = new LinkedHashMap<String,Integer>(MAX_SIZE, 0.1F, true){
#Override
protected boolean removeEldestEntry(Map.Entry eldest) {
return(lru_cache.size() > MAX_SIZE);
}
};
public Integer get1(String s){
return lru_cache.get(s);
}
public void displayMap(){
/**
* Exception in thread "main" java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:384)
at LRU.displayMap(LRU.java:23)
at LRU.main(LRU.java:47)
*/
*for(String key : lru_cache.keySet()){
System.out.println(lru_cache.get(key));
}*
// This parser works fine
// for(Map.Entry<String, Integer> kv : lru_cache.entrySet()){
// System.out.println(kv.getKey() + ":" + kv.getValue());
// }
}
public void set(String s, Integer val){
if(lru_cache.containsKey(s)){
lru_cache.put(s, get1(s) + val);
}
else{
lru_cache.put(s, val);
}
}
public static void main(String[] args) {
LRU lru = new LRU();
lru.set("Di", 1);
lru.set("Da", 1);
lru.set("Daa", 1);
lru.set("Di", 1);
lru.set("Di", 1);
lru.set("Daa", 2);
lru.set("Doo", 2);
lru.set("Doo", 1);
lru.set("Sa", 2);
lru.set("Na", 1);
lru.set("Di", 1);
lru.set("Daa", 1);
lru.displayMap();
}
}
Read the Javadoc for LinkedHashMap:
A structural modification is any operation that adds or deletes one or
more mappings or, in the case of access-ordered linked hash maps,
affects iteration order. In insertion-ordered linked hash maps, merely
changing the value associated with a key that is already contained in
the map is not a structural modification. In access-ordered linked
hash maps, merely querying the map with get is a structural
modification.
Since you're passing in true to the LinkedHashMap constructor, it is in access order and when you are trying to get something from it, you are structurally modifying it.
Also note that when you use the enhanced for syntax, you are actually using an iterator. Simplified quote from JLS §14.14.2:
The enhanced for statement has the form:
EnhancedForStatement:
for ( TargetType Identifier : Expression ) Statement
[...]
If the type of Expression is a subtype of Iterable<X> for some type
argument X, then let I be the type java.util.Iterator<X>; otherwise,
let I be the raw type java.util.Iterator.
The enhanced for statement is equivalent to a basic for statement of
the form:
for (I #i = Expression.iterator(); #i.hasNext(); ) {
TargetType Identifier =
(TargetType) #i.next();
Statement
}
#i is an automatically generated identifier that is distinct from any other identifiers (automatically generated or otherwise) that are in
scope (§6.3) at the point where the enhanced for statement occurs.
Also, in the Javadoc for LinkedHashMap:
The iterators returned by the iterator method of the collections
returned by all of this class's collection view methods are
fail-fast: if the map is structurally modified at any time after the iterator is created, in any way except through the iterator's own
remove method, the iterator will throw a
ConcurrentModificationException.
Therefore, when you are calling get on the map, you are performing structural modifications to it, causing the iterator in the enhanced-for to throw an exception. I think you meant to do this, which avoids calling get:
for (Integer i : lru_cache.values()) {
System.out.println(i);
}
You're using an access-ordered linked hash map: from the spec at http://docs.oracle.com/javase/7/docs/api/java/util/LinkedHashMap.html,
A structural modification is any operation that adds or deletes one or
more mappings or, in the case of access-ordered linked hash maps,
affects iteration order. In insertion-ordered linked hash maps, merely
changing the value associated with a key that is already contained in
the map is not a structural modification. In access-ordered linked
hash maps, merely querying the map with get is a structural
modification.)
Simply calling get is enough to be considered a structural modification, triggering the exception. If you use the entrySet() sequence you're only querying the entry and NOT the map, so you don't trigger the ConcurrentModificationException.
In the constructor of LinkedHashMap you pass true to get the LRU behaviour (meaning the eviction policy is access order rather than false for insertion order).
So every time you call get(key) the underlying Map.Entry increments an access counter AND reorders the collection by moving the (last accessed) Map.Entry to the head of the list.
The iterator (implicitly created by the for loop) checks the modified flag, which is different from the copy it took originally, so throws the ConcurrentModificationException.
To avoid this you should use the entrySet() as the implementation is inherited from java.util.HashMap and therefore the iterator doesn't check the modification flags:
for(Map.Entry<String,Integer> e : lru_cache.entrySet()){
System.out.println(e.getValue());
}
Be aware this class isn't threadsafe so in concurrent environments you will need to use an potentially expensive guards like Collections.synchronizedMap(Map). In this scenario a better option might be Google's Guava Cache.
Your code
for(String key : lru_cache.keySet()){
System.out.println(lru_cache.get(key));
}
Actually compiles to:
Iterator<String> it = lru_cache.keySet().iterator();
while (it.hasNext()) {
String key = it.next();
System.out.println(lru_cache.get(key));
}
Next, your LRU cache shrinks itself to MAX_SIZE elements not when calling set(), but when calling get() - above answers explain why.
Thus we have following behavior:
new iterator created to iterate over lru_cache.keySet() collection
lru_cache.get() called to extract element from your cache
get() invocation truncates lru_cache to MAX_SIZE elements (in your case 3)
iterator it becomes invalid due to collection modification and throws on next iteration.
java.util.ConcurrentModificationException : If there are any structural changes (additions, removals, rehashing, etc.) to the underlying list while the iterator exists. The iterator checks to see if the list has changed before each operation. This is known as 'failsafe operation'.
If a thread modifies a collection directly while it is iterating over the collection with a fail-fast iterator, the iterator will throw this exception.Here you cannot call the get() method while using an iterator because calling get() structurally modifies the map and hence the next call to one of the iterators method fails and throws a ConcurrentModificationException.
It is coz of fail-fast behaviour of collections framework also when you modify the list (by adding or removing elements) while traversing a list with this error will be there with Iterator. I came across this error some time back . Refer below threads to for detail info.
ConcurrentModificationException when adding inside a foreach loop in ArrayList
Though this says array list, it applies for most of the collection(s) data strucutres.
Concurrent Modification Exception : adding to an ArrayList
http://docs.oracle.com/javase/6/docs/api/java/util/ConcurrentModificationException.html
SITUATION: I have a TreeSet of custom Objects and I have also used a custom Comparator. I have created an iterator to use on this TreeSet.
TreeSet<Custom> ts=new TreeSet<Custom>();
Iterator<Custom> itr=ts.iterator();
while(itr.hasNext()){
Custom c=itr.next();
//Code to add a new element to the TreeSet ts
}
QUESTION: Well I want to know that if I add a new element to the TreeSet within the while loop, then will that new element get sorted immediately. In other words, if I add a new element within the while loop and it is less than the one which I am currently holding in c, then in the next iteration will I be getting the same element in c as in the last iteration?(since after sorting, the newly added element will occupy a place somewhere before the current element).
If you add an element during your iteration, your next iterator call will likely throw a ConcurrentModificationException. See the fail-fast behavior in TreeSet docs.
To iterate and add elements, you could copy first to another set:
TreeSet<Custom> ts = ...
TreeSet<Custom> tsWithExtra = new TreeSet(ts);
for (Custom c : ts) {
// possibly add to tsWithExtra
}
// continue, using tsWithExtra
or create a separate collection to be merged with ts after iteration, as Colin suggests.
You will get a java.util.ConcurrentModificationException if you add an element into the TreeSet inside while loop.
Set<String> ts = new TreeSet<>();
ts.addAll(Arrays.asList(new String[]{"abb", "abd", "abg"}));
Iterator<String> itr = ts.iterator();
while(itr.hasNext()){
String s = itr.next();
System.out.println("s: " + s);
if (s.equals("abd"))
ts.add("abc");
}
###Output
Exception in thread "main" java.util.ConcurrentModificationException
public static void main(String[] args) {
TreeSet<Integer> ts=new TreeSet<Integer>();
ts.add(2);
ts.add(4);
ts.add(0);
Iterator<Integer> itr=ts.iterator();
while(itr.hasNext()){
Integer c=itr.next();
System.out.println(c);
//Code
ts.add(1);
}
}
Exception in thread "main" java.util.ConcurrentModificationException
This will come to all collections like List , Map , Set
Because when iterator starts it may be putting some lock on it .
if you iterate list using iterator then this exception will come. I think otherwise this loop will be infinite as you are adding element whole iterating.
Consider without iterator:
public static void main(String[] args) {
List<Integer> list=new ArrayList<Integer>();
list.add(2);
list.add(4);
list.add(0);
for (int i = 0; i < 3; i++) {
System.out.println(list.get(i));
list.add(3);
}
System.out.println("Size" +list.size());
}
this will be fine .
In order to avoid the ConcurrentModificationException you might want to check out my UpdateableTreeSet. I have even added a new test case showing how to add elements during a loop. To be more exact, you mark new elements for a later, deferred update of the set. This works quite nicely. Basically you do something like
for (MyComparableElement element : myUpdateableTreeSet) {
if (someCondition) {
// Add new element (deferred)
myUpdateableTreeSet.markForUpdate(
new MyComparableElement("foo", "bar", 1, 2)
);
}
}
// Perform bulk update
myUpdateableTreeSet.updateMarked();
I guess this is quite exactly what you need. :-)
To prevent the ConcurrentModificationException while walking.
Below is my version to allow high frequency insertion into the TreeSet() and allow concurrently iterate on it. This class use a extra queue to store the inserting object when the TreeSet is being iterating.
public class UpdatableTransactionSet {
TreeSet <DepKey> transactions = new TreeSet <DepKey> ();
LinkedList <DepKey> queue = new LinkedList <DepKey> ();
boolean busy=false;
/**
* directly call it
* #param e
*/
void add(DepKey e) {
boolean bb = getLock();
if(bb) {
transactions.add(e);
freeLock();
} else {
synchronized(queue) {
queue.add(e);
}
}
}
/**
* must getLock() and freeLock() while call this getIterator function
* #return
*/
Iterator<DepKey> getIterator() {
return null;
}
synchronized boolean getLock() {
if(busy) return false;
busy = true;
return true;
}
synchronized void freeLock() {
synchronized(queue) {
for(DepKey e:queue) {
transactions.add(e);
}
}
busy = false;
}
}
While the question has already been answered, I think the most satisfactory answer lies in javadoc of TreeSet itself
The iterators returned by this class's iterator method are fail-fast: if the set is modified at any time after the iterator is created, in any way except through the iterator's own remove method, the iterator will throw a ConcurrentModificationException. Thus, in the face of concurrent modification, the iterator fails quickly and cleanly, rather than risking arbitrary, non-deterministic behavior at an undetermined time in the future.
Note that the fail-fast behavior of an iterator cannot be guaranteed as it is, >generally speaking, impossible to make any hard guarantees in the presence of unsynchronized concurrent modification. Fail-fast iterators throw ConcurrentModificationException on a best-effort basis. Therefore, it would be wrong to write a program that depended on this exception for its correctness: the fail-fast behavior of iterators should be used only to detect bugs.
To avoid the concurrent modification error that's bound to occur when you're doing the insertion, you could also create a temporary copy of the Set, iterate through the copy instead, and modify the original.