Develop Reference

Java - list iterator not working on IdentityHashMap

I am using an identity hash map to keep track of objects that I have seen before in a custom serializer. I have a while loop in which I attempt to iterate over entries in the map. During the method call addAllFields its possible that objects are added to the keyset of the map. After some debugging I have noticed that my iterator.hasNext() returns false even when I know for a fact that object have been added to the map since the last loop body execution. What am I doing wrong here?
public Document serialize(Object obj) throws Exception {
reset();
addToMap(obj);
Set<Object> keys = map.keySet();
Iterator<Object> iterator = keys.iterator();
while(iterator.hasNext()) {
Object key = iterator.next();
Element objectElement = createObjectElement(key.getClass().getName(), map.get(key));
addAllFields(objectElement, key);
document.getRootElement().addContent(objectElement);
}
return document;
}

You need to use ListIterator, and then if you want to iterate over added elements too you need to do the whole thing in reverse:
ListIterator<Object> iter = keys.listIterator(keys.size());
while (iter.hasPrevious()){
Object prev=iter.previous();
// do your Element creation here
// then use the iter.add() method
iter.add(/*new object*/);
}
NB you may need to wrap you keyset in a different collection type as not all support ListIterator, and will probably need to pass iter as an argument to #addAllFields()
You will likely also need to iterate over entry set rather than keySet

Android: Hashmap concurrent Modification Exception

I keep getting a concurrent modification exception on my code. I'm simply iterating through a hashmap and modifying values. From researching this I found people said to use iterators and iterator.remove, etc. I tried implementing with this and still kept getting the error. I thought maybe multiple threads accessed it? (Although in my code this block is only run in one thread) So I put it in a synchronized block. However, I'm still getting the error.....
Map map= Collections.synchronizedMap(questionNumberAnswerCache);
synchronized (map) {
for (Iterator<Map.Entry<String, Integer>> it = questionNumberAnswerCache.entrySet().iterator(); it.hasNext(); ) {
Map.Entry<String, Integer> entry = it.next();
if (entry.getKey() == null || entry.getValue() == null) {
continue;
} else {
try {
Question me = Question.getQuery().get(entry.getKey());
int i = Activity.getQuery()
.whereGreaterThan(Constants.kQollegeActivityCreatedAtKey, lastUpdated.get("AnswerNumberCache " + entry.getKey()))
.whereEqualTo(Constants.kQollegeActivityTypeKey, Constants.kQollegeActivityTypeAnswer)
.whereEqualTo(Constants.kQollegeActivityQuestionKey, me)
.find().size();
lastUpdated.put("AnswerNumberCache " + entry.getKey(), Calendar.getInstance().getTime());
int old_num = entry.getValue();
entry.setValue(i + old_num);
} catch (ParseException e) {
entry.setValue(0);
}
}
}
}
Error:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:787)
at java.util.HashMap$EntryIterator.next(HashMap.java:824)
at java.util.HashMap$EntryIterator.next(HashMap.java:822)
at com.juryroom.qollege_android_v1.QollegeCache.refreshQuestionAnswerNumberCache(QollegeCache.java:379)
at com.juryroom.qollege_android_v1.QollegeCache.refreshQuestionCaches(QollegeCache.java:267)
at com.juryroom.qollege_android_v1.UpdateCacheService.onHandleIntent(UpdateCacheService.java:28)
at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:65)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:135)
at android.os.HandlerThread.run(HandlerThread.java:61)

What is happening:
The iterator is looping through the map. The map isn't really like a list, because it doesn't care about order. So when you add something to the map it might get inserted into the middle, somewhere in the middle of the objects you already looped through, at the end, etc. So instead of giving you random behavior it fails.
Your solutions:
Synchronized map and synchronized blocks allow you to have two threads going at it at the same time. It doesn't really help here, since the problem is that the same thread is modifying it in an illegal manner.
What you should do:
You could just save the keys you want to modify. Making a map with keys and new values won't be a problem unless this is a really time critical piece of code.
Then you just iterate through the newValues map and update the oldValues map. Since you are not iterating through the map being updated it's not a problem.
Or you could simply iterate just through the keys (for String s : yourMap) and then look up the values you want to change. Since you are just iterating through the keys you are free to change the values (but you can't remove values).
You could also try to use a ConcurrentHashMap which should allow you to modify it, but the behavior is undefined so this is risky. Just changing values shouldn't lead to problems, but if you add or remove you never know if it will end up being iterated through or not.

Create an object, and is locked to it - a good way to shoot yourself in the foot.
I recommend the following code to remove the hash map.
HashMap<Key, Object> hashMap = new HashMap<>();
LinkedList<Key> listToRemove = new LinkedList<>();
for(Map.Entry<Key, Object> s : hashMap.entrySet()) {
if(s.getValue().equals("ToDelete")){
listToRemove.add(s.getKey());
}
}
for(Key s : listToRemove) {
hashMap.remove(s);
}
It's not the most beautiful and fastest option, but it should help you to understand how to work with HashMap.
As you will understand how to work my option. You can learn how to work iterators, how to work iterators in loop. (rather than simply copy-paste)
Iterator it = tokenMap.keySet())
while(it.hasNext()) {
if(/* some condition */) it.remove();
}

I would suggest the following for your use case:
for(Key key : hashMap.keySet()) {
Object value = hashMap.get(key);
if(<condition>){
hashMap.put(key, <new value>);
}
If you are not deleting any entries and just changing the value, this should work for you.

ConcurrentModificationException with LinkedHashMap

Not sure what is triggering a java.util.ConcurrentModificationException when I iterate over the LinkedHashMap structure in the code below. Using the Map.Entry approach works fine. Did not get a good explanation on what is triggering this from the previous posts.
Any help would be appreciated.
import java.util.LinkedHashMap;
import java.util.Map;
public class LRU {
// private Map<String,Integer> m = new HashMap<String,Integer>();
// private SortedMap<String,Integer> lru_cache = Collections.synchronizedSortedMap(new TreeMap<String, Integer>());
private static final int MAX_SIZE = 3;
private LinkedHashMap<String,Integer> lru_cache = new LinkedHashMap<String,Integer>(MAX_SIZE, 0.1F, true){
#Override
protected boolean removeEldestEntry(Map.Entry eldest) {
return(lru_cache.size() > MAX_SIZE);
}
};
public Integer get1(String s){
return lru_cache.get(s);
}
public void displayMap(){
/**
* Exception in thread "main" java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:384)
at LRU.displayMap(LRU.java:23)
at LRU.main(LRU.java:47)
*/
*for(String key : lru_cache.keySet()){
System.out.println(lru_cache.get(key));
}*
// This parser works fine
// for(Map.Entry<String, Integer> kv : lru_cache.entrySet()){
// System.out.println(kv.getKey() + ":" + kv.getValue());
// }
}
public void set(String s, Integer val){
if(lru_cache.containsKey(s)){
lru_cache.put(s, get1(s) + val);
}
else{
lru_cache.put(s, val);
}
}
public static void main(String[] args) {
LRU lru = new LRU();
lru.set("Di", 1);
lru.set("Da", 1);
lru.set("Daa", 1);
lru.set("Di", 1);
lru.set("Di", 1);
lru.set("Daa", 2);
lru.set("Doo", 2);
lru.set("Doo", 1);
lru.set("Sa", 2);
lru.set("Na", 1);
lru.set("Di", 1);
lru.set("Daa", 1);
lru.displayMap();
}
}

Read the Javadoc for LinkedHashMap:
A structural modification is any operation that adds or deletes one or
more mappings or, in the case of access-ordered linked hash maps,
affects iteration order. In insertion-ordered linked hash maps, merely
changing the value associated with a key that is already contained in
the map is not a structural modification. In access-ordered linked
hash maps, merely querying the map with get is a structural
modification.
Since you're passing in true to the LinkedHashMap constructor, it is in access order and when you are trying to get something from it, you are structurally modifying it.
Also note that when you use the enhanced for syntax, you are actually using an iterator. Simplified quote from JLS §14.14.2:
The enhanced for statement has the form:
EnhancedForStatement:
for ( TargetType Identifier : Expression ) Statement
[...]
If the type of Expression is a subtype of Iterable<X> for some type
argument X, then let I be the type java.util.Iterator<X>; otherwise,
let I be the raw type java.util.Iterator.
The enhanced for statement is equivalent to a basic for statement of
the form:
for (I #i = Expression.iterator(); #i.hasNext(); ) {
TargetType Identifier =
(TargetType) #i.next();
Statement
}
#i is an automatically generated identifier that is distinct from any other identifiers (automatically generated or otherwise) that are in
scope (§6.3) at the point where the enhanced for statement occurs.
Also, in the Javadoc for LinkedHashMap:
The iterators returned by the iterator method of the collections
returned by all of this class's collection view methods are
fail-fast: if the map is structurally modified at any time after the iterator is created, in any way except through the iterator's own
remove method, the iterator will throw a
ConcurrentModificationException.
Therefore, when you are calling get on the map, you are performing structural modifications to it, causing the iterator in the enhanced-for to throw an exception. I think you meant to do this, which avoids calling get:
for (Integer i : lru_cache.values()) {
System.out.println(i);
}

You're using an access-ordered linked hash map: from the spec at http://docs.oracle.com/javase/7/docs/api/java/util/LinkedHashMap.html,
A structural modification is any operation that adds or deletes one or
more mappings or, in the case of access-ordered linked hash maps,
affects iteration order. In insertion-ordered linked hash maps, merely
changing the value associated with a key that is already contained in
the map is not a structural modification. In access-ordered linked
hash maps, merely querying the map with get is a structural
modification.)
Simply calling get is enough to be considered a structural modification, triggering the exception. If you use the entrySet() sequence you're only querying the entry and NOT the map, so you don't trigger the ConcurrentModificationException.

In the constructor of LinkedHashMap you pass true to get the LRU behaviour (meaning the eviction policy is access order rather than false for insertion order).
So every time you call get(key) the underlying Map.Entry increments an access counter AND reorders the collection by moving the (last accessed) Map.Entry to the head of the list.
The iterator (implicitly created by the for loop) checks the modified flag, which is different from the copy it took originally, so throws the ConcurrentModificationException.
To avoid this you should use the entrySet() as the implementation is inherited from java.util.HashMap and therefore the iterator doesn't check the modification flags:
for(Map.Entry<String,Integer> e : lru_cache.entrySet()){
System.out.println(e.getValue());
}
Be aware this class isn't threadsafe so in concurrent environments you will need to use an potentially expensive guards like Collections.synchronizedMap(Map). In this scenario a better option might be Google's Guava Cache.

Your code
for(String key : lru_cache.keySet()){
System.out.println(lru_cache.get(key));
}
Actually compiles to:
Iterator<String> it = lru_cache.keySet().iterator();
while (it.hasNext()) {
String key = it.next();
System.out.println(lru_cache.get(key));
}
Next, your LRU cache shrinks itself to MAX_SIZE elements not when calling set(), but when calling get() - above answers explain why.
Thus we have following behavior:
new iterator created to iterate over lru_cache.keySet() collection
lru_cache.get() called to extract element from your cache
get() invocation truncates lru_cache to MAX_SIZE elements (in your case 3)
iterator it becomes invalid due to collection modification and throws on next iteration.

java.util.ConcurrentModificationException : If there are any structural changes (additions, removals, rehashing, etc.) to the underlying list while the iterator exists. The iterator checks to see if the list has changed before each operation. This is known as 'failsafe operation'.
If a thread modifies a collection directly while it is iterating over the collection with a fail-fast iterator, the iterator will throw this exception.Here you cannot call the get() method while using an iterator because calling get() structurally modifies the map and hence the next call to one of the iterators method fails and throws a ConcurrentModificationException.

It is coz of fail-fast behaviour of collections framework also when you modify the list (by adding or removing elements) while traversing a list with this error will be there with Iterator. I came across this error some time back . Refer below threads to for detail info.
ConcurrentModificationException when adding inside a foreach loop in ArrayList
Though this says array list, it applies for most of the collection(s) data strucutres.
Concurrent Modification Exception : adding to an ArrayList
http://docs.oracle.com/javase/6/docs/api/java/util/ConcurrentModificationException.html

How to iterate over a SortedSet to modify items within

lets say I have an List. There is no problem to modify list's item in for loop:
for (int i = 0; i < list.size(); i++) { list.get(i).setId(i); }
But I have a SortedSet instead of list. How can I do the same with it?
Thank you

First of all, Set assumes that its elements are immutable (actually, mutable elements are permitted, but they must adhere to a very specific contract, which I doubt your class does).
This means that generally you can't modify a set element in-place like you're doing with the list.
The two basic operations that a Set supports are the addition and removal of elements. A modification can be thought of as a removal of the old element followed by the addition of the new one:
You can take care of the removals while you're iterating, by using Iterator.remove();
You could accumulate the additions in a separate container and call Set.addAll() at the end.

You cannot modify set's key, because it causes the set rehasing/reordering. So, it will be undefined behaviour how the iteration will run further.
You could remove elements using iterator.remove(). But you cannot add elements, usually better solution is to accumulate them in a new collection and addAll it after the iteration.
Set mySet = ...;
ArrayList newElems = new ArrayList();
for(final Iterator it = mySet.iterator(); it.hasNext(); )
{
Object elem = it.next();
if(...)
newElems.add(...);
else if(...)
it.remove();
...
}
mySet.addAll(newElems);

Since Java 1.6, you're able to use a NavigableSet.

You should use an Iterator or better still the enhanced for-loop syntax (which depends on the class implementing the Iterable interface), irrespective of the Collection you're using. This abstracts away the mechanism used to traverse the collection and allows a new implementation to be substituted in without affecting the iteration routine.
For example:
Set<Foo> set = ...
// Enhanced for-loop syntax
for (Foo foo : set) {
// ...
}
// Iterator approach
Iterator it = set.iterator();
while (it.hasNext()) {
Foo foo = it.next();
}
EDIT
Kan makes a good point regarding modifying the item's key. Assuming that your class's equals() and hashCode() methods are based solely on the "id" attribute (which you're changing) the safest approach would be to explicitly remove these from the Set as you iterate and add them to an "output" Set; e.g.
SortedSet<Foo> input = ...
SortedSet<Foo> output = new TreeSet<Foo>();
Iterator<Foo> it = input.iterator();
while (it.hasNext()) {
Foo foo = it.next();
it.remove(); // Remove from input set before updating ID.
foo.setId(1);
output.add(foo); // Add to output set.
}

You cannot do that. But you may try, maybe you'll succeed, maybe you'll get ConcurrentModificationException. It's very important to remember, that modifying elements while iterating may have unexpected results. You should instead collect that elements in some collection. And after the iteration modify them one by one.

This will only work, if id is not used for equals, or the comperator you used for the sorted set:
int counter = 0;
for(ElementFoo e : set) {
e.setId(counter);
couter++;
}

Performance considerations for keySet() and entrySet() of Map

All,
Can anyone please let me know exactly what are the performance issues between the 2? The site : CodeRanch provides a brief overview of the internal calls that would be needed when using keySet() and get(). But it would be great if anyone can provide exact details about the flow when keySet() and get() methods are used. This would help me understand the performance issues better.

The most common case where using entrySet is preferable over keySet is when you are iterating through all of the key/value pairs in a Map.
This is more efficient:
for (Map.Entry entry : map.entrySet()) {
Object key = entry.getKey();
Object value = entry.getValue();
}
than:
for (Object key : map.keySet()) {
Object value = map.get(key);
}
Because in the second case, for every key in the keySet the map.get() method is called, which - in the case of a HashMap - requires that the hashCode() and equals() methods of the key object be evaluated in order to find the associated value*. In the first case that extra work is eliminated.
Edit: This is even worse if you consider a TreeMap, where a call to get is O(log(n)), i.e. the comparator may need to run log2(n) times (n = size of the Map) before finding the associated value.
*Some Map implementations have internal optimisations that check the objects' identity before the hashCode() and equals() are called.

First of all, this depends entirely on which type of Map you're using. But since the JavaRanch thread talks about HashMap, I'll assume that that's the implementation you're referring to. And let's assume also that you're talking about the standard API implementation from Sun/Oracle.
Secondly, if you're concerned about performance when iterating through your hash map, I suggest you have a look at LinkedHashMap. From the docs:
Iteration over the collection-views of a LinkedHashMap requires time proportional to the size of the map, regardless of its capacity. Iteration over a HashMap is likely to be more expensive, requiring time proportional to its capacity.
HashMap.entrySet()
The source-code for this implementation is available. The implementation basically just returns a new HashMap.EntrySet. A class which looks like this:
private final class EntrySet extends AbstractSet<Map.Entry<K,V>> {
public Iterator<Map.Entry<K,V>> iterator() {
return newEntryIterator(); // returns a HashIterator...
}
// ...
}
and a HashIterator looks like
private abstract class HashIterator<E> implements Iterator<E> {
Entry<K,V> next; // next entry to return
int expectedModCount; // For fast-fail
int index; // current slot
Entry<K,V> current; // current entry
HashIterator() {
expectedModCount = modCount;
if (size > 0) { // advance to first entry
Entry[] t = table;
while (index < t.length && (next = t[index++]) == null);
}
}
final Entry<K,V> nextEntry() {
if (modCount != expectedModCount)
throw new ConcurrentModificationException();
Entry<K,V> e = next;
if (e == null)
throw new NoSuchElementException();
if ((next = e.next) == null) {
Entry[] t = table;
while (index < t.length && (next = t[index++]) == null);
}
current = e;
return e;
}
// ...
}
So there you have it... That's the code dictating what will happen when you iterate through an entrySet. It walks through the entire array, which is as long as the map's capacity.
HashMap.keySet() and .get()
Here you first need to get hold of the set of keys. This takes time proportional to the capacity of the map (as opposed to size for the LinkedHashMap). After this is done, you call get() once for each key. Sure, in the average case, with a good hashCode-implementation this takes constant time. However, it will inevitably require lots of hashCode() and equals() calls, which will obviously take more time than just doing a entry.value() call.

Here is the link to an article comparing the performance of entrySet(), keySet() and values(), and advice regarding when to use each approach.
Apparently the use of keySet() is faster (besides being more convenient) than entrySet() as long as you don't need to Map.get() the values.