Develop Reference

Java: get+clear atomic for map

I would like to implement the following logic:
-the following structure is to be used
//Map<String, CopyOnWriteArrayList> keeping the pending updates
//grouped by the id of the updated object
final Map<String, List<Update>> updatesPerId = new ConcurrentHashMap<>();
-n producers will add updates to updatesPerId map (for the same id, 2 updates can be added at the same time)
-one TimerThread will run from time to time and has to process the received updates. Something like:
final Map<String, List<Update>> toBeProcessed = new HashMap<>(updatesPerId);
updatesPerId.clear();
// iterate over toBeProcessed and process them
Is there any way to make this logic thread safe without synchronizing the adding logic from producers and the logic from timerThread(consumer)? I am thinking about an atomic clear+get but it seems that ConcurrentMap does not provide something like that.
Also, I have to mention that updates should be kept by updated object id so I cannot replace the map with a queue or something else.
Any ideas?
Thanks!

You can leverage the fact that ConcurrentHashMap.compute executes atomically.
You can put into the updatesPerId like so:
updatesPerId.compute(k, (k, list) -> {
if (list == null) list = new ArrayList<>();
// ... add to the list
// Return a non-null list, so the key/value pair is stored in the map.
return list;
});
This is not using computeIfAbsent then adding to the return value, which would not be atomic.
Then in your thread to remove things:
for (String key : updatesPerId.keySet()) {
List<Update> list = updatesPerId.put(key, null);
updatesPerId.compute(key, (k, list) -> {
// ... Process the contents of the list.
// Removes the key/value pair from the map.
return null;
});
}
So, adding a key to the list (or processing all the values for that key) might block if you so happen to try to process the key in both places at once; otherwise, it will not be blocked.
Edit: as pointed out by #StuartMarks, it might be better to simply get all things out of the map first, and then process them later, in order to avoid blocking other threads trying to add:
Map<String, List<Update>> newMap = new HashMap<>();
for (String key : updatesPerId.keySet()) {
newMap.put(key, updatesPerId.remove(key));
}
// ... Process entries in newMap.

I'd suggest using LinkedBlockingQueue instead of CopyOnWriteArrayList as the map value. With COWAL, adds get successively more expensive, so adding N elements results in N^2 performance. LBQ addition is O(1). Also, LBQ has drainTo which can be used effectively here. You could do this:
final Map<String, Queue<Update>> updatesPerId = new ConcurrentHashMap<>();
Producer:
updatesPerId.computeIfAbsent(id, LinkedBlockingQueue::new).add(update);
Consumer:
updatesPerId.forEach((id, queue) -> {
List<Update> updates = new ArrayList<>();
queue.drainTo(updates);
processUpdates(id, updates);
});
This is somewhat different from what you had suggested. This technique processes the updates for each id, but lets producers continue to add updates to the map while this is going on. This leaves map entries and queues in the map for each id. If the ids end up getting reused a lot, the number of map entries will plateau at a high-water mark.
If new ids are continually coming in, and old ids becoming disused, the map will grow continually, which probably isn't what you want. If this is the case you could use the technique in Andy Turner's answer.
If the consumer really needs to snapshot and clear the entire map, I think you have to use locking, which you wanted to avoid.

Is there any way to make this logic thread safe without synchronizing the adding logic from producers and the logic from timerThread(consumer)?
In short, no - depending on what you mean by "synchronizing".
The easiest way is to wrap your Map into a class of your own.
class UpdateManager {
Map<String,List<Update>> updates = new HashMap<>();
public void add(Update update) {
synchronized (updates) {
updates.computeIfAbsent(update.getKey(), k -> new ArrayList<>()).add(update);
}
}
public Map<String,List<Update>> getUpdatesAndClear() {
synchronized (updates) {
Map<String,List<Update>> copy = new HashMap<>(updates);
updates.clear();
return copy;
}
}
}

How can I process all elements of a Queue without blocking using streams?

Suppose I have a Queue<String> and I want to empty the current contents of the queue and do something with each element. Using a loop I could do something like:
while (true) {
String element = queue.poll();
if (element == null) {
return;
}
System.out.println(element);
}
This feels a bit ugly. Could I do this better with streams?
Note that there may be other threads accessing the queue at the same time, so relying on the size of the queue to know how many items to poll would be error prone.

Since you asked about “without blocking”, it seems you are referring to a BlockingQueue. In that case, it’s recommended to avoid repeatedly calling poll().
Instead, transfer all pending elements to a local collection in one go, then process them:
List<String> tmp = new ArrayList<>();
queue.drainTo(tmp);
tmp.forEach(System.out::println);
You may also avoid synchronizing on System.out (implicitly) multiple times:
List<String> tmp=new ArrayList<>();
queue.drainTo(tmp);
System.out.println(tmp.stream().collect(Collectors.joining(System.lineSeparator())));
or
List<String> tmp=new ArrayList<>();
queue.drainTo(tmp);
System.out.println(String.join(System.lineSeparator(), tmp));
(though that doesn’t bear a stream operation)…

You don't need to use streams to make the code less ugly (a stream solution would probably be more ugly).
String s = null;
while((s = queue.poll()) != null)
System.out.println(s);

The best answer I can come up with at present is the following:
StreamEx.generate(() -> queue.poll())
.takeWhile(Objects::nonNull)
.forEach(System.out::println);
but that uses a library (StreamEx). Can we do this with vanilla Java?

As per my understanding, if you want to use stream then you can't modify the same collection on which you are doing operations.
Below code might help you:
queue.stream().forEach(e -> {
System.out.println(e);
});
queue.clear();

I don’t think you want the complication, but if I am wrong, you may do:
Spliterator<String> spliterator = new Spliterators.AbstractSpliterator<String>(queue.size(), 0) {
#Override
public boolean tryAdvance(Consumer<? super String> action) {
String element = queue.poll();
if (element == null) {
return false;
} else {
action.accept(element);
return true;
}
}
};
StreamSupport.stream(spliterator, false).forEach(System.out::println);
You may look into the docs of Spliterators and StreamSupport for numerous possible refinements.

Android: Hashmap concurrent Modification Exception

I keep getting a concurrent modification exception on my code. I'm simply iterating through a hashmap and modifying values. From researching this I found people said to use iterators and iterator.remove, etc. I tried implementing with this and still kept getting the error. I thought maybe multiple threads accessed it? (Although in my code this block is only run in one thread) So I put it in a synchronized block. However, I'm still getting the error.....
Map map= Collections.synchronizedMap(questionNumberAnswerCache);
synchronized (map) {
for (Iterator<Map.Entry<String, Integer>> it = questionNumberAnswerCache.entrySet().iterator(); it.hasNext(); ) {
Map.Entry<String, Integer> entry = it.next();
if (entry.getKey() == null || entry.getValue() == null) {
continue;
} else {
try {
Question me = Question.getQuery().get(entry.getKey());
int i = Activity.getQuery()
.whereGreaterThan(Constants.kQollegeActivityCreatedAtKey, lastUpdated.get("AnswerNumberCache " + entry.getKey()))
.whereEqualTo(Constants.kQollegeActivityTypeKey, Constants.kQollegeActivityTypeAnswer)
.whereEqualTo(Constants.kQollegeActivityQuestionKey, me)
.find().size();
lastUpdated.put("AnswerNumberCache " + entry.getKey(), Calendar.getInstance().getTime());
int old_num = entry.getValue();
entry.setValue(i + old_num);
} catch (ParseException e) {
entry.setValue(0);
}
}
}
}
Error:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:787)
at java.util.HashMap$EntryIterator.next(HashMap.java:824)
at java.util.HashMap$EntryIterator.next(HashMap.java:822)
at com.juryroom.qollege_android_v1.QollegeCache.refreshQuestionAnswerNumberCache(QollegeCache.java:379)
at com.juryroom.qollege_android_v1.QollegeCache.refreshQuestionCaches(QollegeCache.java:267)
at com.juryroom.qollege_android_v1.UpdateCacheService.onHandleIntent(UpdateCacheService.java:28)
at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:65)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:135)
at android.os.HandlerThread.run(HandlerThread.java:61)

What is happening:
The iterator is looping through the map. The map isn't really like a list, because it doesn't care about order. So when you add something to the map it might get inserted into the middle, somewhere in the middle of the objects you already looped through, at the end, etc. So instead of giving you random behavior it fails.
Your solutions:
Synchronized map and synchronized blocks allow you to have two threads going at it at the same time. It doesn't really help here, since the problem is that the same thread is modifying it in an illegal manner.
What you should do:
You could just save the keys you want to modify. Making a map with keys and new values won't be a problem unless this is a really time critical piece of code.
Then you just iterate through the newValues map and update the oldValues map. Since you are not iterating through the map being updated it's not a problem.
Or you could simply iterate just through the keys (for String s : yourMap) and then look up the values you want to change. Since you are just iterating through the keys you are free to change the values (but you can't remove values).
You could also try to use a ConcurrentHashMap which should allow you to modify it, but the behavior is undefined so this is risky. Just changing values shouldn't lead to problems, but if you add or remove you never know if it will end up being iterated through or not.

Create an object, and is locked to it - a good way to shoot yourself in the foot.
I recommend the following code to remove the hash map.
HashMap<Key, Object> hashMap = new HashMap<>();
LinkedList<Key> listToRemove = new LinkedList<>();
for(Map.Entry<Key, Object> s : hashMap.entrySet()) {
if(s.getValue().equals("ToDelete")){
listToRemove.add(s.getKey());
}
}
for(Key s : listToRemove) {
hashMap.remove(s);
}
It's not the most beautiful and fastest option, but it should help you to understand how to work with HashMap.
As you will understand how to work my option. You can learn how to work iterators, how to work iterators in loop. (rather than simply copy-paste)
Iterator it = tokenMap.keySet())
while(it.hasNext()) {
if(/* some condition */) it.remove();
}

I would suggest the following for your use case:
for(Key key : hashMap.keySet()) {
Object value = hashMap.get(key);
if(<condition>){
hashMap.put(key, <new value>);
}
If you are not deleting any entries and just changing the value, this should work for you.

How can I extract ArrayList from HashMap and loop through it in Java?

I have set up a HashMap like so:
Map<String, ArrayList<String>> theAccused = new HashMap<String, ArrayList<String>>();
... and I populate this by storing for every name (key), a list of names (value). So:
ArrayList<String> saAccused = new ArrayList<String>();
// populate 'saAccused' ArrayList
...
// done populating
theAccused.put(sAccuser, saAccused);
So now, I want to look through all of the entries in the HashMap and see if (for each 'sAccuser'), the list 'saAccused' contains a certain name. This is my failed attempt so far:
Set<String> setAccusers = theAccused.keySet();
Iterator<String> iterAccusers = setAccusers.iterator();
iterAccusers.next();
ArrayList<String> saTheAccused;
// check if 'sAccuser' has been accused by anyone before
for (int i = 0; i < theAccused.size(); i++) {
saTheAccused = theAccused.get(iterAccusers);
if (saTheAccused.contains(sAccuser)) {
}
iterAccusers.next();
}
... however I'm not sure how the Set and Iterator classes work :/ The problem is that I don't have the "values"... the names... the 'sAccuser's... for the HashMap available.
In a nutshell, I want to iterate through the HashMap and check if a specific name is stored in any of the lists. So how can I do this? Let me know if you need me to go into further detail or clear up any confusion.
Thanks.

In a nutshell, I want to iterate through the HashMap and check if a specific name is stored in any of the lists. So how can I do this?
There's two ways of iterating through the map that might be of interest here. Firstly, you can iterate through all of the mappings (i.e. pairs of key-value relations) using the entrySet() method, which will let you know what the key is for each arraylist. Alternatively, if you don't need the key, you can simply get all of the lists in turn via the values() method. Using the first option might look something like this:
for (Map.Entry<String, ArrayList<String>> entry : theAccused.entrySet())
{
String sListName = entry.getKey();
ArrayList<String> saAccused = entry.getValue();
if (saAccused.contains(sAccuser))
{
// Fire your logic for when you find a match, which can
// depend on the list's key (name) as well
}
}
To answer the broader questions - the Set interface simply represents an (unordered) collection of non-duplicated values. As you can see by the linked Javadoc, there are methods available that you might expect for such an unordered collection. An Iterator is an object that traverses some data structure presenting each element in turn. Typical usage of an iterator would look something like the following:
Iterator<?> it = ...; // get the iterator somehow; often by calling iterator() on a Collection
while (it.hasNext())
{
Object obj = it.next();
// Do something with the obj
}
that is, check whether the iterator is nonexhausted (has more elements) then call the next() method to get that element. However, since the above pattern is so common, it can be elided with Java 5's foreach loop, sparing you from dealing with the iterator itself, as I took advantage of in my first example.

Something like this?
for (List<String> list : theAccused.values()) {
if (list.contains("somename")) {
// found somename
}
}

This should make it work:
saTheAccused = theAccused.get(iterAccused.next());
However, to make your code more readable, you can have either:
for (List<String> values : theAccused.values()) {
if (value.contains(sAcuser)) {
..
}
}
or, if you need the key:
for (String key : theAccused.keySet()) {
List<String> accused = theAccused.get(key);
if (accused.contains(sAccuser)) {
}
}

You need to use the value from Iterator.next() to index into the Map.
String key = iterAccusers.next();
saTheAccused = theAccused.get(key);
Currently you're getting values from the Map based on the iterator, not the values returned by the iterator.

It sounds like you need to do two things: first, find out if a given name is "accused", and second, find out who the accuser is. For that, you need to iterate over the Entry objects within your Map.
for (Entry<String, List<String>> entry : theAccused.entrySet()) {
if (entry.getValue().contains(accused)) {
return entry.getKey();
}
}
return null; // Or throw NullPointerException, or whatever.
In this loop, the Entry object holds a single key-value mapping. So entry.getValue() contains the list of accused, and entry.getKey() contains their accuser.

Make a method that does it:
private String findListWithKeyword(Map<String, ArrayList<String>> map, String keyword) {
Iterator<String> iterAccusers = map.keySet().iterator();
while(iterAccusers.hasNext()) {
String key = iterAccusers.next();
ArrayList<String> list = theAccused.get(key);
if (list.contains(keyword)) {
return key;
}
}
}
And when you call the method:
String key = findListWithKeyword(map, "foobar");
ArrayList<String> theCorrectList = map.get(key);

Hashmap.keySet(), foreach, and remove

I know that it's typically a big no-no to remove from a list using java's "foreach" and that one should use iterator.remove(). But is it safe to remove() if I'm looping over a HashMap's keySet()? Like this:
for(String key : map.keySet()) {
Node n = map.get(key).optimize();
if(n == null) {
map.remove(key);
} else {
map.put(key, n);
}
}

EDIT:
I hadn't noticed that you weren't really adding to the map - you were just changing the value within the entry. In this case, pstanton's (pre-edit1) solution is nearly right, but you should call setValue on the entry returned by the iterator, rather than calling map.put. (It's possible that map.put will work, but I don't believe it's guaranteed - whereas the docs state that entry.setValue will work.)
for (Iterator<Map.Entry<String, Node>> it = map.entrySet().iterator();
it.hasNext();)
{
Map.Entry<String, Node> entry = it.next();
Node n = entry.getValue().optimize();
if(n == null)
{
it.remove();
}
else
{
entry.setValue(n);
}
}
(It's a shame that entry doesn't have a remove method, otherwise you could still use the enhanced for loop syntax, making it somewhat less clunky.)
Old answer
(I've left this here for the more general case where you just want to make arbitrary modifications.)
No - you should neither add to the map nor remove from it directly. The set returned by HashSet.keySet() is a view onto the keys, not a snapshot.
You can remove via the iterator, although that requires that you use the iterator explicitly instead of via an enhanced for loop.
One simple option is to create a new set from the original:
for (String key : new HashSet<String>(map.keySet())) {
...
}
At this point you're fine, because you're not making any changes to the set.
EDIT: Yes, you can definitely remove elements via the key set iterator. From the docs for HashMap.keySet():
The set supports element removal,
which removes the corresponding
mapping from the map, via the
Iterator.remove, Set.remove,
removeAll, retainAll, and clear
operations. It does not support the
add or addAll operations.
This is even specified within the Map interface itself.
1 I decided to edit my answer rather than just commenting on psanton's, as I figured the extra information I'd got for similar-but-distinct situations was sufficiently useful to merit this answer staying.

you should use the entry set:
for(Iterator<Map.Entry<String, Node>> it = map.entrySet().iterator(); it.hasNext();)
{
Map.Entry<String, Node> entry = it.next();
Node n = entry.getValue().optimize();
if(n == null)
it.remove();
else
entry.setValue(n);
}
EDIT fixed code