Develop Reference

Java TreeMap put vs HashMap put, custom Object as key

My objective was to use the TreeMap to make Box key objects sorted by Box.volume attribute while able to put keys distinct by the Box.code. Is it not possible in TreeMap?
As per below test 1, HashMap put works as expected, HashMap keeps both A, B key objects, but in test 2, TreeMap put doesn't treat D as a distinct key, it replaces C's value, note that i used the TreeMap comparator as Box.volume, because i want keys to be sorted by volume in TreeMap.
import java.util.*;
public class MapExample {
public static void main(String[] args) {
//test 1
Box b1 = new Box("A");
Box b2 = new Box("B");
Map<Box, String> hashMap = new HashMap<>();
hashMap.put(b1, "test1");
hashMap.put(b2, "test2");
hashMap.entrySet().stream().forEach(o-> System.out.println(o.getKey().code+":"+o.getValue()));
//output
A:test1
B:test2
//test 2
Box b3 = new Box("C");
Box b4 = new Box("D");
Map<Box, String> treeMap = new TreeMap<>((a,b)-> Integer.compare(a.volume, b.volume));
treeMap.put(b3, "test3");
treeMap.put(b4, "test4");
treeMap.entrySet().stream().forEach(o-> System.out.println(o.getKey().code+":"+o.getValue()));
//output
C:test4
}
}
class Box {
String code;
int volume;
public Box(String code) {
this.code = code;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Box box = (Box) o;
return code.equals(box.code);
}
#Override
public int hashCode() {
return Objects.hash(code);
}
}
Thank you

TreeMap considers 2 keys for which the comparison method returns 0 to be identical, even if they are not equal to each other, so your current TreeMap cannot contain two keys with the same volume.
If you want to keep the ordering by volume and still have multiple keys with the same volume in your Map, change your Comparator's comparison method to compare the Box's codes when the volumes are equal. This way it will only return 0 if the keys are equal.
Map<Box, String> treeMap = new TreeMap<>((a,b)-> a.volume != b.volume ? Integer.compare(a.volume, b.volume) : a.code.compareTo(b.code));
Now the output is:
C:test3
D:test4

b3 and b4 has the same volume, that is 0 (int default value).
For it work, assign a value to the Box volume variables before comparing.

TreeSet to find k most frequent words in a book?

The commonly occurring question of finding k most frequent words in a book ,(words can dynamically be added), is usually solved using combination of trie and heap.
However, I think even using a TreeSet should suffice and be cleaner with log(n) performance for insert and retrievals.
The treeset would contain a custom object:
class MyObj implements Comparable{
String value;
int count;
public int incrementCount(){count++;}
//override equals and hashcode to make this object unique by string 'value'
//override compareTo to compare count
}
Whenever we insert object in the treeset we first check if the element is already present in the treeset if yes then we get the obj and increment the count variable of that object.
Whenever, we want to find the k largest words , we just iterate over the first k elements of the treeset
What are your views on the above approach? I feel this approach is easier to code and understand and also matches the time complexity of the trie and heap approach to get k largest elements
EDIT: As stated in one of the answers , incrementing count variable after myobj has been inserted wouldn't re-sort the treeset/treemap. So ,after incrementing the count , I will additionally need to remove and reinsert the object in the treeset/treemap

Once you enter an object into the TreeSet, if the properties used in the comparison of the compareTo method changes, the TreeSet (or the underlying TreeMap) does not reorder the elements. Hence, this approach does not work as you expect.
Here's a simple example to demonstrate it
public static class MyObj implements Comparable<MyObj> {
String value;
int count;
MyObj(String v, int c) {
this.value = v;
this.count = c;
}
public void incrementCount(){
count++;
}
#Override
public int compareTo(MyObj o) {
return Integer.compare(this.count, o.count); //This does the reverse. Orders by freqency
}
}
public static void main(String[] args) {
Set<MyObj> set = new TreeSet<>();
MyObj o1 = new MyObj("a", 1);
MyObj o2 = new MyObj("b", 4);
MyObj o3 = new MyObj("c", 2);
set.add(o1);
set.add(o2);
set.add(o3);
System.out.println(set);
//The above prints [a-1, c-2, b-4]
//Increment the count of c 4 times
o3.incrementCount();
o3.incrementCount();
o3.incrementCount();
o3.incrementCount();
System.out.println(set);
//The above prints [a-1, c-6, b-4]
As we can see the object corresponding to c-6 does not get pushed to the last.
//Insert a new object
set.add(new MyObj("d", 3));
System.out.println(set);
//this prints [a-1, d-3, c-6, b-4]
}
EDIT:
Caveats/Problems:
Using count when comparing two words would remove one word if both words have the same frequency. So, you need to compare the actual words if their frequencies are same.
It would work if we remove and reinsert the object with the updated frequency. But for that, you need to get that object(MyObj instance for a specified value to know the frequency so far) from the TreeSet. A Set does not have a get method. Its contains method just delegates to the underlying TreeMap's containsKey method which identifies the object by using the compareTo logic (and not equals). The compareTo function also takes into account the frequency of the word, so we cannot identify the word in the set to remove it (unless we iterate the whole set on each add)

A TreeMap should work if you remove and insert the object, with an integer key as a frequency and a list of MyObj as a value, the keys are sorted by frequency. An update of the above code demonstrate it:
public class MyObj {
String value;
int count;
MyObj(String v, int c) {
this.value = v;
this.count = c;
}
public int getCount() {
return count;
}
public void incrementCount() {
count++;
}
#Override
public String toString() {
return value + " " + count;
}
public static void put(Map<Integer, List<MyObj>> map, MyObj value) {
List<MyObj> myObjs = map.get(value.getCount());
if (myObjs == null) {
myObjs = new ArrayList<>();
map.put(value.getCount(),myObjs);
}
myObjs.add(value);
}
public static void main(String[] args) {
TreeMap<Integer, List<MyObj>> set = new TreeMap<>();
MyObj o1 = new MyObj("a", 1);
MyObj o2 = new MyObj("b", 4);
MyObj o3 = new MyObj("c", 2);
MyObj o4 = new MyObj("f", 4);
put(set,o1);
put(set,o2);
put(set,o3);
System.out.println(set);
put(set,o4);
System.out.println(set);
}
}

Creating a smart data structure in Java

So I'm trying to create a smart data structure based off AVL tree and Hash Table.
I'm making sure I need to check first which implementation the data type will have depending on the size the list given to it.
For example, if I have a list n of size 1000, it'll be implemented using a Hash table. For anything more than 1000, using an AVL tree.
Code for this:
public class SmartULS<K,V> {
protected TreeMap<K,V> tree = new TreeMap<>();
protected AbstractHashMap<K,V> hashMap = new AbstractHashMap<K,V>();
public void setSmartThresholdULS(size){
int threshold = 1000;
if (size >= threshold) {
map = new AbtractMap<K,V>();
}
else
map = new TreeMap<K,V>();
}
}
Now after this, I should be writing the standard methods such as
get(SmartULS, Key), add(SmartULS, Key, Value), remove(SmartULS,Key), nextKey(Key), previousKey(Key), etc.
I'm really lost as to how to start this? I've thought about creating these methods like this(written in pseudo):
Algorithm add(SmartULS, Key, Value):
i<- 0
If SmartULS instanceof AbstractHashMap then
For i to SmartULS.size do
If Key equals to SmartULS[i] then
SmartULS.get(Key).setValue(Value)
Else
SmartULS.add(Key, Value)
Else if SmartULS instanceof TreeMap then
Entry newAdd equals new MapEntry(Key, Value)
Position<Entry> p = treeSearch(root( ), Key)

You're on the correct track, this is how I understood your question and implemented it:
public class SmartULS<K, V> {
Map<K,V> map;
public static final int THRESHOLD = 1000;
public SmartULS(int size) {
if(size < THRESHOLD) {
map = new HashMap();
} else {
map = new TreeMap();
}
}
public V get(K key) {
return map.get(key);
}
public V put(K key, V value) {
return map.put(key, value);
}
public V remove(K key) {
return map.remove(key);
}
}
Based on the initial size given, the constructor decides if to initialize a hash table or a tree. I also added a the get, put and remove functions and used the Map's interface functions.
I didn't understand what the nextKey and previousKey functions are suppose to do or return, so couldn't help there.
A way of using the class would be as follows:
public static void main(String[] args) {
SmartULS<String, String> smartULS = new SmartULS(952);
smartULS.put("firstKey", "firstValue");
smartULS.put("secondKey", "secondsValue");
String value = smartULS.get("firstKey");
smartULS.remove("secondKey");
}
Hope this helps:)

Finding most common key in Reducer, Error: java.lang.ArrayIndexOutOfBoundsException: 1

I need to find the most common key emitted by Mapper in the Reducer. My reducer works fine in this way:
public static class MyReducer extends Reducer<NullWritable, Text, NullWritable, Text> {
private Text result = new Text();
private TreeMap<Double, Text> k_closest_points= new TreeMap<Double, Text>();
public void reduce(NullWritable key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
Configuration conf = context.getConfiguration();
int K = Integer.parseInt(conf.get("K"));
for (Text value : values) {
String v[] = value.toString().split("#"); //format of value from mapper: "Key#1.2345"
double distance = Double.parseDouble(v[1]);
k_closest_points.put(distance, new Text(value)); //finds the K smallest distances
if (k_closest_points.size() > K)
k_closest_points.remove(k_closest_points.lastKey());
}
for (Text t : k_closest_points.values()) //it perfectly emits the K smallest distances and keys
context.write(NullWritable.get(), t);
}
}
It finds the K instances with the smallest distances and writes to the output file. But I need to find the most common key in my TreeMap. So I'm trying it like below:
public static class MyReducer extends Reducer<NullWritable, Text, NullWritable, Text> {
private Text result = new Text();
private TreeMap<Double, Text> k_closest_points = new TreeMap<Double, Text>();
public void reduce(NullWritable key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
Configuration conf = context.getConfiguration();
int K = Integer.parseInt(conf.get("K"));
for (Text value : values) {
String v[] = value.toString().split("#");
double distance = Double.parseDouble(v[1]);
k_closest_points.put(distance, new Text(value));
if (k_closest_points.size() > K)
k_closest_points.remove(k_closest_points.lastKey());
}
TreeMap<String, Integer> class_counts = new TreeMap<String, Integer>();
for (Text value : k_closest_points.values()) {
String[] tmp = value.toString().split("#");
if (class_counts.containsKey(tmp[0]))
class_counts.put(tmp[0], class_counts.get(tmp[0] + 1));
else
class_counts.put(tmp[0], 1);
}
context.write(NullWritable.get(), new Text(class_counts.lastKey()));
}
}
Then I get this error:
Error: java.lang.ArrayIndexOutOfBoundsException: 1
at KNN$MyReducer.reduce(KNN.java:108)
at KNN$MyReducer.reduce(KNN.java:98)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
Can you please help me to fix this?

A few things... first, your problem is here:
double distance = Double.parseDouble(v[1]);
You're splitting on "#" and it may not be in the string. If it's not, it will throw the OutOfBoundsException. I would add a clause like:
if(v.length < 2)
continue;
Second (and this shouldn't even compile unless I'm crazy), tmp is a String[], and yet here you're actually just concatenating '1' to it in the put operation (it's a parenthesis issue):
class_counts.put(tmp[0], class_counts.get(tmp[0] + 1));
It should be:
class_counts.put(tmp[0], class_counts.get(tmp[0]) + 1);
It's also expensive to look the key up twice in a potentially large Map. Here's how I'd re-write your reducer based on what you've given us (this is totally untested):
public static class MyReducer extends Reducer<NullWritable, Text, NullWritable, Text> {
private Text result = new Text();
private TreeMap<Double, Text> k_closest_points = new TreeMap<Double, Text>();
public void reduce(NullWritable key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
Configuration conf = context.getConfiguration();
int K = Integer.parseInt(conf.get("K"));
for (Text value : values) {
String v[] = value.toString().split("#");
if(v.length < 2)
continue; // consider adding an enum counter
double distance = Double.parseDouble(v[1]);
k_closest_points.put(distance, new Text(v[0])); // you've already split once, why do it again later?
if (k_closest_points.size() > K)
k_closest_points.remove(k_closest_points.lastKey());
}
// exit early if nothing found
if(k_closest_points.isEmpty())
return;
TreeMap<String, Integer> class_counts = new TreeMap<String, Integer>();
for (Text value : k_closest_points.values()) {
String tmp = value.toString();
Integer current_count = class_counts.get(tmp);
if (null != current_count) // avoid second lookup
class_counts.put(tmp, current_count + 1);
else
class_counts.put(tmp, 1);
}
context.write(NullWritable.get(), new Text(class_counts.lastKey()));
}
}
Next, and more semantically, you're performing a KNN operation using a TreeMap as your datastructure of choice. While this makes sense in that it internally stores keys in comparative order, it doesn't make sense to use a Map for an operation that will almost undoubtedly be required to break ties. Here's why:
int k = 2;
TreeMap<Double, Text> map = new TreeMap<>();
map.put(1.0, new Text("close"));
map.put(1.0, new Text("equally close"));
map.put(1500.0, new Text("super far"));
// ... your popping logic...
Which are the two closest points you've retained? "equally close" and "super far". This is due to the fact that you can't have two instance of the same key. Thus, your algorithm is incapable of breaking ties. There are a few things you could do to fix that:
First, if you're set on performing this operation in the Reducer and you know your incoming data will not cause an OutOfMemoryError, consider using a different sorted structure, like a TreeSet and build a custom Comparable object that it will sort:
static class KNNEntry implements Comparable<KNNEntry> {
final Text text;
final Double dist;
KNNEntry(Text text, Double dist) {
this.text = text;
this.dist = dist;
}
#Override
public int compareTo(KNNEntry other) {
int comp = this.dist.compareTo(other.dist);
if(0 == comp)
return this.text.compareTo(other.text);
return comp;
}
}
And then instead of your TreeMap, use a TreeSet<KNNEntry>, which will internally sort itself based on the Comparator logic we just built above. Then after you've gone through all the keys, just iterate through the first k, retaining them in order. This has a drawback, though: if your data is truly big, you can overflow the heapspace by loading all of the values from the reducer into memory.
Second option: make the KNNEntry we built above implement WritableComparable, and emit that from your Mapper, then use secondary sorting to handle the sorting of your entries. This gets a bit more hairy, as you'd have to use lots of mappers and then only one reducer to capture the first k. If your data is small enough, try the first option to allow for tie breaking.
But, back to your original question, you're getting an OutOfBoundsException because the index you're trying to access does not exist, i.e., there is no "#" in the input String.

ArrayList as key in HashMap

Would it be possible to add an ArrayList as the key of HashMap. I would like to keep the frequency count of bigrams. The bigram is the key and the value is its frequency.
For each of the bigrams like "he is", I create an ArrayList for it and insert it into the HashMap. But I am not getting the correct output.
public HashMap<ArrayList<String>, Integer> getBigramMap(String word1, String word2) {
HashMap<ArrayList<String>, Integer> hm = new HashMap<ArrayList<String>, Integer>();
ArrayList<String> arrList1 = new ArrayList<String>();
arrList1 = getBigram(word1, word2);
if (hm.get(arrList1) != null) {
hm.put(arrList1, hm.get(arrList1) + 1);
} else {
hm.put(arrList1, 1);
}
System.out.println(hm.get(arrList1));
return hm;
}
public ArrayList<String> getBigram(String word1, String word2) {
ArrayList<String> arrList2 = new ArrayList<String>();
arrList2.add(word1);
arrList2.add(word2);
return arrList2;
}

Yes you can have ArrayLists as a keys in a hash map, but it is a very bad idea since they are mutable.
If you change the ArrayList in any way (or any of its elements), the mapping will basically be lost, since the key won't have the same hashCode as it had when it was inserted.
The rule of thumb is to use only immutable data types as keys in a hash map. As suggested by Alex Stybaev, you probably want to create a Bigram class like this:
final class Bigram {
private final String word1, word2;
public Bigram(String word1, String word2) {
this.word1 = word1;
this.word2 = word2;
}
public String getWord1() {
return word1;
}
public String getWord2() {
return word2;
}
#Override
public int hashCode() {
return word1.hashCode() ^ word2.hashCode();
}
#Override
public boolean equals(Object obj) {
return (obj instanceof Bigram) && ((Bigram) obj).word1.equals(word1)
&& ((Bigram) obj).word2.equals(word2);
}
}

Why can't you use something like this:
class Bigram{
private String firstItem;
private String secondItem;
<getters/setters>
#Override
public int hashCode(){
...
}
#Override
public boolean equals(){
...
}
}
instead of using the dynamic collection for limited number of items (two).

From the documentation:
Note: great care must be exercised if mutable objects are used as map
keys. The behavior of a map is not specified if the value of an object is
changed in a manner that affects equals comparisons while the
object is a key in the map. A special case of this prohibition is that it
is not permissible for a map to contain itself as a key. While it is
permissible for a map to contain itself as a value, extreme caution is
advised: the equals and hashCode methods are no longer
well defined on such a map.
You have to take care when you are using mutable objects as keys for the sake of hashCode and equals.
The bottom line is that it is better to use immutable objects as keys.

Try this ,this will work.
public Map<List, Integer> getBigramMap (String word1,String word2){
Map<List,Integer> hm = new HashMap<List, Integer>();
List<String> arrList1 = new ArrayList<String>();
arrList1 = getBigram(word1, word2);
if(hm.get(arrList1) !=null){
hm.put(arrList1, hm.get(arrList1)+1);
}
else {
hm.put(arrList1, 1);
}
System.out.println(hm.get(arrList1));
return hm;
}

I've come up with this solution. It is obviously not usable in all cases, for example over stepping the hashcodes int capacity, or list.clone() complications(if the input list gets changed, key stays the same as intended, but when the items of List are mutable, cloned list has the same reference to its items, which would result in changing the key itself).
import java.util.ArrayList;
public class ListKey<T> {
private ArrayList<T> list;
public ListKey(ArrayList<T> list) {
this.list = (ArrayList<T>) list.clone();
}
#Override
public int hashCode() {
final int prime = 31;
int result = 1;
for (int i = 0; i < this.list.size(); i++) {
T item = this.list.get(i);
result = prime * result + ((item == null) ? 0 : item.hashCode());
}
return result;
}
#Override
public boolean equals(Object obj) {
return this.list.equals(obj);
}
}
---------
public static void main(String[] args) {
ArrayList<Float> createFloatList = createFloatList();
ArrayList<Float> createFloatList2 = createFloatList();
Hashtable<ListKey<Float>, String> table = new Hashtable<>();
table.put(new ListKey(createFloatList2), "IT WORKS!");
System.out.println(table.get(createFloatList2));
createFloatList2.add(1f);
System.out.println(table.get(createFloatList2));
createFloatList2.remove(3);
System.out.println(table.get(createFloatList2));
}
public static ArrayList<Float> createFloatList() {
ArrayList<Float> floatee = new ArrayList<>();
floatee.add(34.234f);
floatee.add(new Float(33));
floatee.add(null);
return floatee;
}
Output:
IT WORKS!
null
IT WORKS!

Sure it possible. I suppose the issue in your put. Try obtain key for bigram, increment it, remove entry with this bigram and insert updated value

Unlike Array, List can be used as the key of a HashMap, but it is not a good idea, since we should always try to use an immutable object as the key.
.toString() method getting the String represtenation is a good key choice in many cases, since String is an immuteable object and can prefectly stands for the array or list.

Please check below my code in order to understand if key is ArrayList in Map and how JVM will do it for inputs:
here i write hashCode and equals method for TesthashCodeEquals class.
package com.msq;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
class TesthashCodeEquals {
private int a;
private int b;
public TesthashCodeEquals() {
// TODO Auto-generated constructor stub
}
public TesthashCodeEquals(int a, int b) {
super();
this.a = a;
this.b = b;
}
public int getA() {
return a;
}
public void setA(int a) {
this.a = a;
}
public int getB() {
return b;
}
public void setB(int b) {
this.b = b;
}
public int hashCode() {
return this.a + this.b;
}
public boolean equals(Object o) {
if (o instanceof TesthashCodeEquals && o != null) {
TesthashCodeEquals c = (TesthashCodeEquals) o;
return ((this.a == c.a) && (this.b == c.b));
} else
return false;
}
}
public class HasCodeEquals {
public static void main(String[] args) {
Map<List<TesthashCodeEquals>, String> m = new HashMap<>();
List<TesthashCodeEquals> list1=new ArrayList<>();
list1.add(new TesthashCodeEquals(1, 2));
list1.add(new TesthashCodeEquals(3, 4));
List<TesthashCodeEquals> list2=new ArrayList<>();
list2.add(new TesthashCodeEquals(10, 20));
list2.add(new TesthashCodeEquals(30, 40));
List<TesthashCodeEquals> list3=new ArrayList<>();
list3.add(new TesthashCodeEquals(1, 2));
list3.add(new TesthashCodeEquals(3, 4));
m.put(list1, "List1");
m.put(list2, "List2");
m.put(list3, "List3");
for(Map.Entry<List<TesthashCodeEquals>,String> entry:m.entrySet()){
for(TesthashCodeEquals t:entry.getKey()){
System.out.print("value of a: "+t.getA()+", value of b: "+t.getB()+", map value is:"+entry.getValue() );
System.out.println();
}
System.out.println("######################");
}
}
}
.
output:
value of a: 10, value of b: 20, map value is:List2
value of a: 30, value of b: 40, map value is:List2
######################
value of a: 1, value of b: 2, map value is:List3
value of a: 3, value of b: 4, map value is:List3
######################
so this will check the number of objects in List and the values of valriabe in object. if number of objects are same and the values of instance variables is also same then it will consider duplicate key and override the key.
now if i change only the value of object on list3
list3.add(new TesthashCodeEquals(2, 2));
then it will print:
output
value of a: 2, value of b: 2, map value is:List3
value of a: 3, value of b: 4, map value is:List3
######################
value of a: 10, value of b: 20, map value is:List2
value of a: 30, value of b: 40, map value is:List2
######################
value of a: 1, value of b: 2, map value is:List1
value of a: 3, value of b: 4, map value is:List1
######################
so that It always check the number of objects in List and the value of instance variable of object.
thanks

ArrayList.equals() is inherited from java.lang.Object - therefore equals() on ArrayList is independent of the content of the list.
If you want to use an ArrayList as a map key, you will need to override equals() and hashcode() in order to make two arraylists with the same content in the same order return true on a call to equals() and return the same hashcode on a call to hashcode().
Is there any particular reason you have to use an ArrayList as opposed to say a simple String as the key?
edit: Ignore me, as Joachim Sauer pointed out below, I am so wrong it's not even funny.