Develop Reference

HashMap vs array performance difference in following approach

To solve Dynamic programming problem I used two approaches to store table entries, one using multi dimension array ex:tb[m][n][p][q] and other using hashmap and using indexes of 1st approach to make string to be used as key as in "m,n,p,q". But on one input first approach completes in 2 minutes while other takes more than 3 minutes.
If access time of both hashmap and array is asymptotically equal than why so big difference in performance ?

Like mentioned here:
HashMap uses an array underneath so it can never be faster than using
an array correctly.
You are right, array's and HashMap's access time is in O(1) but this just says it is independent on input size or the current size of your collection. But it doesn't say anything about the actual work which has to be done for each action.
To access an entry of an array you have to calculate the memory address of your entry. This is easy as array's memory address + (index * size of entity).
To access an entry of a HashMap, you first have to hash the given key (which needs many cpu cycles), then access the entry of the HashMap's array using the hash which holds a list (depends on implementation details of the HashMap), and last you have to linear search the list for the correct entry (those lists are very short most of the time, so it is treated as O(1)).
So you see it is more like O(10) for arrays and O(5000) hash maps. Or more precise T(Array access) for arrays and T(hashing) + T(Array access) + T(linear search) for HashMaps with T(X) as actual time of action x.

SparseArray vs HashMap

I can think of several reasons why HashMaps with integer keys are much better than SparseArrays:
The Android documentation for a SparseArray says "It is generally slower than a traditional HashMap".
If you write code using HashMaps rather than SparseArrays your code will work with other implementations of Map and you will be able to use all of the Java APIs designed for Maps.
If you write code using HashMaps rather than SparseArrays your code will work in non-android projects.
Map overrides equals() and hashCode() whereas SparseArray doesn't.
Yet whenever I try to use a HashMap with integer keys in an Android project, IntelliJ tells me I should use a SparseArray instead. I find this really difficult to understand. Does anyone know any compelling reasons for using SparseArrays?

SparseArray can be used to replace HashMap when the key is a primitive type.
There are some variants for different key/value types, even though not all of them are publicly available.
Benefits are:
Allocation-free
No boxing
Drawbacks:
Generally slower, not indicated for large collections
They won't work in a non-Android project
HashMap can be replaced by the following:
SparseArray <Integer, Object>
SparseBooleanArray <Integer, Boolean>
SparseIntArray <Integer, Integer>
SparseLongArray <Integer, Long>
LongSparseArray <Long, Object>
LongSparseLongArray <Long, Long> //this is not a public class
//but can be copied from Android source code
In terms of memory, here is an example of SparseIntArray vs HashMap<Integer, Integer> for 1000 elements:
SparseIntArray:
class SparseIntArray {
int[] keys;
int[] values;
int size;
}
Class = 12 + 3 * 4 = 24 bytes
Array = 20 + 1000 * 4 = 4024 bytes
Total = 8,072 bytes
HashMap:
class HashMap<K, V> {
Entry<K, V>[] table;
Entry<K, V> forNull;
int size;
int modCount;
int threshold;
Set<K> keys
Set<Entry<K, V>> entries;
Collection<V> values;
}
Class = 12 + 8 * 4 = 48 bytes
Entry = 32 + 16 + 16 = 64 bytes
Array = 20 + 1000 * 64 = 64024 bytes
Total = 64,136 bytes
Source: Android Memories by Romain Guy from slide 90.
The numbers above are the amount of memory (in bytes) allocated on heap by JVM.
They may vary depending on the specific JVM used.
The java.lang.instrument package contains some helpful methods for advanced operations like checking the size of an object with getObjectSize(Object objectToSize).
Extra info is available from the official Oracle documentation.
Class = 12 bytes + (n instance variables) * 4 bytes
Array = 20 bytes + (n elements) * (element size)
Entry = 32 bytes + (1st element size) + (2nd element size)

I came here just wanting an example of how to use SparseArray. This is a supplemental answer for that.
Create a SparseArray
SparseArray<String> sparseArray = new SparseArray<>();
A SparseArray maps integers to some Object, so you could replace String in the example above with any other Object. If you are mapping integers to integers then use SparseIntArray.
Add or update items
Use put (or append) to add elements to the array.
sparseArray.put(10, "horse");
sparseArray.put(3, "cow");
sparseArray.put(1, "camel");
sparseArray.put(99, "sheep");
sparseArray.put(30, "goat");
sparseArray.put(17, "pig");
Note that the int keys do not need to be in order. This can also be used to change the value at a particular int key.
Remove items
Use remove (or delete) to remove elements from the array.
sparseArray.remove(17); // "pig" removed
The int parameter is the integer key.
Lookup values for an int key
Use get to get the value for some integer key.
String someAnimal = sparseArray.get(99); // "sheep"
String anotherAnimal = sparseArray.get(200); // null
You can use get(int key, E valueIfKeyNotFound) if you want to avoid getting null for missing keys.
Iterate over the items
You can use keyAt and valueAt some index to loop through the collection because the SparseArray maintains a separate index distinct from the int keys.
int size = sparseArray.size();
for (int i = 0; i < size; i++) {
int key = sparseArray.keyAt(i);
String value = sparseArray.valueAt(i);
Log.i("TAG", "key: " + key + " value: " + value);
}
// key: 1 value: camel
// key: 3 value: cow
// key: 10 value: horse
// key: 30 value: goat
// key: 99 value: sheep
Note that the keys are ordered in ascending value, not in the order that they were added.

Yet whenever I try to use a HashMap with integer keys in an android
project, intelliJ tells me I should use a SparseArray instead.
It is only a warning from this documentation of it sparse array:
It is intended to be more memory efficient than using a HashMap to
map Integers to Objects
The SparseArray is made to be memory efficient than using the regular HashMap, that is does not allow multiple gaps within the array not like HashMap. There is nothing to worry about it you can use the traditional HashMap if you desire not worrying about the memory allocation to the device.

After some googling I try to add some information to the already posted anwers:
Isaac Taylor made a performance comparision for SparseArrays and Hashmaps. He states that
the Hashmap and the SparseArray are very similar for data structure
sizes under 1,000
and
when the size has been increased to the 10,000 mark [...] the Hashmap
has greater performance with adding objects, while the SparseArray has
greater performance when retrieving objects. [...] At a size of 100,000 [...] the Hashmap loses performance very quickly
An comparision on Edgblog shows that a SparseArray need much less memory than a HashMap because of the smaller key (int vs Integer) and the fact that
a HashMap.Entry instance must keep track of the references for the
key, the value and the next entry. Plus it also needs to store the
hash of the entry as an int.
As a conclusion I would say that the difference could matter if you are going to store a lot of data in your Map. Otherwise, just ignore the warning.

A sparse array in Java is a data structure which maps keys to values. Same idea as a Map, but different implementation:
A Map is represented internally as an array of lists, where each element in these lists is a key,value pair. Both the key and value are object instances.
A sparse array is simply made of two arrays: an arrays of (primitives) keys and an array of (objects) values. There can be gaps in these arrays indices, hence the term “sparse” array.
The main interest of the SparseArray is that it saves memory by using primitives instead of objects as the key.

The android documentation for a SparseArray says "It is generally
slower than a traditional HashMap".
Yes,it's right. But when you have only 10 or 20 items , the performance difference should be insignificant.
If you write code using HashMaps rather than SparseArrays your code
will work with other implementations of Map and you will be able to
use all of the java APIs designed for Maps
I think most often we only use HashMap to search a value associated with a key while SparseArray is really good at this.
If you write code using HashMaps rather than SparseArrays your code
will work in non-android projects.
The source code of SparseArray is fairly simple and easy to understand so that you only pay little effort moving it to other platforms(through a simple COPY&Paste).
Map overrides equals() and hashCode() whereas SparseArray doesn't
All I can say is, (to most developers)who care?
Another important aspect of SparseArray is that it only uses an array to store all elements while HashMap uses Entry, so SparseArray costs significant less memory than a HashMap, see this

It's unfortunate that the compiler issues a warning. I guess HashMap has been way overused for storing items.
SparseArrays have their place. Given they use a binary search algorithm to find a value in an array you have to consider what you are doing. Binary search is O(log n) while hash lookup is O(1). This doesn't necessarily mean that binary search is slower for a given set of data. However, as the number of entries grows, the power of the hash table takes over. Hence the comments where low number of entries can equal and possibly be better than using a HashMap.
A HashMap is only as good as the hash and also can be impacted by load factor (I think in later versions they ignore the load factor so it can be better optimized). They also added a secondary hash to make sure the hash is good. Also the reason SparseArray works really well for relatively few entries (<100).
I would suggest that if you need a hash table and want better memory usage for primitive integer (no auto boxing), etc., try out trove. (http://trove.starlight-systems.com - LGPL license). (No affiliation with trove, just like their library)
With the simplified multi-dex building we have you don't even need to repackage trove for what you need. (trove has a lot of classes)

Looking for memory efficient design

I'm running some experiments over a large dataset and would like to optimize a particular part. Currently, I have 5-6 Models each of which stores a mapping from Topics to List of Strings. The set of Topics is large and the same between each Model, so there must be a better way. Ultimately the query I need to perform is: what is the String in position x of the List for some Model-Topic combination.
One of the problems with using the mapping method is that if there are say 500k-5M topics, each has a list of 20 strings. Then my Map<Model, Map<Topic, List<String>>> is going to be massive.

Have you tried SortedSet / Maps? Sounds like you need to optimize your search, sorted collections (like TreeMap) should be log(n) while regular list is O(1). Of course, this kind of thing is something at which databases excel...

Not clear where/how you want to achieve "memory efficiency". First one needs to look at the particulars of your detailed data to see how much storage that consumes, then examine various ways of organizing it and analyze their efficiency in terms of % overhead vs your "real" data.
A brief glance shows that a HashMap, when you consider the associated tables, has about 80 bytes of overhead per entry. An ArrayList looks to average out around 10-12. Without looking, I would guess that a TreeMap would be more than a HashMap -- maybe 100.
Generally speaking, links within your own objects will be "cheaper", both in storage and speed to access, than links using these aggregating objects. But the aggregating objects are convenient to use, and have been "optimized" to a degree.
(But looking at your update, you probably should be looking at a DB application, rather than holding everything in heap.)

You could use Topic and Model to construct a composite key in a single Map, e.g.
map.put(topic1_id + model1_id, list1_1);
map.put(topic1_id + model2_id, list1_2);
...
map.get(topic_id + model_id)
where the IDs are Strings (or a similar scheme could be used with numeric identifiers).
A similar approach is to assign each topic and model a unique number, then store the lists of strings in arrays, so looking up the list for a given combination is a matter of looking up two indexes, then accessing a given location in a 2D array. (however, this is easier when you know the number of topics and models in advance of constructing the data structure)
For memory efficiency, also consider the small details. In general, you want to minimise the number of Objects - each Object carries an overhead. ArrayLists can have a lot of wasted space as they grow dynamically, doubling in size when they exceed their current capacity. If you can pre-size them to the required capacity (or use an array instead) then you can save a lot of memory. The same applies when using large numbers of small HashMaps.

One possible data structure is a hierarchy of maps, leading to an array of Strings. E.g.:
HashMap<Model, HashMap<Topic, String[]>> map;
A query function would then look like:
public String query(Model model, Topic topic, int x) {
HashMap<Topic, String[]> childMap = map.get(model);
if (childMap == null) {
return null;
}
String[] list = childMap.get(topic);
if (list == null) {
return null;
}
return list[x];
}
Presuming your Model and Topic structures implement hashCode() and equals() reasonably, the query performance should be quite good.
One potential weakness: I'm assuming you need to index a large number of Model/Topic combinations, and related lists of Strings (if not, you presumably wouldn't be asking about optimization). My guess is that the child String[] arrays will consume a large amount of memory. Each array is a Java object (about 20 bytes) + a pointer at each array location.
2 suggestions there:
1) If many Model/Topic combinations share the same set of Strings, you could gain quite a lot by sharing those String[] instances.
2) If you're using a 64-bit VM, be sure to use compressed ordinary object pointers (-XX:+UseCompressedOops). That will at least keep most of the pointers to 4 bytes instead of 8. Compressed OOPs is the default since 1.6.0_23, so a relatively recent VM will save you some memory here.

One other possibility not mentioned is store the strings using String[][][] and models and topics in a List such as ArrayList and then at query time:
public String query(Model model, Topic topic, int x) {
return strings[models.indexOf(model)][topics.indexOf(topic)][x];
}
It could be further improved for speed if the topics and models were sorted, then binary search rather than indexOf could be used.

Multimap Space Issue: Guava

In my Java code, I am using Guava's Multimap (com.google.common.collect.Multimap) by using this:
Multimap<Integer, Integer> Index = HashMultimap.create()
Here, Multimap key is some portion of a URL and value is another portion of the URL (converted into an integer). Now, I assign my JVM 2560 Mb (2.5 GB) heap space (by using Xmx and Xms). However, it can only store 9 millions of such (key,value) pairs of integers (approx 10 million). But, theoretically (according to memory occupied by int) it should store more.
Can anybody help me,
Why is Multimap using lots of memory? I checked my code and without inserting pairs into the Multimap, it only uses 1/2 MB of memory.
2.
Is there another way or home-baked solution to solve this memory issue? Means, Is there any way to reduce those object overheads as I want to store only int-int? In any other language ? Or any other solution (home-baked preferred) to solve issue I faced, means DB based or something like that solution.

There's a huge amount of overhead associated with Multimap. At a minimum:
Each key and value is an Integer object, which (at a minimum) doubles the storage requirements of each int value.
Each unique key value in the HashMultimap is associated with a Collection of values (according to the source, the Collection is a Hashset).
Each Hashset is created with default space for 8 values.
So each key/value pair requires (at a minimum) perhaps an order of magnitude more space than you might expect for two int values. (Somewhat less when multiple values are stored under a single key.) I would expect 10 million key/value pairs to take perhaps 400MB.
Although you have 2.5GB of heap space, I wouldn't be all that surprised if that's not enough. The above estimate is, I think, on the low side. Plus, it only accounts for how much is needed to store the map once it is built. As the map grows, the table needs to be reallocated and rehashed, which temporarily at least doubles the amount of space used. Finally, all this assumes that int values and object references require 4 bytes. If the JVM is using 64-bit addressing, the byte count probably doubles.

Probably the simplest way to minimize the memory overhead would be to potentially mix Trove's primitive collection implementations (to avoid memory overhead of boxing) and Guava's Multimap, something like
SetMultimap<Integer, Integer> multimap = Multimaps.newSetMultimap(
TDecorators.wrap(TIntObjectHashMap<Collection<Integer>>()),
new Supplier<Set<Integer>>() {
public Set<Integer> get() {
return TDecorators.wrap(new TIntHashSet());
}
});
That still has the overhead of boxing and unboxing on queries, but the memory it consumes just sitting there would be significantly reduced.

It sounds like you need a sparse boolean matrix. Sparse matrices / arrays in Java should provide pointers to library code. Then instead of putting (i, j) into the multimap, just put a 1 into the matrix at [i][j].

You could use probably an ArrayListMultimap, which requires less memory than a HashMultimap, since ArrayLists are smaller than HashSets. Or, you could modify Louis's Trove solution, replacing the Set with a List, to reduce memory usage further.
Some applications depend on the fact that HashMultimap satisfies the SetMultimap interface, but most don't.

Time complexity of set in Java

Can someone tell me the time complexity of the below code?
a is an array of int.
Set<Integer> set = new HashSet<Integer>();
for (int i = 0; i < a.length; i++) {
if (set.contains(arr[i])) {
System.out.println("Hello");
}
set.add(arr[i]);
}
I think that it is O(n), but I'm not sure since it is using Set and this contains methods as well. It is also calling the add method of set.
Can anyone confirm and explain what the time complexity of the entire above code is? Also, how much space would it take?

i believe its O(n) because you loop over the array, and contains and add should be constant time because its a hash based set. If it were not hash based and required iteration over the entire set to do lookups, the upper bound would be n^2.
Integers are immutable, so the space complexity would be 2n, which I believe simplifies to just n, since constants don't matter.
If you had objects in the array and set, then you would have 2n references and n objects, so you are at 3n, which is still linear (times a constant) space constraints.
EDIT-- yep "This class offers constant time performance for the basic operations (add, remove, contains and size), assuming the hash function disperses the elements properly among the buckets."
see here.

Understanding HashSet is the key of question
According to HashSet in Javadoc,
This class implements the Set interface, backed by a hash table
(actually a HashMap instance)...This class offers constant time
performance for the basic operations (add, remove, contains and size)
A more complete explain about HashSet https://www.geeksforgeeks.org/hashset-contains-method-in-java/?ref=rp
So the HashSet insert and contains are O(1). ( As HashSet is based on HashMap and Its memory complexity is O(n))
The rest is simple, the main array you are looping is order of O(n) , so the total order of function will be O(n).