I'm trying to store a large amount of longs into an array, but I get an OOM error with my array and ArrayList.
ArrayList<Long> A = new ArrayList<Long>();
Long[] B = new Long[100000000];
for(int i = 0; i < 100000000; i++) A.add(i);
for(int i = 0; i < 100000000; i++) B[i] = (long) i;
//Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
//at java.lang.Integer.valueOf(Unknown Source)
Is there any other simple data structure I can use to store such a large amount of longs or integers?
If you use the -Xmx flag appropriately when running java (see here for more info), you can increase your Heap size. This will allow you to use more memory if needed, in a controlled way. To my knowledge, there is not a way of "asking" for more heap memory from within a program itself (similarly to running the sbrk() or mmap() syscalls in C)
As the answer I linked to says:
For example, starting a JVM like so will start it with 256MB of
memory, and will allow the process to use up to 2048MB of memory:
java -Xmx2048m -Xms256m
Also, you can use "k", "m", or "g" for Kilobytes, Megabytes and Gigabytes respectively.
You cannot exceed 1GB (Heap size, that is), however, unless you are using the 64-bit JVM.
If you do the math with your particular use-case, assuming 64-bit longs * 100000000 costs about 800MB of space.
Does your program require you to store longs? If you were to use Integers instead of longs then you could store much more, either way, what requires you to store so many longs in an array?
Other solutions:
You could give your program more heap space when you start it with the argument -Xmx2G
or some other length greater than the standard 512M or 1G
You could process a less number of array values, then save the array to hard drive. Then process the rest of the array and store it into the file.(Would require basic knowledge of Java's garbage collection)
Any solution depends upon what you're trying to do. For example, if you simply want to iterate through the values, you can declare an Iterator<Long>:
Iterator<Long> myIterator = new Iterator<Long>() {
private long nextValue = 0;
#Override
public boolean hasNext() {
return nextValue < 100000000;
}
#Override
public Long next() {
return nextValue++;
}
#Override
public void remove() { throw new UnsupportedOperationException(); }
};
Such a solution has O(1) memory usage, because it doesn't require you to store anything more than the next value. But it may not be appropriate to the application.
Related
I have read some answers for this question(Why I can't create an array with large size? and https://bugs.openjdk.java.net/browse/JDK-8029587) and I don't understand the following.
"In the GC code we pass around the size of objects in words as an int." As I know the size of a word in JVM is 4 bytes. According to this, if we pass around the size of long array of large size (for example, MAX_INT - 5) in words as an int, we must get OutOfMemoryException with Requested array size exceeds VM limit because the size is too large for int even without size of header. So why arrays of different types have the same limit on max count of elements?
Only addressing the why arrays of different types have the same limit on max count of elements? part:
Because it doesn't matter to much in practical reality; but allows the code implementing the JVM to be simpler.
When there is only one limit; that is the same for all kinds of arrays; then you can deal all arrays with that code. Instead of having a lot of type-specific code.
And given the fact that the people that need "large" arrays can still create them; and only those that need really really large arrays are impacted; why spent that effort?
The answer is in the jdk sources as far as I can tell (I'm looking at jdk-9); also after writing it I am not sure if it should be a comment instead (and if it answers your question), but it's too long for a comment...
First the error is thrown from hotspot/src/share/vm/oops/arrayKlass.cpp here:
if (length > arrayOopDesc::max_array_length(T_ARRAY)) {
report_java_out_of_memory("Requested array size exceeds VM limit");
....
}
Now, T_ARRAY is actually an enum of type BasicType that looks like this:
public static final BasicType T_ARRAY = new BasicType(tArray);
// tArray is an int with value = 13
That is the first indication that when computing the maximum size, jdk does not care what that array will hold (the T_ARRAY does not specify what types will that array hold).
Now the method that actually validates the maximum array size looks like this:
static int32_t max_array_length(BasicType type) {
assert(type >= 0 && type < T_CONFLICT, "wrong type");
assert(type2aelembytes(type) != 0, "wrong type");
const size_t max_element_words_per_size_t =
align_size_down((SIZE_MAX/HeapWordSize - header_size(type)), MinObjAlignment);
const size_t max_elements_per_size_t =
HeapWordSize * max_element_words_per_size_t / type2aelembytes(type);
if ((size_t)max_jint < max_elements_per_size_t) {
// It should be ok to return max_jint here, but parts of the code
// (CollectedHeap, Klass::oop_oop_iterate(), and more) uses an int for
// passing around the size (in words) of an object. So, we need to avoid
// overflowing an int when we add the header. See CRs 4718400 and 7110613.
return align_size_down(max_jint - header_size(type), MinObjAlignment);
}
return (int32_t)max_elements_per_size_t;
}
I did not dive too much into the code, but it is based on HeapWordSize; which is 8 bytes at least. here is a good reference (I tried to look it up into the code itself, but there are too many references to it).
I am trying to create 2D array in Java as follows:
int[][] adjecancy = new int[96295][96295];
but it is failing with the following error:
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/04/07 11:58:55 - please wait.
JVMDUMP032I JVM requested System dump using 'C:\eclipse\workspaces\TryJavaProj\core.20170407.115855.7840.0001.dmp' in response to an event
JVMDUMP010I System dump written to C:\eclipse\workspaces\TryJavaProj\core.20170407.115855.7840.0001.dmp
JVMDUMP032I JVM requested Heap dump using 'C:\eclipse\workspaces\TryJavaProj\heapdump.20170407.115855.7840.0002.phd' in response to an event
JVMDUMP010I Heap dump written to C:\eclipse\workspaces\TryJavaProj\heapdump.20170407.115855.7840.0002.phd
A way to solve this is by increasing the JVM memory but I am trying to submit the code for an online coding challenge. There it is also failing and I will not be able to change the settings there.
Is there any standard limit or guidance for creating large arrays which one should not exceed?
int[][] adjecancy = new int[96295][96295];
When you do that you are trying to allocate 96525*96525*32 bits which is nearly 37091 MB which is nearly 37 gigs. That is highly impossible to get the memory from a PC for Java alone.
I don't think you need that much data in your hand on initialization of your program. Probably you have to look at ArrayList which gives you dynamic allocation of size and then keep on freeing up at runtime is a key to consider.
There is no limit or restriction to create an array. As long as you have memory, you can use it. But keep in mind that you should not hold a block of memory which makes JVM life hectic.
Array must obviously fit into memory. If it does not, the typical solutions are:
Do you really need int (max value 2,147,483,647)? Maybe byte (max
value 127) or short is good enough? byte is 8 times smaller than int.
Do you have really many identical values in array (like zeros)? Try to use sparse arrays.
for instance:
Map<Integer, Map<Integer, Integer>> map = new HashMap<>();
map.put(27, new HashMap<Integer, Integer>()); // row 27 exists
map.get(27).put(54, 1); // row 27, column 54 has value 1.
They need more memory per value stored, but have basically no limits on the array space (you can use Long rather than Integer as index to make them really huge).
Maybe you just do not know how long the array should be? Try ArrayList, it self-resizes. Use ArrayList of ArrayLists for 2D array.
If nothing else is helpful, use RandomAccessFile to store your overgrown data into the filesystem. 100 Gb or about are not a problem in these times on a good workstation, you just need to compute the required offset in the file. The filesystem is obviously much slower than RAM but with good SSD drive may be bearable.
It is recommended to allocate Maximum Heap Size that can be allocated is 1/4th of the Machine RAM Size.
1 int in Java takes 4 bytes and your array allocation needs approximately 37.09GB of Memory.
In that case even if I assume you are allocating Full Heap to just an Array your machine should be around 148GB RAM. That is huge.
Have a look at below.
Ref: http://docs.oracle.com/javase/8/docs/technotes/guides/vm/gc-ergonomics.html
Hope this helps.
It depends on maximum memory available to your JVM and the content type of the array. For int we have 4 bytes of memory. Now if 1 MB of memory is available on your machine , it can hold maximum of 1024 * 256 integers(1 MB = 1024 * 1024 bytes). Keeping that in mind you can create your 2D array accordingly.
Array that you can create depends upon JVM heap size.
96295*96295*4(bytes per number) = 37,090,908,100 bytes = ~34.54 GBytes. Most JVMs in competitive code judges don't have that much memory. Hence the error.
To get a good idea of what array size you can use for given heap size -
Run this code snippet with different -Xmx settings:
Scanner scanner = new Scanner(System.in);
while(true){
System.out.println("Enter 2-D array of size: ");
size = scanner.nextInt();
int [][]numbers = new int[size][size];
numbers = null;
}
e.g. with -Xmx 512M -> 2-D array of ~10k+ elements.
Generally most of online judges have ~1.5-2GB heap while evaluating submissions.
I wrote this code:
public static void main(String[] args) {
// TODO Auto-generated method stub
HashMap<Long,Long> mappp = new HashMap<Long, Long>(); Long a = (long)55; Long c = (long)12;
for(int u = 1;u<=1303564/2 + 1303564/3;u++){
mappp.put(a, c);
a = a+1;
c = c+1;
}
System.out.println(" " + mappp.size());
}
And it does not finish, beacause the progrm stops with the message in the console:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
I calculated how memory I need to have such HashMAp in memory and in my opinion my computer memory is enough. I have 1024 RAM on my computer.
I use eclipse. Also have set the parameters :
i am starting eclipse from the command line with this:'eclipse -vmargs -Xms512m -Xmx730m'
And second from Run Configurations i have set the tab Arguments with this:'-Xmx730m'
And this still gives java.lang.OutOfMemoryError.
What is the reason for this?
ps. Just to add some strange fact - in the bottom right corner of eclipse is shown the heap memory usage, and it is written 130M of 495M.
Well, when the HashMap mappp increases in size, doesn't this info '130M of 495M' have to change,for example '357M of 495M', 1 second later to be '412M of 495M' and so on in order to reach this 495M?In my case this 130M stays the same, just a little changes, from 130M to 131M or to 132M.
Strange
Java does not allows map of primitive data types. So if you are using Hashmap you will have to pay for boxing/unboxing and overhead of object references.
To avoid the overhead you can write your custom hashmap or use existing implementation from one of those libs.
boxing and unboxing in java
You should not put millions of items in a map. A Long is an object containing an 8 byte long field, plus some object overhead. Then you use two instances per map entry.
Since the key is numeric, you could (if the maximum key value is low enough) use an array as the 'map'.
long[] mappp = new long[4000000]; // takes 4M * 8 = 32M memory
If you need to know whether a value is 'not in the map', use 0 for that value. If 0 needs to be in your map, you can do some tricks like increasing all values by 1 (if the values are always positive).
I have a Java Card smart card and I want to assess the available EEPROM.
To do it, I use the function JCSystem.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT).
As the return statment of this function is a short, without allocating any data, I get the value 0x7FFF. To solve this problem, I create byte arrays this way: new byte[(short) 0x7FFF] to deduce the available persistant memory.
If I create two arrays:
arr1 = new byte[(short) 0x7FFF];
arr2 = new byte[(short) 0x7FFF];
Then it rests 0x1144 bytes of available memory according to JCSystem.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT). So if I sum, it means that there is 32767*2 + 4420 = 69954 bytes available.
But when I change the size of my arrays:
arr1 = new byte[(short) 0x7FFF];
arr2 = new byte[(short) 0x6FFF];
then it rests 0x2244 bytes of available memory. So if I sum, it means that there is 70210 bytes available.
Another example:
With
arr1 = new byte[(short) 0x7FFF];
arr2 = new byte[(short) 0x5FFF];
it rests 0x3344 bytes of available memory. So if I sum it means that there is 70466 bytes available.
Even if it's negligible, why these differences? (70210 differs from 70466).
In the same way, I want to test how many AESKey I can allocate in one applet. So I try to find the available memory as I described before but with AESKey arrays.
With the same card, when I create an AESKey array this way:
arr = new AESKey[(short) 0x03E8];
for (short i = 0x0000; i < 0x03E8; i++) {
arr[i] = (AESKey) KeyBuilder.buildKey(KeyBuilder.TYPE_AES, KeyBuilder.LENGTH_AES_256, false);
}
So I create an array of a thousand 256 bits AESKey. I thought that it would take 32Ko, but the method JCSystem.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT) indicates that there is 0x0022 bytes available. Why this result?
If I test with half keys (e.g 500):
arr = new AESKey[(short) 0x01F4];
for (short i = 0x0000; i < 0x01F4; i++) {
arr[i] = (AESKey) KeyBuilder.buildKey(KeyBuilder.TYPE_AES, KeyBuilder.LENGTH_AES_256, false);
}
the method JCSystem.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT) indicates that there is 0x55EE (21998) bytes available: I definitely don't see the relation with the case when I create 1000 keys if the available EEPROM is about 70Ko like I explained at the beginning...
Could someone describe in detail how the memory is allocated in Java Card to explain the results cited above?
There are a few reasons for this:
there is object allocation overhead;
there may be overhead with regards to aligning data;
there may be overhead with regards to memory fragmentation;
for keys, there may be overhead to keep them secure.
All these issues will reduce the amount of memory available to you. In that regard you should see getAvailableMemory as a rough indication of the maximum amount of memory available.
How much overhead is required depends on the Java Card runtime.
Well, for a short answer:
Java Card isn't to clever storing arrays / it needs additional data. So if you fill up a byte array with x bytes to get under the 0x7FFF treshhold the array will internally need more than x bytes to store the data and therefore you have a difference that you noticed.
If you are working with JCOP cards you can circumvent the problem by using UtilX.getAvailableMemory().
For a little more knowledge read this:
http://ruimtools.com/doc.php?doc=jc_best on the point reducing EEPROM consumption (however some parts are outdated)
Using the -Xmx1G flag to provide a heap of one gigabyte, the following works as expected:
public class Biggy {
public static void main(String[] args) {
int[] array = new int[150 * 1000 * 1000];
}
}
The array should represent around 600 MB.
However, the following throws OutOfMemoryError:
public class Biggy {
public static void main(String[] args) {
int[] array = new int[200 * 1000 * 1000];
}
}
Despite the array should represent around 800 MB and therefore easily fit in memory.
Where's the missing memory gone?
In Java you typically have multiple regions (and sub regions) in the heap. You have a young and tenured region with most collectors. Large arrays are added to the tenured area straight away however based on your maximum memory size, some space will be reserved for the young space. If you allocate memory slowly these regions will resize however a large block like this can simply fail as you have seen.
Given memory is usually relatively cheap (not always the case) I would just increase the maximum to the point where you would want the application fail if it ever used that much.
BTW: If you have a large structure like this you might consider using direct memory.
IntBuffer array = ByteBuffer.allocateDirect(200*1000*1000*4)
.order(ByteOrder.nativeOrder()).asIntBuffer();
int a = array.get(n);
array.put(n, a+1);
Its a bit tedious to write but has one big advantage, it uses almost no heap. (there is less than 1 KB over head)
There is enough memory available but not as a single continuous
block of memory, as needed for an array.
Can you use a different data structure that uses smaller blocks
of memory, or several smaller arrays?
For example, the following code does work with -Xmx1G:
public class Biggy {
public static void main(String[] args) {
int [][]array = new int[200][];
for (int i = 0; i < 200; i++) {
array[i] = new int[1000 * 1000];
System.out.println("i=" + i);
}
}
}
Heap memory is divided between three spaces:
Old Generation
Survivor Space
Eden Space
At start this object will live in the old generation and will remain here for a while.
By default, the virtual machine grows or shrinks the heap at each collection to try to keep the proportion of free space to live objects at each collection within a specific range. This target range is set as a percentage by the parameters -XX:MinHeapFreeRatio= and -XX:MaxHeapFreeRatio=, and the total size is bounded below by -Xms and above by -Xmx.
Default ratio in my jvm is 30/70 so max size of object in old generation is limited (with -Xmx1G) by 700Mb(btw, I'm getting the same exception when running with default jvm parameters).
However you could size generations using jvm options. For example you could
run your class with parameters -Xmx1G -XX:NewRatio=10 and new int[200 * 1000 * 1000]; will succeed.
From what I could say Java wasn't designed to hold large objects in memory. Typical usage of memory in application is graph of bunch of relatively small objects and typically you'll get OutOfMemoryError only if you run out of space in all of spaces.
Below are couple useful (and interesting to read) articles:
Ergonomics in the 5.0 Java[tm] Virtual Machine
Tuning Garbage Collection with the 5.0 Java[tm] Virtual Machine