Develop Reference

Not enough memory (less than 50 m on heap) error message in weka 3.8

I am using weka 3.8. when I apply association rules on my dataset, an error message appears which says that (not enough memory. less than 50m left on heap. please load a smaller dataset or use a larger heap size).

In your RunWeka.ini you can change
javaOpts=%JAVA_OPTS%
to
javaOpts=%JAVA_OPTS% -Xmx2048m
The parameter -Xmx says how much memory your vm is allowed to use.
See https://docs.oracle.com/en/java/javase/16/docs/specs/man/java.html for more info about this and other parameters.

Used vs Max vs Size -Jvisualvm?

In jvisual vm i see three attributes under Monitor>Heap, i see 3 attributes depicting memory details all with differnt figures
Size : ?
Used :- I believe this is the actual memory used
Max :- I believe this is the max heap size allocated to java process (specified with Xmx)
I am not sure what size actually depicts?

The three attributes can be defined as next:
Size: The actual total reserved heap size
Used: The actual used heap size.
Max: The max size of the Java heap (young generation + tenured generation)
Indeed when you launch your JVM, the initial heap size (can be defined with -Xms) will be the initial total reserved heap size, then according to how your application behaves, it could need to increase the total reserved size until it reaches the max size and if it is still not enough you could get OOME.

Size depicts the heap block size assigned to java process. Try with -Xms 512m or 1024m then your size to start with will be 512m but used memory may be much lower. As soon as used memory grows , heap resizing occurs proactively so that memory can be allocated to live objects.
Its like you have Gas tank of 30 litre max capacity . But you know for now you may just need 20 litres for the trip but actually used in trip is 5 litres

Heap size is actual size of heap your running application has.
Used heap is used portion of heap size.
Max heap size is the maximum value the application's heap size can have (can be defined by the arg option -Xmx).
When monitor memory usage of a java application, you see that heap size may vary during running of the application. It can not be greater than max heap size. For a sample profiling (monitoring of an application), see below image:

Mapper side OutOfMeory

I'm facing heap space OutOfMemory error during my Mapper side cleanup method, where i'm reading the data from inputStream and converting it into byte array using IOUtils.toByteArray(inptuStream);
I know i can resolve it by increasing the max heap space(Xmx), but i should be having enough heap space(1Gb) already. I found the below info on debugging(approximate space value),
runtime.maxMemory() - 1024Mb
runtime.totalMemory - 700Mb
runtime.freeMemory - 200Mb
My block size is 128 Mb and i'm not adding any additional data to it on my RecordReader. My output size from the mapper wont be more than 128 Mb.
And also i saw the available bytes in inputStream(.available()) which is provided an approximate value of 128 Mb.
I'm also a bit confused about the memory allocation of JVM. Let's say I set my heap space value as Xms-128m;Xmx-1024m. My tasktracker has 16Gb RAM and already I've 8jobs(8JVM) running in that tasktracker. Lets assume that the tasktracker can allocate only 8.5 Gb RAM for JVM and it'll use the rest for it's internal purpose. So we have 8.5Gb RAM available and 8 tasks are running which is currently using only 6Gb RAM. Is it possible for a new task be assigned to the same task tracker since already 8 tasks are running which might require 8Gb in which case the new task wont be able to provide user requested heap size(1Gb) if required.
PS: I know that not all heap needs to be in RAM(paging). My main question is, will the user be able to get the maximum requested heap size in all scenario?

Error while initializing Array:OutOfMemoryError

I have to allocate space to an array int input[] depending on the configuration parameters height and width.
int input[]=new int[height * width]; //this is line no 538
One of the configurations has parameters height=8192 and width=8192. So the size of the array becomes 67108864. But when i do this i get OutOfMemoryError.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at Test.main(Test.java:538)
I have ran this program on eclipse as well as on cygwin but i am facing the same problem. I think this is not an error and not exception. How can i rectify this?

Since 8192 * 8192 * 4 = 256 M (integers are 4 bytes each), your matrix is using 256 MB of heap space by itself.
You can tell the JVM how much heap space should be available to your application. From running man java and looking through the nonstandard options:
-Xmxn
Specify the maximum size, in bytes, of the memory allocation
pool. This value must a multiple of 1024 greater than 2MB.
Append the letter k or K to indicate kilobytes, or m or M to
indicate megabytes. The default value is chosen at runtime
based on system configuration. For more information, see
HotSpot Ergonomics
Examples:
-Xmx83886080
-Xmx81920k
-Xmx80m
On Solaris 7 and Solaris 8 SPARC platforms, the upper limit for
this value is approximately 4000m minus overhead amounts. On
Solaris 2.6 and x86 platforms, the upper limit is approximately
2000m minus overhead amounts. On Linux platforms, the upper limit
is approximately 2000m minus overhead amounts.
To use this option, you would start your application with a command like
java -Xmxn1024m -jar foo.jar
In Eclipse, you can add command-line options as well. This page on eclipse.org describes how to add command-line arguments to a Java program. You should add the -Xmxn1024m (or some other sufficiently large heap specification) to the "VM arguments" section of the dialog shown on that site.

You probably have too little heap space to hold an array of the size you are targeting. You can increase the size of your heap with command line switches. For example, to set it to 256MB, include this switch:
-Xmx256m
If you multiply height * width * 4 (4 is the storage in bytes for an int) you can get a rough gauge of the amount of heap you will need, assuming the rest of the program does not need a significant amount. You will certainly need some more heap than that quick calculation suggests. Add maybe 20% extra, and try that out.
To get a better number than a rule-of-thumb calculation, you can look into heap profilers. There are several open source options:
http://java-source.net/open-source/profilers
See http://javarevisited.blogspot.com/2011/05/java-heap-space-memory-size-jvm.html for a good discussion of the heap in Java.

memory is not enough for your program, may be memory leak there.
you may try below,if not solve try to increase jmx value.
java -xmx1g -xms512m

Depends on how much heap the JVM has. If you run it on the command line try adding -Xmx512m. If you work in an IDE add it to the "Run" properties.

An int is 32 bits (i.e. 4 bytes). So your array requires 8192*8192*4 bytes. This comes out at 256MB.
Java called with default arguments has only 64MB of heap space.
To get a larger heap, call Java using the -Xmx argument (Maximum memory size).
e.g. java -Xmx300M

Increase your memory arguments for your Java process by adding this flag to increase the heap. You might need to play around to get the optimal size for the heap. This will set the "max" heap size. The default is probably really small. 64M is a common max size for many Java EE containers.
*Note I'm not saying this is exactly the size you'll need. Your unique case will dictate the size you'll need which you may need to experiment with.
-Xmx256M

limit for max heap size in java heap setting

is there limit to increase the max heap size in java? I am wondering if the large heap size can be set as long as the physical memory is available.
For example, if a server has 100G for RAM, then can i set the max heap at 90G? I know that GC will halt the app, but I am just curious.
Thanks.

With a 32 bit JVM, the hard limit would be 4 GB but the actual one would be lower as, at least if you aren't running a 64 bit OS, some space must be left for non heap memory, like the JVM own address space (non java), stacks for all threads, architecture/OS limitations and the likes. A 64 bit JVM has no such limitation so you could set the limit to 90 GB although I wouldn't recommend it for the reason you already pointed.