I'm facing heap space OutOfMemory error during my Mapper side cleanup method, where i'm reading the data from inputStream and converting it into byte array using IOUtils.toByteArray(inptuStream);
I know i can resolve it by increasing the max heap space(Xmx), but i should be having enough heap space(1Gb) already. I found the below info on debugging(approximate space value),
runtime.maxMemory() - 1024Mb
runtime.totalMemory - 700Mb
runtime.freeMemory - 200Mb
My block size is 128 Mb and i'm not adding any additional data to it on my RecordReader. My output size from the mapper wont be more than 128 Mb.
And also i saw the available bytes in inputStream(.available()) which is provided an approximate value of 128 Mb.
I'm also a bit confused about the memory allocation of JVM. Let's say I set my heap space value as Xms-128m;Xmx-1024m. My tasktracker has 16Gb RAM and already I've 8jobs(8JVM) running in that tasktracker. Lets assume that the tasktracker can allocate only 8.5 Gb RAM for JVM and it'll use the rest for it's internal purpose. So we have 8.5Gb RAM available and 8 tasks are running which is currently using only 6Gb RAM. Is it possible for a new task be assigned to the same task tracker since already 8 tasks are running which might require 8Gb in which case the new task wont be able to provide user requested heap size(1Gb) if required.
PS: I know that not all heap needs to be in RAM(paging). My main question is, will the user be able to get the maximum requested heap size in all scenario?
Related
I'm Trying to run a Job on Flink task manager and I'm getting this exception :
Initializing the input processing failed: Too little memory provided to sorter to perform task. Required are at least 12 pages. Current page size is 32768 bytes.
I've set heap size in both task and job manager's via flink-conf.yml , anything else I should change to increase the memory ?
taskmanager.heap.size: 4096m
taskmanager.memory.size: 4096m
jobmanager.heap.size: 2048m
The error message indicates that the sorter does not get enough memory pages. The reason is that the available managed memory is not sufficient. There are multiple ways to solve this problem:
Increase the available memory for a TaskManager via taskmanager.heap.size
Increase the fraction of the managed memory which is taken from taskmanager.heap.size via taskmanager.memory.fraction (per default it is 0.7)
Decrease the page size via taskmanager.memory.segment-size
Decrease the number of slots on a TaskManager since a reduced parallelism per TM will decrease the number of memory consumers on the TM (operators get a bigger share of the available memory)
If you are running exclusively batch loads, then you should also activate taskmanager.memory.preallocate: true which will enable the memory allocation at start-up time. This is usually faster because it reduces the garbage collection pressure.
Another comment concerning taskmanager.memory.size: This value always needs to be smaller or equal than taskmanager.heap.size since it specifies how much memory from the overall heap space will be used for managed memory. If this parameter is not specified, then Flink will take a fraction of the available heap memory for the managed memory (specified via taskmanager.memory.fraction).
If I try to increase the spark.kryoserializer.buffer.mb to more than 1GB (1024), I get the following error.
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Note that my system has 128 GB of RAM, so there is plenty memory to have an array of bigger size than that. Is it because the serializer is using 32 bit indexing? How can we overcome this limitation? Or it could be that Java restricts array sizes to such limit. How can we tell Java to use a larger limit? And how can we specifically set it through Spark?
I recently made a heapdump in a hprof format when my jboss server was running with a xms of 4096m and xmx of 4096m and a permsize of 512m.
The hprof file generated is over 5gb. When I load the heapdump in visualvm, mat analyzer or yourkit, I only see a total bytes of approximately 1gb. I've tried changed the reachability scope in yourkit but it does not show more than 1 gb.
Any idea what this big difference in filesize vs displayed heapdump size can cause?
ps: I'm using jdk1.6.0_23
Unfortunately I'm not allowed to submit screenshots here.
On the filesystem the hprof size is of 5.227.659 kb and in yourkit it states:
Objects: 9.738.282 / shallow size 740 mb / retained size: 740 mb String reachable among them: 6.652.515 (68%) / shallow size: 381 mb (51%) / retained size: 381 MB (51%)
The largest retained size is a byte[] of 206.810.176
which command did you use to generate heap dump?
$JAVA_HOME/bin/jmap -dump:live,format=b,file=c:/tmp/heap_dump.bin PID
maybe you need to pass live option, according to spec
-dump:<dump-options> to dump java heap in hprof binary format
dump-options:
live dump only live objects; if not specified,
all objects in the heap are dumped.
Did you try "Unreachable Objects Histogram" (you can find the link from the top of "Overview" page)? In one of my heapdumps sized 1509MB, mat shows only 454MB, but the rest is essentially garbage, and sure enough, the sum of "Shallow Heap" in unreachable objects histogram is 966MB.
This just means that most likely your heap-dump consisted of a large amount of unreachable objects that would have been garbage collected, if a GC were to run.
Now that does not mean that you don't still have a leak, it just means that in your 5 GB Hprof, 4 GB of objects were unreachable and hence were not interesting sources of a leak.
In Java a memory leak can only occur if Garbage Collection can't clean out an object because something is holding a reference to it (unexpectedly). So your leak (if any) is to be found in the 1 GB of objects that remained in your hprof.
I've set up a glassfish cluster with 1 DAS and 2 Node Agents.
The system has TimedObjects which are batched once a day. As glassfish architecture, there is only 1 cluster instance allowed to trigger timeout event of each Timer created by TimerService.
My problems is about Heap size of a cluster instance which triggers batch job. The VisualVM shows that one instance always has scalable heap size (increase when the server is loaded and decrease after that) but another one always has heap size at the maximum and never decrease.
It is acceptable to tell me that the heap size is at the maximum because the batch job is huge. But, the only question I have is why it does not decrease after the job is done???
VisualVM shows that the "Used Heap Memory" of the instance which triggers timeout event decreases after the batch job. But, why its "Heap Size" is not scaled down accordingly?
Thank you for your advice!!! ^^
Presumably you have something referencing the memory. I suggest getting a copy of MAT and doing a heap dump. From there you can see what's been allocated and what is referencing it.
This is the final answer (thanks Preston ^^)
From the article :
http://www.ibm.com/developerworks/java/library/j-nativememory-linux/index.html
I captured these statements to answer my question!
1 :
"Runtime environments (JVM) provide capabilities that are driven by some unknown user code; that makes it impossible to predict which resources the runtime environment will require in every situation"
2 : This is why the node which triggers batch job always consumes the memory at all time.
"Reserving native memory is not the same as allocating it. When native memory is reserved, it is not backed with physical memory or other storage. Although reserving chunks of the address space will not exhaust physical resources, it does prevent that memory from being used for other purposes"
3 : And this is why the node which does not trigger batch job has scalable Heap Size behavior.
"Some garbage collectors minimise the use of physical memory by decommitting (releasing the backing storage for) parts of the heap as the used area of heap shrinks."
is there limit to increase the max heap size in java? I am wondering if the large heap size can be set as long as the physical memory is available.
For example, if a server has 100G for RAM, then can i set the max heap at 90G? I know that GC will halt the app, but I am just curious.
Thanks.
With a 32 bit JVM, the hard limit would be 4 GB but the actual one would be lower as, at least if you aren't running a 64 bit OS, some space must be left for non heap memory, like the JVM own address space (non java), stacks for all threads, architecture/OS limitations and the likes. A 64 bit JVM has no such limitation so you could set the limit to 90 GB although I wouldn't recommend it for the reason you already pointed.