Develop Reference

Is the CompressedClassSpace allocated on JVM startup? Is there any risk in decreasing the default value?

We have multiple, JVM11-based apps running on k8s. A couple of weeks ago Kubernetes killed all of the pods from the same deployment at the same time in our testing environment. The reason was OOMKilled.
The exciting part is that we have set the -Xmx to 3200m, and the pods have a memory request of 4Gi, limit of 5Gi. The heap size is correctly set, the app can run with the 3200Mi heap.
During the investigation the following looked interesting:
the memory usage of the pod increases during the morning, and never decreases. The pods starts with a new dataset every midnight, and the same behaviour happens every day. We have peak usage during the morning and during the afternoon hours, but the pod's memory usage never decreases. Not even in the nonpeak time during the middle of the day.
(some new pods spin up during these peak hours, but we have a few pods that run all day. the picture is from one all-day-running pod)
it's a spring boot application, and the micrometer framework is set up. On the dashboards I can see the following:
the heap never extends 3200Mi, which is the expected behavior
the max of the nonheap memory usage is way higher than the used/committed values
the max of the JVM Total memory usage is equal to the pod's memory usage. And it's equal to sum(max-nonheap, max-heap).
According to the Prometheus metrics, the nonheap part of the JVM is allocated of (jvm_memory_used_bytes{container="<mycontainer>", area="nonheap"}):
CodeHeap 'non-nmethods'
CodeHeap 'non-profiled nmethods'
CodeHeap 'profiled nmethods'
Compressed Class Space
Metaspace
What's interesting is, that the jvm_memory_max_bytes is reasonable for every nonheap part, except for the Compressed Class Space.
The Compressed Class Space never uses more than ~20-30 megabytes of RAM (jvm_memory_used_bytes), but as far as I understand, it always allocates the 1 gigabyte jvm_memory_max_bytes.
I've read a couple of docs, and all mention that the default value of the -XX:CompressedClassSpaceSize is 1 gigabyte.
Am I correctly assuming that the JVM allocates the whole gigabyte? If so, is it safe to set the -XX:CompressedClassSpaceSize to 256M for example? See the metrics below, JVM never uses more than ~20 megabytes of the CompressedClassSpaceSize part out of the avaiable the memory. Is there any downside, risk for it?
Thanks,
Regards,

How does maximum heap size xmx influence the memory allocation / used heap in Java?

A. If I execute a huge simulation program with -Xmx100000m (~100GB) I see some spikes in the used heap (~30 GB). That spikes increase the heap size and decreases the memory that can be used by other programs. I would like to limit the heap size to the size that is actually required to run the program without memory exceptions.
B. If I execute my simulation program with -Xmx10000 (~10GB) I am able to limit the used heap size (~ 7 GB). The total heap size is less, too (of course). I do not get out of memory exceptions in the first phase of the program that is shown in the VisualVM figures (about 16 minutes).
I naively expected that if I increase xmx from 10GB (B) to 100GB (A) that the used heap would stay about the same and that Java only would use more memory in order to avoid out of memory exceptions. However, the behavior seems to be different. I guess that Java works this way in order to improve performance.
An explanation for the large used heap in A might be that the growth behavior of hash maps is different if xmx is larger? Does xmx have an effect on the load factor?
In the phase of the program where a lot of mini spikes exist (see for example B at 12:06) instead of a few large ones (A) some java streams are processed. Does the memory allocation for stream processing automatically adapt with the xmx value? (There is still some memory left that could be used to have less mini spikes at 12:06 in B.)
If not, what might be the reasons for the larger used heap in A?
How can I tell Java to keep the used heap low if possible (like in the curves for B) but to take more memory if an out of memory exception could occur (allow to temporarily switch to A). Could this be done by tuning some garbage collection properties?
Edit
As stated by the answer below, the profile can be altered by garbage collection parameters. Applying -Xmx100000m -XX:MaxGCPauseMillis=1000 adapts the profile from A to consume less memory (~ 20 GB used) and more time (~ 22 min).

I would like to limit the heap size to the size that is actually required to run the program without memory exceptions.
You do not actually want to do that because it would make your program extremely slow because only providing the amount equivalent to the application peak footprint means that every single allocation would trigger a garbage collection while the application is near the maximum.
I guess that Java works this way in order to improve performance.
Indeed.
The JVM has several goals, in descending order:
pause times (latency)
allocation throughput
footprint
If you want to prioritize footprint over other goals you have to relax the other ones.
set -XX:MaxGCPauseMillis=18446744073709551615, this is the default for the parallel collector but G1 has a 200ms default.
configure it to keep less breathing room

Mapper side OutOfMeory

I'm facing heap space OutOfMemory error during my Mapper side cleanup method, where i'm reading the data from inputStream and converting it into byte array using IOUtils.toByteArray(inptuStream);
I know i can resolve it by increasing the max heap space(Xmx), but i should be having enough heap space(1Gb) already. I found the below info on debugging(approximate space value),
runtime.maxMemory() - 1024Mb
runtime.totalMemory - 700Mb
runtime.freeMemory - 200Mb
My block size is 128 Mb and i'm not adding any additional data to it on my RecordReader. My output size from the mapper wont be more than 128 Mb.
And also i saw the available bytes in inputStream(.available()) which is provided an approximate value of 128 Mb.
I'm also a bit confused about the memory allocation of JVM. Let's say I set my heap space value as Xms-128m;Xmx-1024m. My tasktracker has 16Gb RAM and already I've 8jobs(8JVM) running in that tasktracker. Lets assume that the tasktracker can allocate only 8.5 Gb RAM for JVM and it'll use the rest for it's internal purpose. So we have 8.5Gb RAM available and 8 tasks are running which is currently using only 6Gb RAM. Is it possible for a new task be assigned to the same task tracker since already 8 tasks are running which might require 8Gb in which case the new task wont be able to provide user requested heap size(1Gb) if required.
PS: I know that not all heap needs to be in RAM(paging). My main question is, will the user be able to get the maximum requested heap size in all scenario?

Setting a smaller JVM heap size within a JNI client application

I'm attempting to debug a problem with pl/java, a procedural language for PostgreSQL. I'm running this stack on a Linux server.
Essentially, each Postgres backend (connection process) must start its own JVM, and does so using the JNI. This is generally a major limitation of pl/java, but it has one particularly nasty manifestation.
If native memory runs out (I realise that this may not actually be due to malloc() returning NULL, but the effect is about the same), this failure is handled rather poorly. It results in an OutOfMemoryError due to "native memory exhaustion". This results in a segfault of the Postgres backend, originating from within libjvm.so, and a javacore file that says something like:
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received
1TIDATETIME Date: 2012/09/13 at 16:36:01
1TIFILENAME Javacore filename: /var/lib/PostgreSQL/9.1/data/javacore.20120913.104611.24742.0002.txt
***SNIP***
Now, there are reasonably well-defined ways of ameliorating these types of problems with Java, described here:
http://www.ibm.com/developerworks/java/library/j-nativememory-linux/
I think that it would be particularly effective if I could set the maximum heap size to a value that is far lower than the default. Ordinarily, it is possible to do something along these lines:
The heap's size is controlled from the Java command line using the -Xmx and -Xms options (mx is the maximum size of the heap, ms is the initial size). Although the logical heap (the area of memory that is actively used) can grow and shrink according to the number of objects on the heap and the amount of time spent in GC, the amount of native memory used remains constant and is dictated by the -Xmx value: the maximum heap size. Most GC algorithms rely on the heap being allocated as a contiguous slab of memory, so it's impossible to allocate more native memory when the heap needs to expand. All heap memory must be reserved up front.
However, it is not apparent how I can follow these steps such that pl/java's JNI initialisation initialises a JVM with a smaller heap; I can't very well pass these command line arguments to Postgres. So, my question is, how can I set the maximum heap size or otherwise control these problems in this context specifically? This appears to be a general problem with pl/java, so I expect to be able to share whatever solution I eventually arrive at with the Postgres community.
Please note that I am not experienced with JVM internals, and am not generally familiar with Java.
Thanks

According to slide 19 in this presentation postgresql.conf can have the parameter pljava.vmoptions where you can pass arguments to the JVM.

Glassfish V2.1.1 Heap size never decrease after server batch job

I've set up a glassfish cluster with 1 DAS and 2 Node Agents.
The system has TimedObjects which are batched once a day. As glassfish architecture, there is only 1 cluster instance allowed to trigger timeout event of each Timer created by TimerService.
My problems is about Heap size of a cluster instance which triggers batch job. The VisualVM shows that one instance always has scalable heap size (increase when the server is loaded and decrease after that) but another one always has heap size at the maximum and never decrease.
It is acceptable to tell me that the heap size is at the maximum because the batch job is huge. But, the only question I have is why it does not decrease after the job is done???
VisualVM shows that the "Used Heap Memory" of the instance which triggers timeout event decreases after the batch job. But, why its "Heap Size" is not scaled down accordingly?
Thank you for your advice!!! ^^

Presumably you have something referencing the memory. I suggest getting a copy of MAT and doing a heap dump. From there you can see what's been allocated and what is referencing it.

This is the final answer (thanks Preston ^^)
From the article :
http://www.ibm.com/developerworks/java/library/j-nativememory-linux/index.html
I captured these statements to answer my question!
1 :
"Runtime environments (JVM) provide capabilities that are driven by some unknown user code; that makes it impossible to predict which resources the runtime environment will require in every situation"
2 : This is why the node which triggers batch job always consumes the memory at all time.
"Reserving native memory is not the same as allocating it. When native memory is reserved, it is not backed with physical memory or other storage. Although reserving chunks of the address space will not exhaust physical resources, it does prevent that memory from being used for other purposes"
3 : And this is why the node which does not trigger batch job has scalable Heap Size behavior.
"Some garbage collectors minimise the use of physical memory by decommitting (releasing the backing storage for) parts of the heap as the used area of heap shrinks."