Develop Reference

How to read heap status with jdk 11?

With jdk 8 these were my steps to figure out how much memory is being consumed while a process is running:
/usr/java/latest/bin>: ./jps
27116 Main
7591 Jps
2879 AmbusProcessor
Then picked up process id to check the status of the heap :
/usr/java/latest/bin>: ./jmap -heap 2879
Attaching to process ID 2879, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.45-b08
using thread-local object allocation.
Parallel GC with 13 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 68719476736 (65536.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 134217728 (128.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 15427698688 (14713.0MB)
used = 9122094480 (8699.507217407227MB)
free = 6305604208 (6013.492782592773MB)
59.128031111311266% used
From Space:
capacity = 2062024704 (1966.5MB)
used = 813973552 (776.2656707763672MB)
free = 1248051152 (1190.2343292236328MB)
39.474481097196396% used
To Space:
capacity = 1944059904 (1854.0MB)
used = 0 (0.0MB)
free = 1944059904 (1854.0MB)
0.0% used
PS Old Generation
capacity = 8520204288 (8125.5MB)
used = 6649238896 (6341.208358764648MB)
free = 1870965392 (1784.2916412353516MB)
78.04083882548333% used
PS Perm Generation
capacity = 31981568 (30.5MB)
used = 16156728 (15.408256530761719MB)
free = 15824840 (15.091743469238281MB)
50.518873871349896% used
6141 interned Strings occupying 609896 bytes.
I am looking for a way to read the status that should look like the above example, However, not finding an option like that with openjdk 11. I have tried all possible option on jmap on openjdk11.
Is there a way still to get that kind of status reading with openjdk11?
I am trying to dump exactly when the outOfMemory is happening

For newer Java versions like JDK 11 you can use this command:
jhsdb jmap --heap --pid <pid>
Output is almost the same as with Java 8 and jmap.

There are some JVM options that may help you by logging GC info:
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=10M
The file created by HeapDumpOnOutOfMemoryError can be analyzed with tools shipped with the JDK. See also https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html
edit: There's also some official docs about debugging memory issues in Java 11: https://docs.oracle.com/en/java/javase/11/troubleshoot/troubleshoot-memory-leaks.html

Explain and Control elastic search memory components

I have configured 16GB heap in my elastic search node and the node has 36GB
in it, the elastic search java process is consuming 95% of it. If we put together the heap and non heap memory, together they are not 95%. I want to control the memory usage and don't want the system memory usage to go beyond 90%.
Following is the results of ps aux |grep elasticsearch
103 3242 55.0 95.4 208321876 36468416 ? Sl 00:16 778:35 /usr/bin/java -Xms16g -Xmx16g -XX:MaxDirectMemorySize=16g -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.1.1.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
Following is the results of jmap -heap
root#ice-bsd-none-551475:~# jmap -heap 3242
Attaching to process ID 3242, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.5-b02
using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 17179869184 (16384.0MB)
NewSize = 959643648 (915.1875MB)
MaxNewSize = 959643648 (915.1875MB)
OldSize = 16220225536 (15468.8125MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 863698944 (823.6875MB)
used = 89336768 (85.19818115234375MB)
free = 774362176 (738.4893188476562MB)
10.343507841547158% used
Eden Space:
capacity = 767754240 (732.1875MB)
used = 82442520 (78.6233139038086MB)
free = 685311720 (653.5641860961914MB)
10.738139329585467% used
From Space:
capacity = 95944704 (91.5MB)
used = 6894248 (6.574867248535156MB)
free = 89050456 (84.92513275146484MB)
7.18564726615864% used
To Space:
capacity = 95944704 (91.5MB)
used = 0 (0.0MB)
free = 95944704 (91.5MB)
0.0% used
concurrent mark-sweep generation:
capacity = 16220225536 (15468.8125MB)
used = 2346399531588267400 (2.237700969303386E12MB)
free = 15354485090581 MB
1.4465887212113964E10% used
31656 interned Strings occupying 4070216 bytes.
Inspite of the total memory from jmap -heap being less than 20GB, the total memory utilisation of the java process is 95.4% from the results of the top command.
I checked the Direct Memory used and Mapped memory used from jconsole
Direct memory used = 10798837846(10GB)
Mapped Memory used = 166282997194(166GB)
I want to reduce the total memory usage on the machine and control the total memory of this elastic search process.

Why is set and actual value of PermSize and MaxPermSize different?

I have set the values of PermSize and MaxPermSize using JAVA_OPTS variable of tomcat as below
JAVA_OPTS="$JAVA_OPTS -XX:PermSize=512M"
JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512M"
and have the values of Xms and Xmx set as
JAVA_OPTS="$JAVA_OPTS -Xms1024M"
JAVA_OPTS="$JAVA_OPTS -Xmx1024M"
but when I check the actual size of perm-gen using the command jmap -heap [ process id ] they appear as
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 792723456 (756.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
**PermSize = 21757952 (20.75MB)**
**MaxPermSize = 85983232 (82.0MB)**
G1HeapRegionSize = 0 (0.0MB)
Perm Generation:
**capacity = 69271552 (66.0625MB)**
**used = 69247376 (66.03944396972656MB)**
**free = 24176 (0.0230560302734375MB)**
**99.96509967035242% used**
Why is the actual allocation of permgen space different when campared to what was allocated using the attributes PermSize and MaxPermSize ?
additional info: the RAM of the linux machine on which the tomcat server is installed is:
total used free shared buffers cached
Mem: 3089928 3006592 83336 0 67828 1874144
-/+ buffers/cache: 1064620 2025308
Swap: 1020088 112 1019976

The setenv.sh file was missed during the Jenkins build and this file was sourcing the custom user configurations from tomcat.env thus causing the issue.

Command to find out -Xms and -Xmx variable values for a given java process?

I have a java program, which i ran and figured out its process id with jps.
How can i see what is the value of -Xms and -Xmx variable for this java process ?

Try
jcmd <PID> VM.command_line
jcmd <PID> VM.flags

you can use jps and do it from the command line:
jps # shows pids
jps -v <pid> # shows params
jps -v <localhost:pid> # the host must be indicated
if this is not enough you can do it programmatically inside the program to check the maximum amount of memory that the Java virtual machine will attempt to use:
Runtime.getRuntime().maxMemory()
and you can also use use the class MemoryUsage to get the initial, used and max ammount that can be used for memory management.
MemoryMXBean memoryBean = ManagementFactory.getMemoryMXBean();
memoryBean.getHeapMemoryUsage().getMax()
memoryBean.getHeapMemoryUsage().getUsed()
memoryBean.getHeapMemoryUsage().getInit()

I think jmap command will give you everything you want.
usage: jmap -heap {pid}
root#BobServerStation:/usr/local $ jmap -heap 3280
Attaching to process ID 3280, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.65-b04
using thread-local object allocation.
Parallel GC with 8 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 4116709376 (3926.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 85983232 (82.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 65011712 (62.0MB)
used = 42273152 (40.3148193359375MB)
free = 22738560 (21.6851806640625MB)
65.0239021547379% used
From Space:
capacity = 10485760 (10.0MB)
used = 10479760 (9.994277954101562MB)
free = 6000 (0.0057220458984375MB)
99.94277954101562% used
To Space:
capacity = 10485760 (10.0MB)
used = 0 (0.0MB)
free = 10485760 (10.0MB)
0.0% used
PS Old Generation
capacity = 171442176 (163.5MB)
used = 376368 (0.3589324951171875MB)
free = 171065808 (163.1410675048828MB)
0.21953057805332568% used
PS Perm Generation
capacity = 22020096 (21.0MB)
used = 15401488 (14.688003540039062MB)
free = 6618608 (6.3119964599609375MB)
69.94287400018601% used
8464 interned Strings occupying 699456 bytes.

Java OutOfMemory Error

I am seeing an OutOfMemory problem and I am not sure it is the PERM GEN area or the heap space. The error message does not say anything about which area ran out of memory.
Here is a partial stack trace:
The following is information that may be useful to the developer of BETWEENNESS: java.lang.OutOfMemoryError at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.(ZipFile.java:127) at java.util.zip.ZipFile.(ZipFile.java:143) at com..util.internal.ZipFiles.unzip(ZipFiles.java:91)
I looked at the heap space just before it ran out of memory using the jmap -heap command:
using thread-local object allocation.
Parallel GC with 23 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 31675383808 (30208.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 536870912 (512.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 9762177024 (9309.9375MB)
used = 7286558512 (6949.003707885742MB)
free = 2475618512 (2360.933792114258MB)
74.64071276403028% used
From Space:
capacity = 396230656 (377.875MB)
used = 340623360 (324.84375MB)
free = 55607296 (53.03125MB)
85.96592788620576% used
To Space:
capacity = 398131200 (379.6875MB)
used = 0 (0.0MB)
free = 398131200 (379.6875MB)
0.0% used
PS Old Generation
capacity = 1992163328 (1899.875MB)
used = 1455304512 (1387.8865356445312MB)
free = 536858816 (511.98846435546875MB)
73.05146578825087% used
PS Perm Generation
capacity = 418578432 (399.1875MB)
used = 418567008 (399.1766052246094MB)
free = 11424 (0.010894775390625MB)
99.99727076238844% used }
And also, I had supplied the following arguments to the JVM: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/ but I do not see any heap.
My questions is why is the heap not being generated and how do I figure out which part of the JVm is getting full.
Thanks.

Your provided information says, that your PermGen is 99% full. And your Heap is already 73% full. So increasing both would not be bad at all.
Further you could activate the garbage collector's logging with -XX:+PrintGCDetails to get detailed information on how your JVM is using memory. Additionally activate -XX:+PrintGCTimeStamps and -XX:+PrintGCDateStamps. -Xloggc:$filename sends the logs to a file, which you could easily analyze with something like IMB PMAT tool or GCViewer.
Additionally you should consider using VisualVM to monitor your application while running.
Besides:
A colleague of mine found a smart way to get a heapdump many times faster through gdb:
cat > /tmp/dodump <<EOF
gcore jvm.core
detach
quit
EOF
time gdb --batch --command=/tmp/dodump --pid=`pgrep java`
jmap -dump:format=b,file=jvm.hprof `which java` jvm.core
rm jvm.core
gzip -9 jvm.hprof
Source
Credits go fully to him.

According to stack trace, OutOfMemoryError happens inside ZipFile.open() native method. This means that the problem has nothing to do with Java Heap size nor with PermGen. It is likely related with ZIP cache which is malloc'ed or mmap'ed internally by JDK library.
Try adding -Dsun.zip.disableMemoryMapping=true JVM option to see if it helps.
How large is ZIP file being opened?