With jdk 8 these were my steps to figure out how much memory is being consumed while a process is running:
/usr/java/latest/bin>: ./jps
27116 Main
7591 Jps
2879 AmbusProcessor
Then picked up process id to check the status of the heap :
/usr/java/latest/bin>: ./jmap -heap 2879
Attaching to process ID 2879, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.45-b08
using thread-local object allocation.
Parallel GC with 13 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 68719476736 (65536.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 134217728 (128.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 15427698688 (14713.0MB)
used = 9122094480 (8699.507217407227MB)
free = 6305604208 (6013.492782592773MB)
59.128031111311266% used
From Space:
capacity = 2062024704 (1966.5MB)
used = 813973552 (776.2656707763672MB)
free = 1248051152 (1190.2343292236328MB)
39.474481097196396% used
To Space:
capacity = 1944059904 (1854.0MB)
used = 0 (0.0MB)
free = 1944059904 (1854.0MB)
0.0% used
PS Old Generation
capacity = 8520204288 (8125.5MB)
used = 6649238896 (6341.208358764648MB)
free = 1870965392 (1784.2916412353516MB)
78.04083882548333% used
PS Perm Generation
capacity = 31981568 (30.5MB)
used = 16156728 (15.408256530761719MB)
free = 15824840 (15.091743469238281MB)
50.518873871349896% used
6141 interned Strings occupying 609896 bytes.
I am looking for a way to read the status that should look like the above example, However, not finding an option like that with openjdk 11. I have tried all possible option on jmap on openjdk11.
Is there a way still to get that kind of status reading with openjdk11?
I am trying to dump exactly when the outOfMemory is happening
For newer Java versions like JDK 11 you can use this command:
jhsdb jmap --heap --pid <pid>
Output is almost the same as with Java 8 and jmap.
There are some JVM options that may help you by logging GC info:
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=10M
The file created by HeapDumpOnOutOfMemoryError can be analyzed with tools shipped with the JDK. See also https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html
edit: There's also some official docs about debugging memory issues in Java 11: https://docs.oracle.com/en/java/javase/11/troubleshoot/troubleshoot-memory-leaks.html
I have configured 16GB heap in my elastic search node and the node has 36GB
in it, the elastic search java process is consuming 95% of it. If we put together the heap and non heap memory, together they are not 95%. I want to control the memory usage and don't want the system memory usage to go beyond 90%.
Following is the results of ps aux |grep elasticsearch
103 3242 55.0 95.4 208321876 36468416 ? Sl 00:16 778:35 /usr/bin/java -Xms16g -Xmx16g -XX:MaxDirectMemorySize=16g -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.1.1.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
Following is the results of jmap -heap
root#ice-bsd-none-551475:~# jmap -heap 3242
Attaching to process ID 3242, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.5-b02
using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 17179869184 (16384.0MB)
NewSize = 959643648 (915.1875MB)
MaxNewSize = 959643648 (915.1875MB)
OldSize = 16220225536 (15468.8125MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 863698944 (823.6875MB)
used = 89336768 (85.19818115234375MB)
free = 774362176 (738.4893188476562MB)
10.343507841547158% used
Eden Space:
capacity = 767754240 (732.1875MB)
used = 82442520 (78.6233139038086MB)
free = 685311720 (653.5641860961914MB)
10.738139329585467% used
From Space:
capacity = 95944704 (91.5MB)
used = 6894248 (6.574867248535156MB)
free = 89050456 (84.92513275146484MB)
7.18564726615864% used
To Space:
capacity = 95944704 (91.5MB)
used = 0 (0.0MB)
free = 95944704 (91.5MB)
0.0% used
concurrent mark-sweep generation:
capacity = 16220225536 (15468.8125MB)
used = 2346399531588267400 (2.237700969303386E12MB)
free = 15354485090581 MB
1.4465887212113964E10% used
31656 interned Strings occupying 4070216 bytes.
Inspite of the total memory from jmap -heap being less than 20GB, the total memory utilisation of the java process is 95.4% from the results of the top command.
I checked the Direct Memory used and Mapped memory used from jconsole
Direct memory used = 10798837846(10GB)
Mapped Memory used = 166282997194(166GB)
I want to reduce the total memory usage on the machine and control the total memory of this elastic search process.
I have a little gap in understanding how a JVM process allocates its own memory. As far as I know
RSS = Heap size + MetaSpace + OffHeap size
where OffHeap consists of thread stacks, direct buffers, mapped files (libraries and jars) and JVM code itself;
At the moment I’m trying to analyze my Java application (Spring Boot + Infinispan) which RSS is 779M (it runs in a docker container, so pid 1 is ok):
[ root#daf5a5ae9bb7:/data ]$ ps -o rss,vsz,sz 1
RSS VSZ SZ
798324 6242160 1560540
According to jvisualvm, committed Heap size is 374M
Metasapce size is 89M
In other words, I want to explain 799M - (374M + 89M) = 316M of OffHeap memory.
My app has (in average) 36 live threads.
Each of these threads consumes 1M:
[ root#fac6d0dfbbb4:/data ]$ java -XX:+PrintFlagsFinal -version |grep ThreadStackSize
intx CompilerThreadStackSize = 0
intx ThreadStackSize = 1024
intx VMThreadStackSize = 1024
So, here we can add 36M.
The only place where the app uses DirectBuffer is NIO. As far as I can see from JMX, it doesn’t consume a lot of resources - only 98K
The last step is mapped libs and jars. But according to pmap (full output)
[ root#daf5a5ae9bb7:/data ]$ pmap -x 1 | grep ".so.*" | awk '{ sum+=$3} END {print sum}'
12896K
plus
root#daf5a5ae9bb7:/data ]$ pmap -x 1 | grep “.jar" | awk '{ sum+=$3} END {print sum}'
9720K
we only have 20M here.
Hence, we still have to explain 316M - (36M + 20M) = 260M :(
Does anyone have any idea what I missed?
Approach:
You may want to use Java HotSpot Native Memory Tracking (NMT).
This may give you an exact list of memory allocated by the JVM, splitted up into the different areas heap, classes, threads, code, GC, compiler, internal, symbols, memory tracking, pooled free chunks, and unknown.
Usage:
You can start your application with -XX:NativeMemoryTracking=summary.
Observations of the current heap can be done with jcmd <pid> VM.native_memory summary.
Where to find jcmd / pid:
On a default OpedJDK installation on Ubuntu this can be found at /usr/bin/jcmd.
By just running jcmd without any parameter, you get a list of running Java applications.
user#pc:~$ /usr/bin/jcmd
5169 Main <-- 5169 is the pid
Output:
You will then receive a complete overview over your heap, looking something like the following:
Total: reserved=664192KB, committed=253120KB <--- total memory tracked by Native Memory Tracking
Java Heap (reserved=516096KB, committed=204800KB) <--- Java Heap
(mmap: reserved=516096KB, committed=204800KB)
Class (reserved=6568KB, committed=4140KB) <--- class metadata
(classes #665) <--- number of loaded classes
(malloc=424KB, #1000) <--- malloc'd memory, #number of malloc
(mmap: reserved=6144KB, committed=3716KB)
Thread (reserved=6868KB, committed=6868KB)
(thread #15) <--- number of threads
(stack: reserved=6780KB, committed=6780KB) <--- memory used by thread stacks
(malloc=27KB, #66)
(arena=61KB, #30) <--- resource and handle areas
Code (reserved=102414KB, committed=6314KB)
(malloc=2574KB, #74316)
(mmap: reserved=99840KB, committed=3740KB)
GC (reserved=26154KB, committed=24938KB)
(malloc=486KB, #110)
(mmap: reserved=25668KB, committed=24452KB)
Compiler (reserved=106KB, committed=106KB)
(malloc=7KB, #90)
(arena=99KB, #3)
Internal (reserved=586KB, committed=554KB)
(malloc=554KB, #1677)
(mmap: reserved=32KB, committed=0KB)
Symbol (reserved=906KB, committed=906KB)
(malloc=514KB, #2736)
(arena=392KB, #1)
Memory Tracking (reserved=3184KB, committed=3184KB)
(malloc=3184KB, #300)
Pooled Free Chunks (reserved=1276KB, committed=1276KB)
(malloc=1276KB)
Unknown (reserved=33KB, committed=33KB)
(arena=33KB, #1)
This gives a detailed overview of the different memory areas used by the JVM, and also shows the reserved and commited memory.
I don't know of a technique that gives you a more detailed memory consumption list.
Further reading:
You can also use -XX:NativeMemoryTracking=detail in combination with further jcmd commands. A more detailed explaination can be found at Java Platform, Standard Edition Troubleshooting Guide - 2.6 The jcmd Utility. You can check possible commands via "jcmd <pid> help"
I want to controller the old generation size of the java heap. And I have these settings below:
-Xmx1024m -XX:MaxNewSize=640m -XX:NewSize=640m -XX:SurvivorRatio=5 -XX:-UseAdaptiveSizePolicy -XX:PermSize=32m -server
After launch the application, I execute "jmap -heap pid" and it shows:
Attaching to process ID 96921, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.51-b03
using thread-local object allocation.
Parallel GC with 4 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 1073741824 (1024.0MB)
NewSize = 671088640 (640.0MB)
MaxNewSize = 671088640 (640.0MB)
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 5
PermSize = 33554432 (32.0MB)
MaxPermSize = 85983232 (82.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 480247808 (458.0MB)
used = 81462752 (77.68893432617188MB)
free = 398785056 (380.3110656738281MB)
16.962649416194733% used
From Space:
capacity = 95420416 (91.0MB)
used = 61444600 (58.59813690185547MB)
free = 33975816 (32.40186309814453MB)
64.39355703500601% used
To Space:
capacity = 95420416 (91.0MB)
used = 0 (0.0MB)
free = 95420416 (91.0MB)
0.0% used
PS Old Generation
capacity = 5767168 (5.5MB)
used = 90128 (0.0859527587890625MB)
free = 5677040 (5.4140472412109375MB)
1.5627774325284092% used
PS Perm Generation
capacity = 35127296 (33.5MB)
used = 35092152 (33.46648406982422MB)
free = 35144 (0.03351593017578125MB)
99.89995244723647% used
15721 interned Strings occupying 1939128 bytes.
My problem is why the old size is 5mb, I think it should be 300+mb = MaxHeapSize - NewSize - PermSize = 1024 - 640 - 32
What's wrong with my setting and how can I control the old size?
You're not setting a minimum heap size, thus allowing the old gen to grow/shrink. Adaptive size policy only affects the new/old gen boundary. Set -Xms1024m.
Considering that you didn't even know that I suspect the selection of your other parameters may be ill-informed. I would recommend setting high-level goals (pause times, throughput, heap free ratios) instead and letting the heuristics do their work.
Recommended reading: Hotspot GC Tuning Guide
I have a java program, which i ran and figured out its process id with jps.
How can i see what is the value of -Xms and -Xmx variable for this java process ?
Try
jcmd <PID> VM.command_line
jcmd <PID> VM.flags
you can use jps and do it from the command line:
jps # shows pids
jps -v <pid> # shows params
jps -v <localhost:pid> # the host must be indicated
if this is not enough you can do it programmatically inside the program to check the maximum amount of memory that the Java virtual machine will attempt to use:
Runtime.getRuntime().maxMemory()
and you can also use use the class MemoryUsage to get the initial, used and max ammount that can be used for memory management.
MemoryMXBean memoryBean = ManagementFactory.getMemoryMXBean();
memoryBean.getHeapMemoryUsage().getMax()
memoryBean.getHeapMemoryUsage().getUsed()
memoryBean.getHeapMemoryUsage().getInit()
I think jmap command will give you everything you want.
usage: jmap -heap {pid}
root#BobServerStation:/usr/local $ jmap -heap 3280
Attaching to process ID 3280, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.65-b04
using thread-local object allocation.
Parallel GC with 8 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 4116709376 (3926.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 85983232 (82.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 65011712 (62.0MB)
used = 42273152 (40.3148193359375MB)
free = 22738560 (21.6851806640625MB)
65.0239021547379% used
From Space:
capacity = 10485760 (10.0MB)
used = 10479760 (9.994277954101562MB)
free = 6000 (0.0057220458984375MB)
99.94277954101562% used
To Space:
capacity = 10485760 (10.0MB)
used = 0 (0.0MB)
free = 10485760 (10.0MB)
0.0% used
PS Old Generation
capacity = 171442176 (163.5MB)
used = 376368 (0.3589324951171875MB)
free = 171065808 (163.1410675048828MB)
0.21953057805332568% used
PS Perm Generation
capacity = 22020096 (21.0MB)
used = 15401488 (14.688003540039062MB)
free = 6618608 (6.3119964599609375MB)
69.94287400018601% used
8464 interned Strings occupying 699456 bytes.