Develop Reference

java process cannot be started

Background: Since the past few days, my linux development machine Java services one by one by the system kill, I looked at the system logs are OOM caused. Now I can't start the java process if I set the initial heap too large.
I can't see with the usual troubleshooting means, the development machine is a virtual machine (I don't think I can exclude the problem of the physical machine where the virtual machine is located, my colleague's machine and I applied at the same time, also have this problem), the total memory is about 6G, buff/cache + free total of about 5G. Thank you all for your help.
The crash logs at startup are in the attachment, and the system information and jdk information are in there.
enter link description here
Start-up log:
[~ jdk1.8.0_281]$java -Xms1000m -version
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000a5400000, 699400192, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 699400192 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid7617.log
Memory usage is as follows:
[~ jdk1.8.0_281]$free -h
total used free shared buff/cache available
Mem: 5.7G 502M 213M 4.6G 5.0G 328M
Swap: 0B 0B 0B
The io situation is as follows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.03 2.77 0.84 2.80 48.95 97.58 80.57 0.05 22.14 65.48 9.22 0.55 0.20
scd0 0.00 0.00 0.00 0.00 0.01 0.00 66.96 0.00 0.34 0.34 0.00 0.24 0.00

JVM memory usage more than reported by OS [duplicate]

This question already has an answer here:
Why does a JVM report more committed memory than the linux process resident set size?
(1 answer)
Closed 3 years ago.
I have a JVM which reports committed heap memory as around 8GB (Other sections should be over and above this). But my OS shows memory usage as around 5GB. I understand the memory usage can be more than the committed memory due to non-heap, metaspace etc, but how is it possible that the usage is lesser than reported by the jvm?
The output of free shows memory usage of 5.5GB
#free -m
total used free shared buff/cache available
Mem: 24115 5536 16355 10 2223 18209
Swap: 0 0 0
Output of Native Memory Tracker (NMT) shows reserved memory as ~ 11 GB
#jcmd <pid> VM.native_memory
Total: reserved=12904933KB, committed=11679661KB
- Java Heap (reserved=8388608KB, committed=8388608KB)
(mmap: reserved=8388608KB, committed=8388608KB)
- Class (reserved=1161913KB, committed=127417KB)
(classes #20704)
(malloc=2745KB #33662)
(mmap: reserved=1159168KB, committed=124672KB)
- Thread (reserved=2585224KB, committed=2585224KB)
(thread #2505)
(stack: reserved=2574004KB, committed=2574004KB)
(malloc=8286KB #12532)
(arena=2934KB #5004)
- Code (reserved=264623KB, committed=90231KB)
(malloc=15023KB #22507)
(mmap: reserved=249600KB, committed=75208KB)
- GC (reserved=378096KB, committed=378096KB)
(malloc=34032KB #45794)
(mmap: reserved=344064KB, committed=344064KB)
- Compiler (reserved=776KB, committed=776KB)
(malloc=645KB #1914)
(arena=131KB #7)
- Internal (reserved=53892KB, committed=53892KB)
(malloc=53860KB #67113)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=26569KB, committed=26569KB)
(malloc=22406KB #204673)
(arena=4163KB #1)
- Native Memory Tracking (reserved=6756KB, committed=6756KB)
(malloc=494KB #6248)
(tracking overhead=6262KB)
- Arena Chunk (reserved=11636KB, committed=11636KB)
(malloc=11636KB)
- Tracing (reserved=10456KB, committed=10456KB)
(malloc=10456KB #787)
- Unknown (reserved=16384KB, committed=0KB)
(mmap: reserved=16384KB, committed=0KB)
OS - Debian 9
Java -
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
I have read through some awesome answers like this one which explains NMT very well, but it doesn't address this issue. I would like to understand how this is possible.

Duplicate of Why does a JVM report more committed memory than the linux process resident set size?
This pertains to the difference between Reserved / Committed / Resident memory as explained in the above answer.
Closing this question.

SBT compile always gives " GC overhead limit exceeded error" irrespective of the heap size

sbt 0.13.8 compile takes a very long time (> 25 mins) and finally errors out with GC overhead limit
[info] Compiling 323 Scala sources and 1 Java source to .../target/scala-2.12/classes ...
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to .../java_pid10265.hprof ...
Heap dump file created [7078096837 bytes in 71.966 secs]
[error] ## Exception when compiling 324 sources to .../target/scala-2.12/classes
[error] GC overhead limit exceeded
[error] scala.reflect.internal.util.HashSet.growTable(HashSet.scala:97)
[error] scala.reflect.internal.util.HashSet.addEntry(HashSet.scala:63)
...
[addJava] arg = '-Xmx4G'
[addJava] arg = '-Xms1024m'
[addJava] arg = '-XX:ReservedCodeCacheSize=512m'
[addJava] arg = '-XX:MaxMetaspaceSize=2G'
[addJava] arg = '-XX:+PrintFlagsFinal'
[addJava] arg = '-XX:+HeapDumpOnOutOfMemoryError'
[addJava] arg = '-XX:HeapDumpPath=....
Information from hprof file:
Basic info:
Date taken: Wed Mar 20 15:04:26 PDT 2019
File: .../java_pid10265.hprof
File size: 6,750.2 MB
Total bytes: 6,033,077,813
Total classes: 22,538
Total instances: 116,328,564
Classloaders: 163
GC roots: 3,101
Number of objects pending for finalization: 0
Heap dumped on OutOfMemoryError exception
Thread causing OutOfMemoryError exception: pool-6-thread-8
Environment:
OS: Linux (3.10.0-957.5.1.el7.x86_64)
Architecture: amd64 64bit
Java Home: /mnt/software/j/jdk/1.8.0_144/jre
Java Version: 1.8.0_144
JVM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01, mixed mode)
Java Vendor: Oracle Corporation
System properties:
awt.toolkit=sun.awt.X11.XToolkit
com.sun.management.jmxremote.authenticate=false
com.sun.management.jmxremote.port=9933
com.sun.management.jmxremote.ssl=false
file.encoding=UTF-8
file.encoding.pkg=sun.io
file.separator=/
java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment
java.awt.printerjob=sun.print.PSPrinterJob
java.class.path=/mnt/software/s/sbt/0.13.8/bin/sbt-launch.jar
java.class.version=52.0
java.endorsed.dirs=/mnt/software/j/jdk/1.8.0_144/jre/lib/endorsed
java.ext.dirs=/mnt/software/j/jdk/1.8.0_144/jre/lib/ext:/usr/java/packages/lib/ext
java.home=/mnt/software/j/jdk/1.8.0_144/jre
java.io.tmpdir=/tmp
java.library.path=/mnt/software/j/jdk/1.8.0_144/jre/lib/amd64/server:/mnt/software/h/htslib/1.9/lib:/mnt/software/z/zlib/1.2.11/lib:/mnt/software/g/gcc/6.4.0/libc-2.5/compat/lib:/mnt/software/g/gcc/6.4.0/libc-2.5/lib64:/mnt/software/j/jdk/1.8.0_71/jre/lib/amd64/server:/mnt/software/o/openblas/0.2.14/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
java.rmi.server.randomIDs=true
java.runtime.name=Java(TM) SE Runtime Environment
java.runtime.version=1.8.0_144-b01
java.specification.name=Java Platform API Specification
java.specification.vendor=Oracle Corporation
java.specification.version=1.8
java.vendor=Oracle Corporation
java.vendor.url=http://java.oracle.com/
java.vendor.url.bug=http://bugreport.sun.com/bugreport/
java.version=1.8.0_144
java.vm.info=mixed mode
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name=Java Virtual Machine Specification
java.vm.specification.vendor=Oracle Corporation
java.vm.specification.version=1.8
java.vm.vendor=Oracle Corporation
java.vm.version=25.144-b01
jline.esc.timeout=0
jline.shutdownhook=false
jna.loaded=true
jna.nosys=true
jna.platform.library.path=/usr/lib64:/lib64:/usr/lib:/lib:/usr/lib/vmware-tools/lib64/libvmGuestLibJava.so:/usr/lib/vmware-tools/lib32/libvmGuestLibJava.so:/usr/lib/vmware-tools/lib64/libvmGuestLib.so:/usr/lib/vmware-tools/lib32/libvmGuestLib.so:/usr/lib64/mysql:/usr/lib/vmware-tools/lib64/libDeployPkg.so:/usr/lib/vmware-tools/lib32/libDeployPkg.so
jnidispatch.path=/tmp/jna--845103277/jna1647140300662886831.tmp
line.separator=\n
os.arch=amd64
os.name=Linux
os.version=3.10.0-957.5.1.el7.x86_64
path.separator=:
sbt.log.noformat=true
sun.arch.data.model=64
sun.boot.class.path=/mnt/software/j/jdk/1.8.0_144/jre/lib/resources.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/rt.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/sunrsasign.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/jsse.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/jce.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/charsets.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/jfr.jar:/mnt/software/j/jdk/1.8.0_144/jre/classes
sun.boot.library.path=/mnt/software/j/jdk/1.8.0_144/jre/lib/amd64
sun.cpu.endian=little
sun.cpu.isalist=
sun.io.unicode.encoding=UnicodeLittle
sun.java.command=/mnt/software/s/sbt/0.13.8/bin/sbt-launch.jar smrt-server-link/compile
sun.java.launcher=SUN_STANDARD
sun.jnu.encoding=UTF-8
sun.management.compiler=HotSpot 64-Bit Tiered Compilers
sun.os.patch.level=unknown
Tried increasing the heap max upto 32G, same problem happens after a long time with bigger hprof file.
Visual VM shows 100% GC activity and all of the allocated heap is being used.

Not able to restart Jenkins - Memory error

I am getting below error while restarting Jenkins.
INFO: Ready to receive data from Gerrit: Gerrit-QA
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007b1480000, 246939648, 0) failed; error='C
annot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 246939648 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/share/tomcat6/hs_err_pid29653.log
How I resolve this ?

Spring : Illegal access: this web application instance has been stopped already

I am working on a Spring-MVC application in which I am computing the statistics every night. The problem is, yesterdays computation failed and I have this error and an hs_err_something.log file. The file basically says Out of memory error, but our servers have 32GB ram and quite lot of disk space too. Also, the server is kind of relaxed in night. Why I am I getting this error. I will post contents of relevant code.
StatisticsServiceImpl :
#Override
#Scheduled(cron = "0 2 2 * * ?")
public void computeStatisticsForAllUsers() {
// One of the count as part of statistics
int groupNotesCount = this.groupNotesService.getNoteCountForUser(person.getUsername());
}
GroupNotesDAOImpl :
#Override
public int getNoteCountForUser(String noteCreatorEmail) {
Session session = this.sessionFactory.getCurrentSession();
Query query = session.createQuery("select count(*) from GroupNotes as gn where gn.noteCreatorEmail=:noteCreatorEmail");
query.setParameter("noteCreatorEmail", noteCreatorEmail);
return new Integer(String.valueOf(query.uniqueResult()));
}
Error log :
Aug 05, 2015 2:02:02 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load gn. The eventual following stack trace is cause
d by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no function
al impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1612)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
at com.journaldev.spring.dao.GroupNotesDAOImpl.getNoteCountForUser(GroupNotesDAOImpl.java:359)
hs_err.log file :
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 741867520 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2673), pid=20080, tid=140319513569024
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
What should I do. Any help would be nice. Thanks a lot.