Develop Reference

java process cannot be started

Background: Since the past few days, my linux development machine Java services one by one by the system kill, I looked at the system logs are OOM caused. Now I can't start the java process if I set the initial heap too large.
I can't see with the usual troubleshooting means, the development machine is a virtual machine (I don't think I can exclude the problem of the physical machine where the virtual machine is located, my colleague's machine and I applied at the same time, also have this problem), the total memory is about 6G, buff/cache + free total of about 5G. Thank you all for your help.
The crash logs at startup are in the attachment, and the system information and jdk information are in there.
enter link description here
Start-up log:
[~ jdk1.8.0_281]$java -Xms1000m -version
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000a5400000, 699400192, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 699400192 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid7617.log
Memory usage is as follows:
[~ jdk1.8.0_281]$free -h
total used free shared buff/cache available
Mem: 5.7G 502M 213M 4.6G 5.0G 328M
Swap: 0B 0B 0B
The io situation is as follows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.03 2.77 0.84 2.80 48.95 97.58 80.57 0.05 22.14 65.48 9.22 0.55 0.20
scd0 0.00 0.00 0.00 0.00 0.01 0.00 66.96 0.00 0.34 0.34 0.00 0.24 0.00

Tomcat 8 when stated in ec2 machine

This is my scenario:
SWAP Memory avaiable: 4GB
Linux
I'm getting these warnings:
Java HotSpot(TM) 64-Bit ServerVM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000540000000, 7158628352, 0) failed; error='Cannot allocate memory' (errno=12)
Also this error:
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 7158628352 bytes for committing reserved memory.
An error report file with more information is saved as:
/opt/apache-tomcat-8.0.28/hs_err_pid8656.log
ram memory in my machine:
total used free shared buffers cached
Mem: 7986 7839 147 0 94 1390
-/+ buffers/cache: 6354 1632
Swap: 4095 3 4092
WARNING: An attempt was made to authenticate the locked user "root"..

Eclipse Insufficient memory for the Java Runtime Environment

I have reached a problem where java virtual machine simply does not have enough memory to compile. What should I do about this?
The jvm is run by eclipse and it is already part of the 64 bit server. I am running approx 600 lines of code in one class with dependency on a few other classes with data read and saved in text file.
The error report is shown below
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (malloc) failed to allocate 1048576 bytes for
AllocateHeap
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.
Out of Memory Error (memory/allocation.inline.hpp:61), pid=25196,
tid=0x0000000000006218
JRE version: (8.0_144-b01) (build )
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode windows-
amd64 compressed oops)
Failed to write core dump. Minidumps are not enabled by default on client
versions of Windows
--------------- T H R E A D ---------------
Current thread (0x00000000023a0800): JavaThread "Unknown thread"
[_thread_in_vm, id=25112, stack(0x00000000021b0000,0x00000000022b0000)]
Stack: [0x00000000021b0000,0x00000000022b0000]
[error occurred during error reporting (printing stack bounds), id
0xc0000005]
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
Other Threads:
=>0x00000000023a0800 (exited) JavaThread "Unknown thread" [_thread_in_vm,
id=25112, stack(0x00000000021b0000,0x00000000022b0000)]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap:
PSYoungGen total 37888K, used 655K [0x00000000d6400000,
0x00000000d8e00000, 0x0000000100000000)
eden space 32768K, 2% used
[0x00000000d6400000,0x00000000d64a3d80,0x00000000d8400000)
from space 5120K, 0% used
[0x00000000d8900000,0x00000000d8900000,0x00000000d8e00000)
to space 5120K, 0% used
[0x00000000d8400000,0x00000000d8400000,0x00000000d8900000)
ParOldGen total 86016K, used 0K [0x0000000082c00000,
0x0000000088000000, 0x00000000d6400000)
object space 86016K, 0% used
[0x0000000082c00000,0x0000000082c00000,0x0000000088000000)
Metaspace used 746K, capacity 4480K, committed 4480K, reserved
1056768K
class space used 75K, capacity 384K, committed 384K, reserved 1048576K
Card table byte_map: [0x0000000011860000,0x0000000011c50000] byte_map_base:
0x000000001144a000
Marking Bits: (ParMarkBitMap*) 0x000000006bb9d850
Begin Bits: [0x00000000122f0000, 0x0000000014240000)
End Bits: [0x0000000014240000, 0x0000000016190000)
Polling page: 0x0000000000630000
CodeCache: size=245760Kb used=328Kb max_used=328Kb free=245431Kb
bounds [0x00000000024a0000, 0x0000000002710000, 0x00000000114a0000]
total_blobs=58 nmethods=0 adapters=38
compilation: enabled
Compilation events (0 events):
No events
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (0 events):
No events
Events (10 events):
Event: 0.310 loading class java/lang/Short
Event: 0.311 loading class java/lang/Short done
Event: 0.311 loading class java/lang/Integer
Event: 0.312 loading class java/lang/Integer done
Event: 0.312 loading class java/lang/Long
Event: 0.313 loading class java/lang/Long done
Event: 0.320 loading class java/lang/NullPointerException
Event: 0.320 loading class java/lang/NullPointerException done
Event: 0.320 loading class java/lang/ArithmeticException
Event: 0.320 loading class java/lang/ArithmeticException done
--------------- S Y S T E M ---------------
OS: Windows 10.0 , 64 bit Build 14393 (10.0.14393.1198)
CPU:total 4 (initial active 4) (2 cores per cpu, 2 threads per core) family
6 model 78 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1,
sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref, lzcnt, ht, tsc,
tscinvbit, bmi1, bmi2, adx
Memory: 4k page, physical 8204552k(3151664k free), swap 33370376k(6360k
free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.144-b01) for windows-amd64
JRE (1.8.0_144-b01), built on Jul 21 2017 21:57:33 by "java_re" with MS VC++
10.0 (VS2010)
time: Sat Nov 04 21:20:44 2017
elapsed time: 0 seconds (0d 0h 0m 0s)

Spring : Illegal access: this web application instance has been stopped already

I am working on a Spring-MVC application in which I am computing the statistics every night. The problem is, yesterdays computation failed and I have this error and an hs_err_something.log file. The file basically says Out of memory error, but our servers have 32GB ram and quite lot of disk space too. Also, the server is kind of relaxed in night. Why I am I getting this error. I will post contents of relevant code.
StatisticsServiceImpl :
#Override
#Scheduled(cron = "0 2 2 * * ?")
public void computeStatisticsForAllUsers() {
// One of the count as part of statistics
int groupNotesCount = this.groupNotesService.getNoteCountForUser(person.getUsername());
}
GroupNotesDAOImpl :
#Override
public int getNoteCountForUser(String noteCreatorEmail) {
Session session = this.sessionFactory.getCurrentSession();
Query query = session.createQuery("select count(*) from GroupNotes as gn where gn.noteCreatorEmail=:noteCreatorEmail");
query.setParameter("noteCreatorEmail", noteCreatorEmail);
return new Integer(String.valueOf(query.uniqueResult()));
}
Error log :
Aug 05, 2015 2:02:02 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load gn. The eventual following stack trace is cause
d by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no function
al impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1612)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
at com.journaldev.spring.dao.GroupNotesDAOImpl.getNoteCountForUser(GroupNotesDAOImpl.java:359)
hs_err.log file :
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 741867520 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2673), pid=20080, tid=140319513569024
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
What should I do. Any help would be nice. Thanks a lot.

java.lang.OutOfMemoryError: requested 1958536 bytes for Chunk::new. Out of swap space

We are facing the below problem at our production enviournment in unpredictable manner
sometimes the server is down in a day or sometimes in a week, below is the exact error
dump, below are the settings for the server.
JDK: jdk1.6.0_21
Server: Tomcat 7.0.2
OS: Red Hat Enterprise Linux Server release 5.5
In catalina.sh the following setting has been done:
JAVA_OPTS="-Xms1024M -Xmx1536M -XX:+HeapDumpOnOutOfMemoryError -XX:+AggressiveOpts
-XX:-DisableExplicitGC -XX:AdaptiveSizeThroughPutPolicy=0
-XX:+UsePSAdaptiveSurvivorSizePolicy
-XX:+UseAdaptiveGenerationSizePolicyAtMinorCollection
-XX:+UseAdaptiveGenerationSizePolicyAtMajorCollection -XX:PermSize=768M
-XX:MaxPermSize=768M -XX:+PrintGCDetails -Xloggc:/tmp/gcLogs.txt"
export CATALINA_OPTS="-Dcom.sun.management.jmxremote.port=22222
-Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.password.file=/jakarta-tomcat7/apache-tomcat-7.0.2/conf
/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/jakarta-tomcat7/apache-
tomcat-7.0.2/conf/jmxremote.access"
Error Trace:-
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1958536 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=18658, tid=589781904
# Error: Chunk::new
#
# JRE version: 6.0_21-b06
# Java VM: Java HotSpot(TM) Server VM (17.0-b16 mixed mode linux-x86 )
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x23787400): JavaThread "CompilerThread0" daemon [_thread_in_native, id=18668, stack(0x231f5000,0x23276000)]
Stack: [0x231f5000,0x23276000], sp=0x23272e70, free space=1f723276000k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x6a9262]
V [libjvm.so+0x2b277f]
V [libjvm.so+0x12e03c]
V [libjvm.so+0x12e536]
V [libjvm.so+0x5d67d0]
V [libjvm.so+0x2f809d]
V [libjvm.so+0x4f65a9]
V [libjvm.so+0x27b85f]
V [libjvm.so+0x278043]
V [libjvm.so+0x209767]
V [libjvm.so+0x280f8c]
V [libjvm.so+0x280839]
V [libjvm.so+0x66feb6]
V [libjvm.so+0x66959e]
V [libjvm.so+0x57a89e]
C [libpthread.so.0+0x5832]
Current CompileTask:
C2:3230 ! org.apache.jsp.com.common.press_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (4433 bytes)
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x09a21400 JavaThread "http-8080-exec-904" daemon [_thread_in_native, id=17126, stack(SIGTERM: [libjvm.so+0x57aaf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x57aaf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux Server release 5.5 (Tikanga)
uname:Linux 2.6.18-194.17.1.el5PAE #1 SMP Mon Sep 20 07:34:07 EDT 2010 i686
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 114688, NOFILE 1024, AS infinity
load average:0.39 0.54 0.38
CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 6228576k(225096k free), swap 6974456k(6974352k free)
vm_info: Java HotSpot(TM) Server VM (17.0-b16) for linux-x86 JRE (1.6.0_21-b06), built on Jun 22 2010 01:04:46 by "java_re" with gcc 3.2.1-7a (J2SE release)
time: Fri Dec 10 14:01:06 2010
elapsed time: 79552 seconds
Thanks in advance,
Amit

For the record (and Google), this looks like both these bugs, which were fixed in the 1.6u22 release of sun's jdk. So, first thing to do is update your JVM. If it still happens, and always happens on a particular method, you can exclude that method from compilation (as long as you're aware of the performance implications of that) with the following jvm flag:
-XX:CompileCommand=exclude,org/apache/velocity/runtime/directive/Foreach,render
(as suggested here). But, first update your jvm.

You're running with a lot of JVM args that affect memory. Have you tried empirically removing each option to see which one is causing the OOM? This particular OOM is not coming from the Java heap, it's coming from the JVM's own C heap.

As stated by the other answers / comments, you are running out of memory. Given your JVM settings, I'd say that the root cause is 99% likely to be a memory leak.
If you have been doing a lot of hot loading in the Tomcat instance, this could just be caused by that. Hot loading is notorious for leaking memory, and there is not much you can do about it in practice ... except exit and restart your Tomcat more often.
The other possibility is that your application is leaking memory. If this is the case then you will need to use a memory profiler to track down the leak.
The fact that the OOME caused a JVM crash is interesting, but probably not significant. (It looks like the JVM was trying to JIT compile a class generated from a JSP when it ran out of memory. The chunk being requested is rather large, but that probably means you have a rather large / complicated JSP.)