I am working on a Spring-MVC application in which I am computing the statistics every night. The problem is, yesterdays computation failed and I have this error and an hs_err_something.log file. The file basically says Out of memory error, but our servers have 32GB ram and quite lot of disk space too. Also, the server is kind of relaxed in night. Why I am I getting this error. I will post contents of relevant code.
StatisticsServiceImpl :
#Override
#Scheduled(cron = "0 2 2 * * ?")
public void computeStatisticsForAllUsers() {
// One of the count as part of statistics
int groupNotesCount = this.groupNotesService.getNoteCountForUser(person.getUsername());
}
GroupNotesDAOImpl :
#Override
public int getNoteCountForUser(String noteCreatorEmail) {
Session session = this.sessionFactory.getCurrentSession();
Query query = session.createQuery("select count(*) from GroupNotes as gn where gn.noteCreatorEmail=:noteCreatorEmail");
query.setParameter("noteCreatorEmail", noteCreatorEmail);
return new Integer(String.valueOf(query.uniqueResult()));
}
Error log :
Aug 05, 2015 2:02:02 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load gn. The eventual following stack trace is cause
d by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no function
al impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1612)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
at com.journaldev.spring.dao.GroupNotesDAOImpl.getNoteCountForUser(GroupNotesDAOImpl.java:359)
hs_err.log file :
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 741867520 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2673), pid=20080, tid=140319513569024
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
What should I do. Any help would be nice. Thanks a lot.
Related
I'm trying to do a simulation in Cooja simulator, Contiki-ng using cooja motes and CSMA and the energest model.
Even though the simulation works well in Z1 mote with cooja motes the simulation stops and I get a null pointer error:
[15:19:20 - Thread-1] [Simulation.java:308] [FATAL] - Simulation stopped due to error: null
java.lang.NullPointerException: null
at org.contikios.cooja.contikimote.ContikiMoteType.getCoreMemory(ContikiMoteType.java:824) ~[cooja.jar:?]
at org.contikios.cooja.contikimote.ContikiMote.execute(ContikiMote.java:149) ~[cooja.jar:?]
at org.contikios.cooja.motes.AbstractWakeupMote$1.execute(AbstractWakeupMote.java:51) ~[cooja.jar:?]
at org.contikios.cooja.Simulation.run(Simulation.java:296) ~[cooja.jar:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
When I try to do the same simulation with TSCH, 6TiSCH 6top and add the energest model I get this error:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f893449310d, pid=723817, tid=724266
#
# JRE version: OpenJDK Runtime Environment 18.9 (11.0.15+10) (build 11.0.15+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM 18.9 (11.0.15+10-LTS, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0xa5710d] Klass::search_secondary_supers(Klass*) const+0xd
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to usr/contiki-ng/tools/cooja/build/core.723817)
#
# If you would like to submit a bug report, please visit:
# https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-11-openjdk
a quick search says it's a bug, but as it only occurs when I add the energy model I'm not sure!
I am getting below error while restarting Jenkins.
INFO: Ready to receive data from Gerrit: Gerrit-QA
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007b1480000, 246939648, 0) failed; error='C
annot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 246939648 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/share/tomcat6/hs_err_pid29653.log
How I resolve this ?
We are periodically runnign jmap -heap command to monitor a tomcat application. However this is first time when we got OutOfMemoryError.
When we monitor the memory at server level there is around 110MB available. Since it is just monitoring can we ignore this time assuming no harm has been done to the application and the server was low on memory when this command run?
$ ./jmap -heap 13511
Attaching to process ID 13511, please wait...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.<init>(LinuxDebuggerLocal.java:210)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupDebuggerLinux(BugSpotAgent.java:816)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupDebugger(BugSpotAgent.java:518)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:39)
UPDATE: We tried to run jmap -heap again and got java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.
So, the problem seems to be at OS resource level.
$ ./jmap -heap 13511
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.
#
# Internal Error (gcTaskThread.cpp:38), pid=13162, tid=1104300352
# Error: Cannot create GC thread. Out of system resources.
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
# An error report file with more information is saved as:
# /data01/home/s618199/JAVA/jdk1.6.0_18/bin/hs_err_pid13162.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
Aborted
This is not a memory problem (even though it is reported as one) but an operating system resource problem: You can not create any more threads. And it seems to affect the jmap program and not Tomcat.
I am trying to upgrade Cassandra 1 to Cassandra 2.. And to do that I upgraded Java (to Java 7) but whenever I execute : cassandra. Its launching like this :
INFO 17:32:41,413 Logging initialized INFO 17:32:41,437 Loading
settings from file:/etc/cassandra/cassandra.yaml INFO 17:32:41,642
Data files directories: [/var/lib/cassandra/data] INFO 17:32:41,643
Commit log directory: /var/lib/cassandra/commitlog INFO 17:32:41,643
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 17:32:41,643 disk_failure_policy is stop INFO 17:32:41,643
commit_failure_policy is stop INFO 17:32:41,647 Global memtable
threshold is enabled at 986MB INFO 17:32:41,727 Not using
multi-threaded compaction INFO 17:32:41,869 JVM vendor/version:
OpenJDK 64-Bit Server VM/1.7.0_55 WARN 17:32:41,869 OpenJDK is not
recommended. Please upgrade to the newest Oracle Java release INFO
17:32:41,869 Heap size: 4137680896/4137680896 INFO 17:32:41,870 Code
Cache Non-heap memory: init = 2555904(2496K) used = 657664(642K)
committed = 2555904(2496K) max = 50331648(49152K) INFO 17:32:41,870
Par Eden Space Heap memory: init = 335544320(327680K) used =
80545080(78657K) committed = 335544320(327680K) max =
335544320(327680K) INFO 17:32:41,870 Par Survivor Space Heap memory:
init = 41943040(40960K) used = 0(0K) committed = 41943040(40960K) max
= 41943040(40960K) INFO 17:32:41,870 CMS Old Gen Heap memory: init = 3760193536(3672064K) used = 0(0K) committed = 3760193536(3672064K) max
= 3760193536(3672064K) INFO 17:32:41,872 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 14994304(14642K) committed =
21757952(21248K) max = 174063616(169984K) INFO 17:32:41,872 Classpath:
/etc/cassandra:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-internal-only-0.3.3.jar:/usr/share/cassandra/apache-cassandra-2.0.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/apache-cassandra-thrift-2.0.8.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.2.5.jar
INFO 17:32:41,873 JNA not found. Native methods will be disabled. INFO
17:32:41,884 Initializing key cache with capacity of 100 MBs. INFO
17:32:41,890 Scheduling key cache save to each 14400 seconds (going to
save all keys). INFO 17:32:41,890 Initializing row cache with capacity
of 0 MBs INFO 17:32:41,895 Scheduling row cache save to each 0 seconds
(going to save all keys). INFO 17:32:41,968 Initializing
system.schema_triggers INFO 17:32:41,985 Initializing
system.compaction_history INFO 17:32:41,988 Initializing
system.batchlog INFO 17:32:41,991 Initializing system.sstable_activity
INFO 17:32:41,994 Initializing system.peer_events INFO 17:32:41,997
Initializing system.compactions_in_progress INFO 17:32:42,000
Initializing system.hints ERROR 17:32:42,001 Exception encountered
during startup java.lang.RuntimeException: Incompatible SSTable found.
Current version jb is unable to read file:
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-2.
Please run upgradesstables. at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:415)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at
org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at
org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at
org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at
org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
java.lang.RuntimeException: Incompatible SSTable found. Current
version jb is unable to read file:
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-2.
Please run upgradesstables. at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:415)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at
org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at
org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at
org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at
org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
Exception encountered during startup: Incompatible SSTable found.
Current version jb is unable to read file:
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-2.
Please run upgradesstables.
When i try to execute : upgradesstables (nodetool upgradesstables -h 127.0.0.1 -u root ...) I got this :
Failed to connect to '127.0.0.1:7000': Connexion refusée
Anyone can help me please ?
Thanks.
The Cassandra error has nothing to do with OpenJDK, although I do recommend using Oracle's.
You need to make sure you're on a valid upgrade path: http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeC_c.html
You can't trivially upgrade from a Cassandra older than 1.2.9 to 2.0, and you can't upgrade from 1.x to 2.1 without first going to 2.0.7 or later.
Suggested upgrade path per documentation: 1.x > 1.2.9 > 2.0.7 > 2.1.x
Your java -version output shows your not using correct JDK. It should be something like below.
Java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
For this tell the system that there's a new Java version available
$ sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_version/bin/java" 1
Set the new JDK as the default using the following command:
$ sudo update-alternatives --config java
You can refer the link https://devopsmanual.in/2018/03/07/how-to-upgrade-datastax-cassandra/ for more details.
We are facing the below problem at our production enviournment in unpredictable manner
sometimes the server is down in a day or sometimes in a week, below is the exact error
dump, below are the settings for the server.
JDK: jdk1.6.0_21
Server: Tomcat 7.0.2
OS: Red Hat Enterprise Linux Server release 5.5
In catalina.sh the following setting has been done:
JAVA_OPTS="-Xms1024M -Xmx1536M -XX:+HeapDumpOnOutOfMemoryError -XX:+AggressiveOpts
-XX:-DisableExplicitGC -XX:AdaptiveSizeThroughPutPolicy=0
-XX:+UsePSAdaptiveSurvivorSizePolicy
-XX:+UseAdaptiveGenerationSizePolicyAtMinorCollection
-XX:+UseAdaptiveGenerationSizePolicyAtMajorCollection -XX:PermSize=768M
-XX:MaxPermSize=768M -XX:+PrintGCDetails -Xloggc:/tmp/gcLogs.txt"
export CATALINA_OPTS="-Dcom.sun.management.jmxremote.port=22222
-Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.password.file=/jakarta-tomcat7/apache-tomcat-7.0.2/conf
/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/jakarta-tomcat7/apache-
tomcat-7.0.2/conf/jmxremote.access"
Error Trace:-
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1958536 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=18658, tid=589781904
# Error: Chunk::new
#
# JRE version: 6.0_21-b06
# Java VM: Java HotSpot(TM) Server VM (17.0-b16 mixed mode linux-x86 )
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x23787400): JavaThread "CompilerThread0" daemon [_thread_in_native, id=18668, stack(0x231f5000,0x23276000)]
Stack: [0x231f5000,0x23276000], sp=0x23272e70, free space=1f723276000k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x6a9262]
V [libjvm.so+0x2b277f]
V [libjvm.so+0x12e03c]
V [libjvm.so+0x12e536]
V [libjvm.so+0x5d67d0]
V [libjvm.so+0x2f809d]
V [libjvm.so+0x4f65a9]
V [libjvm.so+0x27b85f]
V [libjvm.so+0x278043]
V [libjvm.so+0x209767]
V [libjvm.so+0x280f8c]
V [libjvm.so+0x280839]
V [libjvm.so+0x66feb6]
V [libjvm.so+0x66959e]
V [libjvm.so+0x57a89e]
C [libpthread.so.0+0x5832]
Current CompileTask:
C2:3230 ! org.apache.jsp.com.common.press_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (4433 bytes)
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x09a21400 JavaThread "http-8080-exec-904" daemon [_thread_in_native, id=17126, stack(SIGTERM: [libjvm.so+0x57aaf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x57aaf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux Server release 5.5 (Tikanga)
uname:Linux 2.6.18-194.17.1.el5PAE #1 SMP Mon Sep 20 07:34:07 EDT 2010 i686
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 114688, NOFILE 1024, AS infinity
load average:0.39 0.54 0.38
CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 6228576k(225096k free), swap 6974456k(6974352k free)
vm_info: Java HotSpot(TM) Server VM (17.0-b16) for linux-x86 JRE (1.6.0_21-b06), built on Jun 22 2010 01:04:46 by "java_re" with gcc 3.2.1-7a (J2SE release)
time: Fri Dec 10 14:01:06 2010
elapsed time: 79552 seconds
Thanks in advance,
Amit
For the record (and Google), this looks like both these bugs, which were fixed in the 1.6u22 release of sun's jdk. So, first thing to do is update your JVM. If it still happens, and always happens on a particular method, you can exclude that method from compilation (as long as you're aware of the performance implications of that) with the following jvm flag:
-XX:CompileCommand=exclude,org/apache/velocity/runtime/directive/Foreach,render
(as suggested here). But, first update your jvm.
You're running with a lot of JVM args that affect memory. Have you tried empirically removing each option to see which one is causing the OOM? This particular OOM is not coming from the Java heap, it's coming from the JVM's own C heap.
As stated by the other answers / comments, you are running out of memory. Given your JVM settings, I'd say that the root cause is 99% likely to be a memory leak.
If you have been doing a lot of hot loading in the Tomcat instance, this could just be caused by that. Hot loading is notorious for leaking memory, and there is not much you can do about it in practice ... except exit and restart your Tomcat more often.
The other possibility is that your application is leaking memory. If this is the case then you will need to use a memory profiler to track down the leak.
The fact that the OOME caused a JVM crash is interesting, but probably not significant. (It looks like the JVM was trying to JIT compile a class generated from a JSP when it ran out of memory. The chunk being requested is rather large, but that probably means you have a rather large / complicated JSP.)