Develop Reference

Tomcat 8 when stated in ec2 machine

This is my scenario:
SWAP Memory avaiable: 4GB
Linux
I'm getting these warnings:
Java HotSpot(TM) 64-Bit ServerVM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000540000000, 7158628352, 0) failed; error='Cannot allocate memory' (errno=12)
Also this error:
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 7158628352 bytes for committing reserved memory.
An error report file with more information is saved as:
/opt/apache-tomcat-8.0.28/hs_err_pid8656.log
ram memory in my machine:
total used free shared buffers cached
Mem: 7986 7839 147 0 94 1390
-/+ buffers/cache: 6354 1632
Swap: 4095 3 4092
WARNING: An attempt was made to authenticate the locked user "root"..

SBT compile always gives " GC overhead limit exceeded error" irrespective of the heap size

sbt 0.13.8 compile takes a very long time (> 25 mins) and finally errors out with GC overhead limit
[info] Compiling 323 Scala sources and 1 Java source to .../target/scala-2.12/classes ...
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to .../java_pid10265.hprof ...
Heap dump file created [7078096837 bytes in 71.966 secs]
[error] ## Exception when compiling 324 sources to .../target/scala-2.12/classes
[error] GC overhead limit exceeded
[error] scala.reflect.internal.util.HashSet.growTable(HashSet.scala:97)
[error] scala.reflect.internal.util.HashSet.addEntry(HashSet.scala:63)
...
[addJava] arg = '-Xmx4G'
[addJava] arg = '-Xms1024m'
[addJava] arg = '-XX:ReservedCodeCacheSize=512m'
[addJava] arg = '-XX:MaxMetaspaceSize=2G'
[addJava] arg = '-XX:+PrintFlagsFinal'
[addJava] arg = '-XX:+HeapDumpOnOutOfMemoryError'
[addJava] arg = '-XX:HeapDumpPath=....
Information from hprof file:
Basic info:
Date taken: Wed Mar 20 15:04:26 PDT 2019
File: .../java_pid10265.hprof
File size: 6,750.2 MB
Total bytes: 6,033,077,813
Total classes: 22,538
Total instances: 116,328,564
Classloaders: 163
GC roots: 3,101
Number of objects pending for finalization: 0
Heap dumped on OutOfMemoryError exception
Thread causing OutOfMemoryError exception: pool-6-thread-8
Environment:
OS: Linux (3.10.0-957.5.1.el7.x86_64)
Architecture: amd64 64bit
Java Home: /mnt/software/j/jdk/1.8.0_144/jre
Java Version: 1.8.0_144
JVM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01, mixed mode)
Java Vendor: Oracle Corporation
System properties:
awt.toolkit=sun.awt.X11.XToolkit
com.sun.management.jmxremote.authenticate=false
com.sun.management.jmxremote.port=9933
com.sun.management.jmxremote.ssl=false
file.encoding=UTF-8
file.encoding.pkg=sun.io
file.separator=/
java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment
java.awt.printerjob=sun.print.PSPrinterJob
java.class.path=/mnt/software/s/sbt/0.13.8/bin/sbt-launch.jar
java.class.version=52.0
java.endorsed.dirs=/mnt/software/j/jdk/1.8.0_144/jre/lib/endorsed
java.ext.dirs=/mnt/software/j/jdk/1.8.0_144/jre/lib/ext:/usr/java/packages/lib/ext
java.home=/mnt/software/j/jdk/1.8.0_144/jre
java.io.tmpdir=/tmp
java.library.path=/mnt/software/j/jdk/1.8.0_144/jre/lib/amd64/server:/mnt/software/h/htslib/1.9/lib:/mnt/software/z/zlib/1.2.11/lib:/mnt/software/g/gcc/6.4.0/libc-2.5/compat/lib:/mnt/software/g/gcc/6.4.0/libc-2.5/lib64:/mnt/software/j/jdk/1.8.0_71/jre/lib/amd64/server:/mnt/software/o/openblas/0.2.14/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
java.rmi.server.randomIDs=true
java.runtime.name=Java(TM) SE Runtime Environment
java.runtime.version=1.8.0_144-b01
java.specification.name=Java Platform API Specification
java.specification.vendor=Oracle Corporation
java.specification.version=1.8
java.vendor=Oracle Corporation
java.vendor.url=http://java.oracle.com/
java.vendor.url.bug=http://bugreport.sun.com/bugreport/
java.version=1.8.0_144
java.vm.info=mixed mode
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name=Java Virtual Machine Specification
java.vm.specification.vendor=Oracle Corporation
java.vm.specification.version=1.8
java.vm.vendor=Oracle Corporation
java.vm.version=25.144-b01
jline.esc.timeout=0
jline.shutdownhook=false
jna.loaded=true
jna.nosys=true
jna.platform.library.path=/usr/lib64:/lib64:/usr/lib:/lib:/usr/lib/vmware-tools/lib64/libvmGuestLibJava.so:/usr/lib/vmware-tools/lib32/libvmGuestLibJava.so:/usr/lib/vmware-tools/lib64/libvmGuestLib.so:/usr/lib/vmware-tools/lib32/libvmGuestLib.so:/usr/lib64/mysql:/usr/lib/vmware-tools/lib64/libDeployPkg.so:/usr/lib/vmware-tools/lib32/libDeployPkg.so
jnidispatch.path=/tmp/jna--845103277/jna1647140300662886831.tmp
line.separator=\n
os.arch=amd64
os.name=Linux
os.version=3.10.0-957.5.1.el7.x86_64
path.separator=:
sbt.log.noformat=true
sun.arch.data.model=64
sun.boot.class.path=/mnt/software/j/jdk/1.8.0_144/jre/lib/resources.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/rt.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/sunrsasign.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/jsse.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/jce.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/charsets.jar:/mnt/software/j/jdk/1.8.0_144/jre/lib/jfr.jar:/mnt/software/j/jdk/1.8.0_144/jre/classes
sun.boot.library.path=/mnt/software/j/jdk/1.8.0_144/jre/lib/amd64
sun.cpu.endian=little
sun.cpu.isalist=
sun.io.unicode.encoding=UnicodeLittle
sun.java.command=/mnt/software/s/sbt/0.13.8/bin/sbt-launch.jar smrt-server-link/compile
sun.java.launcher=SUN_STANDARD
sun.jnu.encoding=UTF-8
sun.management.compiler=HotSpot 64-Bit Tiered Compilers
sun.os.patch.level=unknown
Tried increasing the heap max upto 32G, same problem happens after a long time with bigger hprof file.
Visual VM shows 100% GC activity and all of the allocated heap is being used.

Java jmap -heap command gave OutOfMemoryError

We are periodically runnign jmap -heap command to monitor a tomcat application. However this is first time when we got OutOfMemoryError.
When we monitor the memory at server level there is around 110MB available. Since it is just monitoring can we ignore this time assuming no harm has been done to the application and the server was low on memory when this command run?
$ ./jmap -heap 13511
Attaching to process ID 13511, please wait...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.<init>(LinuxDebuggerLocal.java:210)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupDebuggerLinux(BugSpotAgent.java:816)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupDebugger(BugSpotAgent.java:518)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:39)
UPDATE: We tried to run jmap -heap again and got java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.
So, the problem seems to be at OS resource level.
$ ./jmap -heap 13511
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.
#
# Internal Error (gcTaskThread.cpp:38), pid=13162, tid=1104300352
# Error: Cannot create GC thread. Out of system resources.
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
# An error report file with more information is saved as:
# /data01/home/s618199/JAVA/jdk1.6.0_18/bin/hs_err_pid13162.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
Aborted

This is not a memory problem (even though it is reported as one) but an operating system resource problem: You can not create any more threads. And it seems to affect the jmap program and not Tomcat.

Spring : Illegal access: this web application instance has been stopped already

I am working on a Spring-MVC application in which I am computing the statistics every night. The problem is, yesterdays computation failed and I have this error and an hs_err_something.log file. The file basically says Out of memory error, but our servers have 32GB ram and quite lot of disk space too. Also, the server is kind of relaxed in night. Why I am I getting this error. I will post contents of relevant code.
StatisticsServiceImpl :
#Override
#Scheduled(cron = "0 2 2 * * ?")
public void computeStatisticsForAllUsers() {
// One of the count as part of statistics
int groupNotesCount = this.groupNotesService.getNoteCountForUser(person.getUsername());
}
GroupNotesDAOImpl :
#Override
public int getNoteCountForUser(String noteCreatorEmail) {
Session session = this.sessionFactory.getCurrentSession();
Query query = session.createQuery("select count(*) from GroupNotes as gn where gn.noteCreatorEmail=:noteCreatorEmail");
query.setParameter("noteCreatorEmail", noteCreatorEmail);
return new Integer(String.valueOf(query.uniqueResult()));
}
Error log :
Aug 05, 2015 2:02:02 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load gn. The eventual following stack trace is cause
d by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no function
al impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1612)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
at com.journaldev.spring.dao.GroupNotesDAOImpl.getNoteCountForUser(GroupNotesDAOImpl.java:359)
hs_err.log file :
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 741867520 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2673), pid=20080, tid=140319513569024
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
What should I do. Any help would be nice. Thanks a lot.

Cassandra 1 to Cassandra 2

I am trying to upgrade Cassandra 1 to Cassandra 2.. And to do that I upgraded Java (to Java 7) but whenever I execute : cassandra. Its launching like this :
INFO 17:32:41,413 Logging initialized INFO 17:32:41,437 Loading
settings from file:/etc/cassandra/cassandra.yaml INFO 17:32:41,642
Data files directories: [/var/lib/cassandra/data] INFO 17:32:41,643
Commit log directory: /var/lib/cassandra/commitlog INFO 17:32:41,643
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 17:32:41,643 disk_failure_policy is stop INFO 17:32:41,643
commit_failure_policy is stop INFO 17:32:41,647 Global memtable
threshold is enabled at 986MB INFO 17:32:41,727 Not using
multi-threaded compaction INFO 17:32:41,869 JVM vendor/version:
OpenJDK 64-Bit Server VM/1.7.0_55 WARN 17:32:41,869 OpenJDK is not
recommended. Please upgrade to the newest Oracle Java release INFO
17:32:41,869 Heap size: 4137680896/4137680896 INFO 17:32:41,870 Code
Cache Non-heap memory: init = 2555904(2496K) used = 657664(642K)
committed = 2555904(2496K) max = 50331648(49152K) INFO 17:32:41,870
Par Eden Space Heap memory: init = 335544320(327680K) used =
80545080(78657K) committed = 335544320(327680K) max =
335544320(327680K) INFO 17:32:41,870 Par Survivor Space Heap memory:
init = 41943040(40960K) used = 0(0K) committed = 41943040(40960K) max
= 41943040(40960K) INFO 17:32:41,870 CMS Old Gen Heap memory: init = 3760193536(3672064K) used = 0(0K) committed = 3760193536(3672064K) max
= 3760193536(3672064K) INFO 17:32:41,872 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 14994304(14642K) committed =
21757952(21248K) max = 174063616(169984K) INFO 17:32:41,872 Classpath:
/etc/cassandra:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-internal-only-0.3.3.jar:/usr/share/cassandra/apache-cassandra-2.0.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/apache-cassandra-thrift-2.0.8.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.2.5.jar
INFO 17:32:41,873 JNA not found. Native methods will be disabled. INFO
17:32:41,884 Initializing key cache with capacity of 100 MBs. INFO
17:32:41,890 Scheduling key cache save to each 14400 seconds (going to
save all keys). INFO 17:32:41,890 Initializing row cache with capacity
of 0 MBs INFO 17:32:41,895 Scheduling row cache save to each 0 seconds
(going to save all keys). INFO 17:32:41,968 Initializing
system.schema_triggers INFO 17:32:41,985 Initializing
system.compaction_history INFO 17:32:41,988 Initializing
system.batchlog INFO 17:32:41,991 Initializing system.sstable_activity
INFO 17:32:41,994 Initializing system.peer_events INFO 17:32:41,997
Initializing system.compactions_in_progress INFO 17:32:42,000
Initializing system.hints ERROR 17:32:42,001 Exception encountered
during startup java.lang.RuntimeException: Incompatible SSTable found.
Current version jb is unable to read file:
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-2.
Please run upgradesstables. at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:415)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at
org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at
org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at
org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at
org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
java.lang.RuntimeException: Incompatible SSTable found. Current
version jb is unable to read file:
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-2.
Please run upgradesstables. at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:415)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at
org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at
org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at
org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at
org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
Exception encountered during startup: Incompatible SSTable found.
Current version jb is unable to read file:
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-2.
Please run upgradesstables.
When i try to execute : upgradesstables (nodetool upgradesstables -h 127.0.0.1 -u root ...) I got this :
Failed to connect to '127.0.0.1:7000': Connexion refusée
Anyone can help me please ?
Thanks.

The Cassandra error has nothing to do with OpenJDK, although I do recommend using Oracle's.
You need to make sure you're on a valid upgrade path: http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeC_c.html
You can't trivially upgrade from a Cassandra older than 1.2.9 to 2.0, and you can't upgrade from 1.x to 2.1 without first going to 2.0.7 or later.
Suggested upgrade path per documentation: 1.x > 1.2.9 > 2.0.7 > 2.1.x

Your java -version output shows your not using correct JDK. It should be something like below.
Java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
For this tell the system that there's a new Java version available
$ sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_version/bin/java" 1
Set the new JDK as the default using the following command:
$ sudo update-alternatives --config java
You can refer the link https://devopsmanual.in/2018/03/07/how-to-upgrade-datastax-cassandra/ for more details.