in my company we develope an enterprise web application in Spring (Flex frontend) and deploy this app to our customers in Tomcat 6 in SAAS style.
Recently we are suffering from (seemingly) random OutOfMemory errors so after investigation I knew that we should inspect the memory dump of the JVM in the moment of the error.
The JVM we use is 1.6.18 and the Tomcat version is Tomcat 7.0.23 under Windows Server 2008.
I added the parameter -XX:+HeapDumpOnOutOfMemoryError in the Tomcat monitor panel (under Java tab) but the machines aren't producing any dump.
The full java options are set as follows on the server we are investigating:
-Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 7.0
-Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 7.0
-Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 7.0\endorsed
-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 7.0\temp
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 7.0\conf\logging.properties
-XX:PermSize=128m
-XX:MaxPermSize=1024m
-Xms1024m
-Xmx6144m
-XX:+HeapDumpOnOutOfMemoryError
-Dcom.sun.management.jmxremote.port=3333
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
As you can see the max heap size is pretty large (6 Gigs) since out app is particurarly heavy.
I added for testing reasons the jmx parameters in order to view the JVM status in real time with VisualVM, but nothig occurred while I was looking at it.
All that the stderr shows is this:
java.lang.OutOfMemoryError
Nothing before it (except other errors that I don't know if correlated, but they are logged hours or several minutes before the OOM) and nothing after it.
It seems strange to me that it doesn't report the memory section (Java heap size, or Permgen Space).
Maybe the memory heap reserved is too big and the JVM can't write it at since it's in a critical state (OOM)?
Maybe the JMX parameters are messing things up?
This is the VisualVM screenshot of currently running JVM:
UPDATE:
I installed on another tomcat installation (same version as the one in which the problem originated and same java options) the same web application modified to manually provoke an OOM by undefinitely filling an arraylist in an infinite loop.
The test proved that the java option -XX:+HeapDumpOnOutOfMemoryError works, since a memory dump was produced after I provoked the OOM; in this case however the error was:
java.lang.OutOfMemoryError: Java Heap space
following with the stack trace.
So it seems that the problem is not with my JVM args notation but with the particular kind of error I'm encountering.
Another peculiarity is that after throwing the OOM error in my test, the app continued to work in Tomcat. The OOM in my original issue on the other hand causes the stop of the Tomcat service.
Sadly without a stack trace taken a moment before it stops it looks difficult to further investigate. :(
Add -XX:HeapDumpPath="/some/path/dump.out" to your JVM args explicitly set the heap dump file location
Related
I have a DSpace server, after uprading from version 4.3 to 5.1, it constantly stops working after some hours with following error:
java.lang.OutOfMemoryError: Java heap space
I've read answers to the same questions, but they didn't help me.
My JAVA_OPTS: -Xmx4096M -Xms4096M -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8
Here are my logs:
dspace.log.2015-04-21: http://jpst.it/ylU4
tomcat7-stderr.2015-04-20.log: http://jpst.it/ylTN
localhost_access_log.2015-04-20.txt: http://jpst.it/ylTT
I searched the internet for a solution and could not find one, does any one have a clue?
Your tomcat log file quite clearly says that it isn't using the JAVA_OPTS you're passing in:
INFO: Command line argument: -Xmx256m
In my experience, DSpace 5 needs at least 512MB; your 4GB look a little excessive unless your DSpace instance is very big / gets lots of traffic.
You aren't telling us what operating system you're using, so it's hard to give advice on how where to set these properly. Start with your tomcat startup script and/or the tomcat configuration files. On my Red Hat Enterprise Linux 6.6 boxes (with tomcat6), the correct place for the memory settings is /etc/tomcat6/tomcat6.conf.
I have some memory leak issue in my web app which is deployed in tomcat. To find the root cause I enabled the HeapDumpOnOutOfMemory error by setting:
-XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs
and the memory settings in the tomcat is:
-Xms256m -Xmx768m -XX:PermSize=128m -XX:MaxPermSize=256m
When the out of memory issue happened, I see
java.lang.OutOfMemoryError: Java heap space
on the tomcat log file, but the .hprof file is not generated. Am I missing some settings here?
As #beny23 wrote you should use -XX:+HeapDumpOnOutOfMemoryError
and as is stated here:
The -XX:HeapDumpOnOutOfMemoryError Option This option tells the Java
HotSpot VM to generate a heap dump when an allocation from the Java
heap or the permanent generation cannot be satisfied. There is no
overhead in running with this option, so it can be useful for
production systems where the OutOfMemoryError exception takes a long
time to surface.
Check also your Java version since this option was introduced in 1.4.2 update 12, 5.0 update 7.
i am getting error in stopping tomcat service in server using putty
error is
`[root#vps ~]# service tomcat7 stop
Shutting down Tomcat7:
Using CATALINA_BASE: /usr/tomcat7
Using CATALINA_HOME: /usr/tomcat7
Using CATALINA_TMPDIR: /usr/tomcat7/temp
Using JRE_HOME: /usr
Using CLASSPATH: /usr/tomcat7/bin/bootstrap.jar
/usr/tomcat7/bin/tomcat-juli.jar
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
`
please help ...
Try increasing the heap memory usage for java.
You can do so by editing catalina.bat (for Windows) and catalina.sh (for Linux).
Try editing JAVA_OPTS.
For Linux
export JAVA_OPTS="-server -Xmx512m"
For Windows
set JAVA_OPTS=-server -Xmx512m
You need to look at your memory settings for tomcat, to help set heap size look at
Increase Tomcat memory settings
Take a look at the question about memory size
Could not reserve enough space for object heap
My Web application is running on tomcat apache-tomcat-6.0.35 and It was working fine witout any issue.But today I show bellow exception in catalina.out log and was unable to login to the system.
Jul 9, 2013 2:40:15 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
I was under impression this exception can be ocuured when you deploy or redeploy a application. But I got this when the application is running without any issues. What could be the reason for this. Kindly advice
Each webapp in Tomcat has a separate class loader with a separate set of loaded classes. When we start Tomcat it does not load all classes of all its apps at once but rather it loads the apps lazily. When an app becomes active it starts loading its classes and it may happen that JVM runs out permgen space. That is, it may happen even without redeploying apps.
add these Optionsto JVM command line when Tomcat is started Or add to IDE VM Options
-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
OR
-XX:MaxPermGen=128M
It sounds like one of your webapps is consuming a lot of memory. Have you tried raising the PermGen size? In $CATALINA_HOME/bin/catalina.sh you can add
-XX:PermSize=512m -XX:MaxPermSize=512m
to your JAVA_OPTS so that it allocates enough space to run.
You are getting out of permGen space, try to change this parameters on your tomcat start script
/etc/init.d/tomcat6
JAVA_OPTS="-Djava.awt.headless=true -XX:MaxPermSize=512M"
Set the memory amount as you need.
It definitely is an error that often shows up during deployments. I've seen it occur any number of times in both Grails and Java applications. You can increase your permgen space by adjusting the JVM startup param -XX:MaxPermSize, ie:
-XX:MaxPermSize=256m
This thread covers the issue in detail: Thread
I had this error a few days ago. I fixed it by restarting the real machine - there was a problem with a deploy.
Also you can fix it by adding
JAVA_OPTS="-Djava.awt.headless=true -Dfile.encoding=UTF-8
-server -Xms1536m -Xmx1536m
-XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m
-XX:MaxPermSize=256m -XX:+DisableExplicitGC"
See possible fix
I am running a web application using Tomcat 6. We are using open JDK 6. Ours is AMD 64bit Ubuntu 11.04 server.
Memory dump is taken by this command
jmap -dump:format=b,file=/home/demon/Desktop/mymemorydump.hprof 2762
Memory dump is of 4.5 GB. When we were analyzing it using eclipse MAT, Strong Reachable objects were 80 MB, Unreachable objects were 335 MB..
How to analyze the rest of the dump..
Also memory taken on my server by this process is 4.5 GB. Now this memory never goes down to normal level of 1 GB. And this memory slowly increases to 8-9 GB, then our tomcat stops responding(Actully there is no out of memroy error).
My tomcat config is
/usr/lib/jvm/java-6-openjdk/bin/java -Djava.util.logging.config.file=/var/lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -Xms1024m -Xmx6144m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp org.apache.catalina.startup.Bootstrap start
You may have a Thread leak, having created an excessive number of threads which never terminate. Each thread allocates stack space, which isn't shown above. Pressing Ctrl-\ (unix) (or Ctrl-Break in windows) will dump a stack trace of every thread (you may need to redirect tomcat's STDERR to avoid it scrolling off your terminal). If you didn't start tomcat from the terminal, then use "kill -QUIT YourPID" rather than Ctrl-.
Also, jconsole (in the jdk bin directory) can attach to your executable, before it stops responding. It might be able to show you a leak of threads (or heap) before things get out of hand.