This question already has answers here:
Closed 12 years ago.
I am new to Java and given the task to fix a bug and the issue is as follows. It would be really great if you give suggestions/ideas what is this issue and how can I fix this.:
HTTP Status 500 -
--------------------------------------------------------------------------------
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
org.apache.jasper.JasperException
org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:453)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:375)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.netbeans.modules.web.monitor.server.MonitorFilter.doFilter(MonitorFilter.java:368)
root cause
javax.servlet.ServletException
org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:858)
org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:791)
org.apache.jsp.CustMaint.Jsp.ProfProfileDetails_jsp._jspService(ProfProfileDetails_jsp.java:4016)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.netbeans.modules.web.monitor.server.MonitorFilter.doFilter(MonitorFilter.java:368)
root cause
java.lang.OutOfMemoryError
note The full stack trace of the root cause is available in the Apache Tomcat/5.5.17 logs.
--------------------------------------------------------------------------------
Apache Tomcat/5.5.17
Here's what the Tomcat guys have to say:
An Out Of Memory can be thrown by
several causes:
A servlet trying to load a several
GBytes file into memory will surely
kill the server. These kind of errors
must be considered a simple bug in our
program.
To compensate for the data
your servlet tries to load, you
increase the heap size so that there
is no room to create the stack size
for the threads that need to be
created. The memory required by each
thread will vary by OS but can be as
high as 2M by default and in some OS's
(like Debian Sarge) is not reducible
with the -Xss parameter. 1 Rule of
Thumb, use no more than 1G for heap
space in a 32-bit web application.
Deep recursive algorithms can also
lead to Out Of Memory problems. In
this case, the only fixes are
increasing the thread stack size
(-Xss), or refactoring the algorithms
to reduce the depth, or the local data
size per call.
A webapp that uses lots
of libraries with many dependencies,
or a server maintaining lots of
webapps could exhauste the JVM PermGen
space. This space is where the VM
stores the classes and methods data.
In those cases, the fix is to increase
this size. The Sun VM has the flag
-XX:MaxPermSize that allows to set its size (the default value is 64M)
Hard references to classes can prevent the
garbage collector from reclaiming the
memory allocated for them when a
ClassLoader is discarded. This will
occur on JSP recompilations, and
webapps reloads. If these operations
are common in a webapp having these
kinds of problems, it will be a matter
of time, until the PermGen space gets
full and an Out Of Memory is thrown.
Source: Tomcat Wiki: OutOfMemory
Well... who really caused the out of memory error?
If you ate 8 slices of pizza and you are full, is it the last slice that caused the out of stomach error?
Use Java Heap Analysis Tool (JHAT) with Eclipse MAT http://www.eclipse.org/mat/ to analyse what's going on inside JVM. What is eating how much memory. See the profile and then see the code causing that.
You can also use JConsole, it's dead easy to set it up. And you can see stuffs 'live'. TPTP is also a good option, unfortunately, I find it hard to configure.
This kind of problem is not easy to nail down based on just the stacktrace. It at least boils down to that you've either a memory leak in your application (the code is keeping (unnecessarily) too much objects for an (unnecessarily) long time in memory), or the server simply doesn't have enough memory in order to be able to run your webapp (simply because it is designed that way to require many memory).
To detect and fix memory leaks, use a Java profiler. If you don't have any memory leaks, i.e. the memory usage is stable but the code just really need that much memory, then simply give the server more memory to work with. The profiler is however still useful to spot memory holes in your webapp and optimize code accordingly.
If you're using Eclipse, use TPTP profiler or if you're using Netbeans, use builtin VisualVM profiler. Or when you're using standalone VisualVM, check this blog how to monitor Tomcat with it.
Related
In my application I have a PermGen out of memory error and I want to see what led to it.
I have connected to my application via VisualVM.
I want to see what exactly consumes so much PermGen memory in my application but I cannot take a special "PermGen Heap" in VisualVM. I can only take a standard HeapDump.
Is it possible to find out what exactly consumes all my PermGen memory using VIsualVM or some other tool?
UPDATE
This question was marked as "duplicate" but in fact I dont see an answer to my question in the topic mentioned. The problem is that I dont see a way to list exactly what consumes memory in my permgen space.
Currently from that topic I know the following hints:
Run jmap -permstat and see how much of PermGen is used by intern Strings. Good idea, it works: now I know that String Pool is not a problem in my case.
Run jmap -permstat and see how the memory is consumed by the classloaders. It works, but there are hundreds of classes in AppClassLoader, this doesn't help me a lot.
Run PrintStringTable.java util to see which Strings occupy memory. I cannot run it in my Windows 7 PC because I get
Attaching to process ID XXXX, please wait... Error attaching to
process: Timed out while attempting to connect to debug server (please
start SwDbgSrv.exe).
Whatever, its not my case.
Trace the classloading in the runtime by specifying JVM properties. It works, but does not help me to understand what takes the memory away, there thousands of classes loaded.
So I cannot see an answer to my question here.
I know that PermGen is consumed not only by classes themselfs, but also by their static members. How can I track which static members of which classes consume the memory?
Please let me know if I have missed this answer.
VisualVM's heap dump should be enough to investigate your permgen problem - I have had to solve far too many PermGen memory leaks and VisualVM is always my starting point.
If you perform the heap dump it will include the info about permgen also - classes, classloader etc - Personally, I then use Eclipse Memory Analyser tool - just load the entire dump in and it can provide leak suspect reports, duplicate class reports etc. Generally, I just go to the duplicate classes list, right-click and select "Find GC Roots" and from there you can normally narrow in on some problems.
this might help (although not read in detail):
http://java.jiderhamn.se/2011/12/11/classloader-leaks-i-how-to-find-classloader-leaks-with-eclipse-memory-analyser-mat/
This seems to be the same question/answer: How to analyze PermGen space?
As an aside, are you doing anything particular with classloading etc? Explicitly reloading or loading with classloaders? Is it just straight Java? Is it in a container etc?
This problem happens when my web application runs between 40 minutes to an hour and my application web doesn't work, the browser wait a response from server, i dont know if it is not by any error in the programming that i doing or a bug in tomcat or bug in jvm.
and this is the exception:
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
It's very probably that you have a memory leak in your application. You have to investigate which objects are eating you memory. To do this, you have to:
Get tomcat pid (use "jps -l" or "ps ux "grep java")
Use jmap to show objects histogram: "jmap -histo:live | head -n20"
Then you will see where the problem is. It will also be very helpful if you will check your memory settings? Maybe your application just needs more memory?
It's extremely likely to be your code. Tomcat and the JVM don't go OOM on their own nowadays. You are either leaking memory, or simply your app requires too much memory for your current configuration.
Since you say that it appears after a long time (over 40 mins) it is likely to be caused by memory leaks (you probably continuously take memory for objects and keep it in use so that the garbage collector can do nothing about it).
This is not a trivial problem, the best way to fight it is to try some post-mortem analysis of your object pool after the problem rises again. I would use jconsole (it is for free and included in every modern JDK), and there are other tools.
Once you have that analysis, there are 2 options:
You find some evidence that some objects are out of control (maybe
someone did a programming error and they are leaking objects). A
code change should fix it
Everything is normal, but it just eats a
lot of memory. In that case, talk with your operator (if it is not
you) to see how to size your server JVM memory. Have a look at this:
How can I increase the JVM memory?
I have been working on this project on Java with multiple modules. Since quite some time, I have been occasionally getting "java: Out Of Memory" error! I am pretty new to this 'popular' error and wanted to know the general approach to solve such errors.
Also, are there standard tools accepted by the industry to help figure out the cause of such errors?
The modules in my project include every minute polling from a third party (using web service), multi-threading among other things. However, this is just a pointer and I seek a general approach and not something very specific to my project.
Thanks.
Sometimes you just have an class that uses a lot of memory and you need to increase the heap size or make a more space-efficient algorithm. Other times it is a leak and you need to deference objects.
Run jvisualvm (it's included in the JDK).
Connect to your process and try if you can to recreate the
out-of-memory error while keeping an eye on the heap size.
Perform a heap dump when the memory grows large. Search for the
largest objects by size - often that will give you the culprit
class.
Look at the dependencies to see what is holding a references. If it is a memory leak make sure to dereference unneeded objects.
Also, are there standard tools accepted by the industry to help figure out the cause of such errors?
Yes, there are memory profilers such as VisualVM and YourKit. I use the latter extensively, for both CPU and memory profiling, and find it extremely useful. To get some idea of what it's capable of, take a look at this page: link.
If you can't increase the available memory you have to consume less.
Don't keep references to Objects that you don't need at the time of execution (like data you can reload dynamically) and if necessary redesign your flow (e.g. don't process all objects in parallel and do it sequentially) to require less memory at that time. The garbage collection should do the rest for you.
Especially if you load big data objects into memory consider to use a streaming approach if possible. E.g. you don't need to load a whole file into memory if you want to search through it. You can just step through it.
Besides architectural problems you can also have leaks: keeping unintentional references to objects you don't need anymore. Since they are referenced, the garbage collector can't free the memory and you run out of memory at some point. That is probably the #1 reason for OutOfMemoryExceptions and it usually has to do with static references since classes and therefore the statics are usually not unloaded after the first time you touch a class. The internet has many articles on finding / fixing those, e.g. How to Fix Memory Leaks in Java
one tool I know of is MAT
You likely have a memory leak. Finding it is a challenge. Netbeans has some tools to help you profile the VM . You can profile your project and view men usage while it runs. Apache JMeter is also available as a plug-in or you can run it on its own.
JMeter.apache.org
If you get OOM too often, then start java with correct options, get a heap dump and analyze it with jhat or with memory analyzer from eclipse (http://www.eclipse.org/mat/)
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path to dump file>
I have an instance of Tomcat which periodically crashes for unknown reasons.
There are no errors left in the logs, only a line in Event Viewer saying "Tomcat terminated unexpectedly".
In a test environment I have been unable to replicate the issue. I am therefore mostly restricted to passive monitoring of the production environment.
The problem does not seem to be related to memory as the unexpected terminations show no obvious correlation to the process' memory usage.
What steps could I take to further diagnose this problem?
EDIT:
Some corrections/clarifications:
It is actually not a single "instance" of Tomcat, rather several instances with similar configurations.
OS is Windows 2003.
Java version is Java 6.
UPDATE:
Looks like the issue might be related to memory after all. Discovered some crash dumps which were created in the Tomcat directory (not .../Tomcat/logs).
The dumps mostly contained errors such as:
java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
This is unexpected as the process sometimes crashed when it's memory usage was a relatively low point (compared to historical usage).
In all dumps, perm gen space is at 99% usage, but in absolute terms this usage is not consistent, and is nowhere near the limit specified in -XX:MaxPermSize.
This indicates to me that the whole JVM crashed, which is a rather unusual thing. I would consider the following steps:
First check the hardware is ok. Run memtest86+ - http://www.memtest86.com/ or on a Ubuntu cd - to test the memory. Let it run a while to be absolutely certain.
Then see if the version of Java you use, is ok. Some versions of Java 6 broke some subtle functionality. The latest Java 5 might be a good solution at this point.
Disable the Tomcat native code to improve performance. There is a native library which Tomcat uses for something. Since you have a crashing JVM, getting rid of native code is a very good start.
See if there is some restrictions in the version of Windows you use. A cpu usage limit before termination, or any other quota.
Generally if a process crashes in windows, a dump file is created. Load the dump file in windbg (windows debugger) and get a stack trace of the thread that caused the exception. This should give you a better idea what the problem is.
I've been tasked with debugging a Java (J2SE) application which after some period of activity begins to throw OutOfMemory exceptions. I am new to Java, but have programming experience. I'm interested in getting your opinions on what a good approach to diagnosing a problem like this might be?
This far I've employed JConsole to get a picture of what's going on. I have a hunch that there are object which are not being released properly and therefor not being cleaned up during garbage collection.
Are there any tools I might use to get a picture of the object ecosystem? Where would you start?
I'd start with a proper Java profiler. JConsole is free, but it's nowhere near as full featured as the ones that cost money. I used JProfiler, and it was well worth the money. See https://stackoverflow.com/questions/14762/please-recommend-a-java-profiler for more options and opinions.
Try the Eclipse Memory Analyzer, or any other tool that can process a java heap dump, and then run your app with the flap that generates a heap dump when you run out of memory.
Then analyze the heap dump and look for suspiciously high object counts.
See this article for more information on the heap dump.
EDIT: Also, please note that your app may just legitimately require more memory than you initially thought. You might try increasing the java minimum and maximum memory allocation to something significantly larger first and see if your application runs indefinitely or simply gets slightly further.
The latest version of the Sun JDK includes VisualVM which is essentially the Netbeans profiler by itself. It works really well.
http://www.yourkit.com/download/index.jsp is the only tool you'll need.
You can take snapshots at (1) app start time, and (2) after running app for N amount of time, then comparing the snapshots to see where memory gets allocated. It will also take a snapshot on OutOfMemoryError so you can compare this snapshot with (1).
For instance, the latest project I had to troubleshoot threw OutOfMemoryError exceptions, and after firing up YourKit I realised that most memory were in fact being allocated to some ehcache "LFU " class, the point being that we specified loads of a certain POJO to be cached in memory, but us not specifying enough -Xms and -Xmx (starting- and max- JVM memory allocation).
I've also used Linux's vmstat e.g. some Linux platforms just don't have enough swap enabled, or don't allocate contiguous blocks of memory, and then there's jstat (bundled with JDK).
UPDATE see https://stackoverflow.com/questions/14762/please-recommend-a-java-profiler
You can also add an "UnhandledExceptionHandler" to your Application's Thread. This will catch 'uncaught' exception, like an out of memory error, and you will at least have an idea where the exception was thrown. Usually this not were the problem is but the 'new' that couldn't be satisfied. As a rule I always add the UnhandledExceptionHandler to a Thread if nothing else to add logging.