How to identify the issue when java OutOfMemoryError or stackoverflow comes in production. By which reason it is coming or why the server is down.
For example I am developing an application which is lived on production and UAT. Instantly on production java OutOfMemoryError or stackoverflow.
Then how can we track this issue, by which reason it has happened ? Is there any technique that can tell me by which code flow this is happening ?
Please explain it. I have faced this issue many times.
If you face it in production and you cannot really reason about it from stacktraces or logs, you need to analyze what was in there.
Get the VM to dump on OOM
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath="/tmp"
And use that for analysis. The memory analyzer tool (http://eclipse.org/mat/) is a good standalone program for this analysis.
The Oracle docs:- Troubleshooting Memory Leaks has detailed explanation on it:
This error is thrown when there is insufficient space to allocate an
object in the Java heap or in a particular area of the heap. The
garbage collector cannot make any further space available to
accommodate a new object, and the heap cannot be expanded further.
.....
An early step to diagnose an OutOfMemoryError is to determine what the
error means. Does it mean that the Java heap is full, or does it mean
that the native heap is full? To help you answer this question, the
following subsections explain some of the possible error messages,
with reference to the detail part of the message:
Exception in thread "main": java.lang.OutOfMemoryError: Java heap
space
See 3.1.1 Detail Message: Java heap space.
Exception in thread "main": java.lang.OutOfMemoryError: PermGen space
See 3.1.2 Detail Message: PermGen space.
Exception in thread "main": java.lang.OutOfMemoryError: Requested
array size exceeds VM limit
See 3.1.3 Detail Message: Requested array size exceeds VM limit.
Exception in thread "main": java.lang.OutOfMemoryError: request
bytes for . Out of swap space?
See 3.1.4 Detail Message: request bytes for . Out of
swap space?.
Exception in thread "main": java.lang.OutOfMemoryError:
(Native method)
See 3.1.5 Detail Message: (Native method).
UPDATE:-
You can download the HotSpot VM source code from OpenJDK. If you want to monitor and track the memory footprint of your Java Heap spaces ie, the young generation and old generation spaces is to enable verbose GC from your HotSpot VM. You may add the following parameters within your JVM start-up arguments:
-verbose:gc –XX:+PrintGCDetails –XX:+PrintGCTimeStamps –Xloggc:<app path>/gc.log
You can use jvisualvm to manage your process at the runtime.
You can see memory, heap space, objects etc ...
This program is located in your bin directory of your JDK.
It's best to try to reproduce the problem in place where you are free to debug it - on dev server or on your local machine. Then try to debug, look for recursive invocations, heap size and what objects are being created. Unfortunately it's not always easy to reproduce prod env (with it's load and so on) on local machine, therefore finding the root cause of such error might be a challenge.
The amount of memory given to Java process is specified at startup. memory divided into separate areas, heap and permgen being the most familiar sub-areas.
While you specify the maximum size of the heap allowed for this particular process via -Xmx,
the corresponding parameter for permgen is -XX:MaxPermSize. 90% of the Java apps seem to require between 64 and 512 MB of permgen to work properly. In order to find your limits, experiment a bit.
to solve this issue you have change your VM arguments
-Xms256m -Xmx1024m -XX:+DisableExplicitGC -Dcom.sun.management.jmxremote
-XX:PermSize=256m -XX:MaxPermSize=512m
add above two line in VM argument i am sure you will not face this problem any more
to know more about go to OutOfMemory
Low memory configuration :-
It is possible that you have estimate less memory for your application for example your application need 2 Gb of memory but you have configured only 512 Mb so here you will get an OOME(Out-of-memory errors )
Due to Memoryleak :-
Memory leak is responsible for decreasing the available memory for heap and can lead to out of memory error for more read What is a Memory Leak in java?
Memory fragmentation :-
It is possible that there may be space in heap but it may be not contiguous . And heap needs compaction . Rearrange its memory.
Excess GC overhead :-
Some JVM implementations, such as the Oracle HotSpot, will throw an out-of-memory error when GC overhead becomes too great. This feature is designed to prevent near-constant garbage collection—for example, spending more than 90% of execution time on garbage collection while freeing less than 2% of memory. Configuring a larger heap is most likely to fix this issue, but if not you’ll need to analyze memory usage using a heap dump
Allocating over-sized temporary objects:-
Program logic that attempts to allocate overly large temporary objects. Since the JVM can’t satisfy the request, an out-of-memory error is triggered and the transaction will be aborted. This can be difficult to diagnose, as no heap dump or allocation-analysis tool will highlight the problem. You can only identify the area of code triggering the error, and once discovered, fix or remove the cause of the problem.
for more pls visit my site
http://all-about-java-and-weblogic-server.blogspot.com/2014/02/main-causes-of-out-of-memory-errors-in.html
Related
from time to time jboss serwer throws "An unhandled exception has occurred:
java.lang.OutOfMemoryError: Java heap space".
Could you please explain me how it works?
A few infromation:
Jboss is running all the time in standalone mode on Windows.
From standalone.conf "JAVA_OPTS=-Xms1G -Xmx1G -XX:MaxPermSize=256M"
I deploy ~50MB war file, after tests I remove it.
What is possible cause of this Java heap space exception?
Should I restart server between following deploys?
Is there any command to clean heap space?
If I understand corectly increasing of -Xmx argument will not help. It will only delay the appearance of the exception. Right?
Thanks in advance
What is possible cause of this Java heap space exception?
On the face of it, the explanation is simple. The JVM has run out of heap space, and the GC is unable to reclaim enough space to continue.
But what caused the JVM to get into that state?
There are a number of possible explanations, but they mostly fall into three classes:
Your application has a memory leak.
Repeated deploys are causing memory to leak.
There are no memory leaks, but occasionally your application is getting a request that simply needs too much memory.
Should I restart server between following deploys?
That may help if the deploys are causing the leaks. If not, it won't.
Is there any command to clean heap space?
There is no command that will do this. The JVM will already have run a "full" GC before throwing the OOME.
If I understand corectly increasing of -Xmx argument will not help. It will only delay the appearance of the exception. Right?
That depends. If the root cause is #3 above, then increasing the heap size may solve the problem. But if the root cause is #1 or #2, then tweaking the heap size will (at best) cause the JVM to survive longer between crashes.
My recommendation is to start by treating this as a "normal" (cause #1) memory leak, and use a memory profiler to identify and fix leaks that are likely to build over time.
If / when you can definitively eliminate cause #1, consider the others.
I absolutely agree with the answer from Stephen C, i just want to show you a possible way how you can analyse it.
A minimal tool for monitoring the memory is jstat and comes with JDK.
After starting JBoss you can start monitoring the memory and GC with
jstat -gc <JBOSS_PID> 2s
The output can then be loaded for example in an excel.
Now when you recognize something strange happens with the memory, take a heap dump:
jcmd <JBOSS_PID> GC.heap_dump <filename>
jcmd also comes with JDK.
You can than load the heap dump into MAT and analyse it. It takes some practice and patience to work with MAT. They also have a good Tutorial. You can also compare heap dumps in MAT.
I also suggest you add XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path>" to your JAVA_OPTS.
Please increase the MaxPermSize variable and check it will work.
JAVA_OPTS=-Xms1G -Xmx1G -XX:MaxPermSize=512M
I got the following error in browser when log into the system
java.lang.RuntimeException: javax.servlet.ServletException:
java.lang.OutOfMemoryError: PermGen space
com.opensymphony.sitemesh.webapp.decorator.BaseWebAppDecorator.render(BaseWebAppDecorator.java:39)
com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:84)
java.lang.OutOfMemoryError
Thrown when the Java Virtual Machine cannot allocate an object because
it is out of memory, and no more memory could be made available by the
garbage collector. OutOfMemoryError objects may be constructed by the
virtual machine as if suppression were disabled and/or the stack trace
was not writable.
Increase the heap size of your JVM
It is possible to increase heap size allocated by the JVM by using command line options Here we have 3 options
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
-Xss<size> set java thread stack size
Common causes of OutofMemory in PermGen is ClassLoader. Whenever a class is loaded into JVM, all its meta data, along with Classloader, is kept on PermGen area and they will be garbage collected when the Classloader which loaded them is ready for garbage collection. In Case Classloader has a memory leak than all classes loaded by it will remain in memory and cause permGen outofmemory once you repeat it a couple of times.
Now there are two ways to solve this:
1. Find the cause of Memory Leak or if there is any memory leak.
2. Increase size of PermGen Space by using JVM param -XX:MaxPermSize and -XX:PermSize.
java.lang.OutOfMemoryError: PermGen space
Two step solution.
Step One : Alter your application launch configuration and add (or increase if present) the -XX:MaxPermSize parameter . This is a temporary step as if this is a live system most probably the extra memory will also get filled at some time in the future.But this will fix the error for now.
Step two : Release unnecessary resources. Make sure all database connections are closed. Load and unload classes judiciously.restructure you program to work with small amounts of data at a time. Analyse and Fix memory leaks
Either optimize your program or increase java heap size with run time parameters.(-Xxx)
It is important to know, whether the PermGenError occurs after every depolyment, or after few ones.
If it is the first case, just increase the memory size for your PermGen.
If it is the second, you have some dead cows in your code. In this case, increasing memory size will only postpone the error, as your memory is still growing in size after every deployment. You have to profile your app like for example with jvisualvm (here is nice tutorial) and find some dead cows. It is important to understand, that PermGenError is not about the classes itself, but about classLoaders (Classloader leaks: the dreaded "java.lang.OutOfMemoryError: PermGen space" exception).
In my case (Glssfish server), the problem was the Log4j2 libs added to the web app. I had to add them also to the server libs (domain-dir/lib), so that according to the Glassfish Class Loaders Hierarchy they where loaded by the Common classLoader.
I am running tomcat7 service which process quite a big load from customers. I left application during a weekend and when I back I noticed that tomcat CPU usage increased to 99% and in the logs I have found following errors:
Exception in thread "http-bio-8080-exec-908" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-8080-exec-948" java.lang.OutOfMemoryError: Java heap space
Does it means that at the time I've got OutOfMemory exception I had 908 and 948 opened active threads?
Currently my tomcat is runnning under default configurations I've never increased heap size yet.
We are receiving around 200 queries/sec.
My hardware:
CPU : Intel(R) Xeon(R) CPU X5650 # 2.67GHz
Memory: 2GB
Could you please point me into right direction, what should I look at in order to resolve this issue.
Thanks for any help!
There could be couple of reasons,
Since you have not increased the default memory allocation, with high load, it probably won't have enough resources to serve all requests and some are obviously running out of memory. So first thing to try is to tweak the jvm memory configuration.
If 1 does not working, it could be a bug in your application that might be preventing garbage collection. You might want to run your application with a profiler attached to see what objects are collected over time and use that information to debug.
When your JVM runs out of a particular resource, java.lang.OutOfMemoryError: Java heap space error will occur.
Solution 1 : You should increase the availability of that resource. When your application does not have enough Java heap space memory to run properly. For that, you need to alter your JVM launch configuration and add the following:
-Xmx1024m
The above configuration would give the application 1024MB of Java heap space. You can use g or G for GB, m or M for MB, k or K for KB. For example all of the following are equivalent to saying that the maximum Java heap space is 1GB:
-Xmx1073741824
-Xmx1048576k
-Xmx1024m
-Xmx1g
Solution 2 : If your application contains a memory leak, adding more heap will just postpone the java.lang.OutOfMemoryError: Java heap space error.
If you wish to solve the underlying problem with the Java heap space instead of masking the symptoms, you have several tools available.
For Example: Plumbr tool. Among other performance problems it catches all java.lang.OutOfMemoryErrors and automatically tells you what causes them. You can go through about other available tools. And the choice is yours!
Source:
https://plumbr.eu/outofmemoryerror/java-heap-space
I am new JAVA.
We are using one a tool which is using JAVA Heap size.
Recently we are facing "java.lang.OutOfMemoryError: Java heap space" and tool is geting HUNG.
Now I planning to write a script to get the heap size and if excceds a limt i need to clean the GC so the heap usage wll be low again.
Please help me in this.
Machine OS: Win 2012 Server
Ok. Two things.
If you get an OutOfMemoryError, garbage collection will not help. OutOfMemoryErrors are:
Thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. OutOfMemoryError objects may be constructed by the virtual machine as if suppression were disabled and/or the stack trace was not writable.` Source
There are a few JMX implementation for Perl. Here is the first result off Google
However this will not help, because the problem is not what you think it is. The problem is either:
The program requires more heap space than what is currently available. You need to allocate more Heap space for the JVM.
The application is leaking memory (well technically objects, but the result is the same). You need to hire better programmers.
Weblogic 10.3.5 running with 32 bit JRockit R 28.2.4-14 using -Xmx1024m -Xms1024m always gets out of native memory after 5-8 Undeploy-Redeploy cycles of our Java EE EAR files.
According to the error message and what is displayed in VisualVM, it is not the Java Heap that gets too full but insufficient system memory which is available.
java.lang.OutOfMemoryError: class allocation, 865324184 loaded, 464M footprint,
in check_alloc (src/jvm/model/classload/classalloc.c:215).
Attempting to allocate 1G bytes
There is insufficient native memory for the Java
Runtime Environment to continue.
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Disable compressed references (-XXcompressedRefs=false)
at sun.misc.Unsafe.defineClass(Native Method)
at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:95)
at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:313)
at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1322)
I understand the possible solutions that are suggested, but as everything is fine if the application is only deployed once, it seems classes are not correctly freed when undeploying. A heap dump after undeployment shows there are many of our classes left in memory. Shouldn't they be garbage collected then?
The path to GC Root shows a Thread <JNI Local> java.lang.Thread # 0x129ac778 JDWP Transport Listener: dt_socket Native Stack, Thread. There is no traffic on the server and I don't know why this stays to be active.
This memory leak is most likely caused in the perm-gen space (this is how it's called on Hotspot JVM). JRockit doesnt have dedicated Perm-Gen space, but uses "regular" heap space for this.
Have a look at the following sites which I found really helpful for understanding what's happening here:
What is a PermGen leak
Busting PermGen Myths
I find Eclipse MAT very helpful for debugging PermGen leaks:
My approach is usually something like this:
Do a heap dump after undeployment
Find one of you application classes (doesn't really matter which, all should leak). Alternatively display duplicate classes.
Display the path to the GC root
Why the fix looks like depends on the cause.
Most likely some classes are holding to the classloader which causes the full application to leak on redeploy. You can read about this article about classloaders.