I have a Java web server running as a Windows service.
I use Tomcat 8 with Java 1.8.*
For a few months now, I've detected that the memory usage is increasing quite rapidly. I cannot make up for sure if it's heap or stack.
The process starts with ~200MB and after a week or so, it can reach up to 2GB.
Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
This has repeated multiple times on multiple environments.
I would like to know if there's a way to monitor the process and view it's internal memory usage, even to the level of viewing which objects are using the most amount of memory.
Can 'Java Native Memory Tracking' be used for this?
This will help me to detect any memory leaks that might cause this.
Thanks in advance.
To monitor the memory usage of a Java process, I'd use a JMX client such as JVisualVM, which is bundled with the Oracle JDK:
https://visualvm.java.net/jmx_connections.html
To identify the cause of a memory leak, I'd instruct the JVM to take a heap dump when it runs out of memory (on the Oracle JVM, this can be accomplished by specifying -XX:-HeapDumpOnOutOfMemoryError when starting your Java program), and then analyze that heap dump using a tool such as Eclipse MAT.
quoting:
the process starts with ~200MB and after a week or so, it can reach up to 2GB. Shortly after it will generate OutOfMemory exception (the memory usage will be 2GB - 2.5GB).
The problem might not be as simple as seeing what java objects you have got in JVisualVM (e.g millions of strings)
What you need to do is identify the code that leaks.
One way you could do that is to force the execution of particular code and then monitor the memory.
The easiest way to force the execution of code inside classes/objects is to use a tool like https://github.com/lorenzoongithub/nudge4j (particularly since you are on java 8)
alternatively you could just wire up nashorn to a command line or run your progam via jjs https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/shell.html
Related
I am programming a Java application allowing to minimize any boolean expression using QuineMcCluskey methodology. When I compile my code, I have an OutOfMemory Error with the message "Java heap space"!
If I understand, the exception may have several origins:
The memory space allocated to the JVM heap is insufficient to create the objects required by the application.
A memory leak prevents the garbage collector from releasing objects that are yet unused but still have references. Thus these objects are never released and occupy more and more space in the pile until occupying all the available space.
...
I know that use a profiling tool may be necessary to analyze the contents of the memory of the JVM and determine the origin of the memory consumption. But how use thoses tools ? Have I to modify xmx and xms data ? What could be the consequences if I change them ?
(I know also that it is necessary to optimize my code).
What are the different debugging steps ?
Furthermore, this application has to be use by lot of users (so by diferent computers...)
As you can see, I have lot of questions about this problem (I am a novice lol)... That's why I create this post.. I would like to resolve this problem by the better way and also, learn good reflexes.
Thank you!
Increasing of the memory is NOT the solution, unless you have unlimited resources!
If that happens in production or any machine other than your own, you can force the JVM which runs your service to generate a dump file which you can then download and analyse by adding VM flag -XX:+HeapDumpOnOutOfMemoryError to your jvm.conf file.
That is pretty useful when the machine that runs the app becomes unresponsive so no JMX connections could be made in order to run Oracle's JMC (or similar) monitoring tools. If you are for some reason able to get to the VM you can run the flight recording and try to analyse which method is causing the OOM.
Check here how to use flight recording.
And check this for heap dump analysis preview
It is possible to increase heap size allocated by the JVM by using command line options Here we have 3 options
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
-Xss<size> set java thread stack size
java -Xms16m -Xmx64m ClassName
It is also possible to increase heap size allocated by the JVM in eclipse directly In eclipse IDE goto
Run---->Run Configurations---->Arguments
And set VM arguments
We are running weblogic and appear to have a memory leak - we eventually run out of heap space.
We have 5 apps (5 war deployments) on the server.
Can you think of a way to gather memory usage on a per application basis?
(Then we can concentrate our search by looking through the code in the appropriate app.)
I have run jmap to get a heap dump and loaded the results in jvisualvm but it's unclear where the bulk of objects have come from - for example Strings.
I was thinking that weblogic perhaps uses separate classloaders per application and so we may be able to figure something out via that route...
try using Eclipse MAT, it gives hint of memory leaks, among others features
I think I may have a memory leak in a servlet application running in production on jetty 8.1.7.
Is there a way of seeing how much heap memory is actually being used at an instance of time, not the max memory allocated with -Xmx, but the actual amount of memory being used.
Can I force a garbage collection to occur for an application running within jetty
yes, both are easily achievable using: VisualVM (see: http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/monitor_tab.html) This one is shipped with Oracle JDK by default (=> no extra installation required)
However for the memory leak detection, I'd suggest to do memory dump and analyze it later with eclipse MAT ( http://www.eclipse.org/mat/ ) as it has quite nice UI visualizing java memory dumps.
EDIT:
For the ssh only access, yes you can use the mentioned two tools. However you need to run them on the machine with running window manager and remotely connect over ssh to the other machine (you need to have java on both of these machines):
For visualVM: you need to have VisualVM running on one maching and via the ssh connect to remote one, see: VisualVM over ssh
and for the memory dump: use jmap (for sample usage see: http://kadirsert.blogspot.de/2012/01/…) afterwards download the dump file and load if locally to eclipse MAT
enable jmx and connect up to it using jconsole
http://wiki.eclipse.org/Jetty/Tutorial/JMX
You can call System.gc(). That will typically perform a full GC ... but this facility can be disabled. (There is a JVM option to do this with HotSpot JVMs.)
However, if your problem is a memory leak, running the GC won't help. In fact, it is likely to make your server even slower than it currently is.
You can also monitor the memory usage (in a variety of ways - see other Answers) but that only gives you evidence that a memory leak might leak.
What you really need to do is find and fix the cause of the memory leak.
Reference:
How to find a Java Memory Leak
You can use jvisualvm.exe which is under the %JAVA_HOME%\bin folder. By using this application you can monitor memory usage and can force gc.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Limit jvm process memory on ubuntu
In my application I'm uploading documents to a server, which does some analyzing on it.
Today I analyzed my application using jconsole.exe and heap dumps as I tried to find out if I'm having memory issues / a memory leak. I thought I might suffer of one since my application is growing very much on RAM while the application is running.
As I watched the heap / codecache / perm gen etc. memory with jconsole after some runs, I was surprised as I saw the following:
picture link: https://www7.pic-upload.de/13.06.12/murk9qrka8al.png
As you can see at the jconsole on the right, the heap is increasing when I'm doing analyzing-related stuff, but it's also decreasing again to its normal size when the work is over. On the left you can see the "htop" of the sever the application is deployed on. And there it is: The RAM is, although the heap acts normally and it also seems the garbage collector is running correct, incredible high at almost 3,2gb.
This is now really confusing me. I was thinking if my java vm stack could have to do something with this? I did some research and what I found spoke about the vm stack as a little memory with only a few megabytes (or even only kb).
My technical background:
The application is running on glassfish v.3.1.2
The database is running on MySQL
Hibernate is used as ORM framework
Java version is 1.7.0_04
It's implemented using VAADIN
MySQL database and glassfish are the only things running on this server
I'm constructing XML-DOM-style documents using JAXB during the analysis and save them in the database
Uploaded documents are either .txt or .pdf files
OS is linux
Solution?
Do you have any ideas why this happens and what I can do for fixing it? I'm really surprised at the moment, since I thought the memory problems came from a memory leak which causes the heap to explode. But now, the heap isn't the problem. It's the RAM that goes higher and higher while the heap stays on the same level. And I don't know what to do to resolve it.
Thanks for every thought you're sharing with me.
Edit: Maybe I should also state out that this behaviour is currently making me impossible to really let other people use my application. When the RAM is full and the server doesn't respond anymore I'm out.
Edit2: Maybe I should also add that this RAM keeps increasing after every successfull further analyzation.
There are lots more things that use memory in a JVM implementation than the Heap Settings.
The Heap settings via -Xmx only controls the Java Heap, it doesn't control consumption of native memory by the JVM, which is consumed completely differently based on implementation.
From the following article Thanks for the Memory ( Understanding How the JVM uses Native Memory on Windows and Linux )
Maintaining the heap and garbage collector use native memory you can't control.
More native memory is required to maintain the state of the
memory-management system maintaining the Java heap. Data structures
must be allocated to track free storage and record progress when
collecting garbage. The exact size and nature of these data structures
varies with implementation, but many are proportional to the size of
the heap.
and the JIT compiler uses native memory just like javac would
Bytecode compilation uses native memory (in the same way that a static
compiler such as gcc requires memory to run), but both the input (the
bytecode) and the output (the executable code) from the JIT must also
be stored in native memory. Java applications that contain many
JIT-compiled methods use more native memory than smaller applications.
and then you have the classloader(s) which use native memory
Java applications are composed of classes that define object structure
and method logic. They also use classes from the Java runtime class
libraries (such as java.lang.String) and may use third-party
libraries. These classes need to be stored in memory for as long as
they are being used. How classes are stored varies by implementation.
I won't even start quoting the section on Threads, I think you get the idea that
the Java Heap isn't the only thing that consumes memory in a JVM implementation, not everything
goes in the JVM heap, and the heap takes up way more native memory that what you specify for
management and book keeping.
Native Code
App Servers many times have native code that runs outside the JVM but still shows up to the OS as memory associated with the process that controls the app server.
I have a standalone program that I run locally, it is meant to be a server type program running 24/7. Recently I found that it has a memory leak, right now our only solution is to restart it every 4 hours. What is the best way to go about finding this memory leak? Which tool and method should we use?
If you are using Java from Sun and you use at least Java 6 update 10 (i.e. the newest), then try running jvisualvm from the JDK on the same machine as your program is running, and attach to it and enable profiling.
This is most likely the simplest way to get started.
When it comes to hunting memory problems, I use SAP Memory Analyzer Eclipse Memory Analyser (MAT), a Heap Dump analysis tool.
The Memory Analyzer provides a general purpose toolkit to analyze Java heap dumps. Besides heap walking and fast calculation of retained sizes, the Eclipse tool reports leak suspects and memory consumption anti-patterns. The main area of application are Out Of Memory Errors and high memory consumption.
Initiated by SAP, the project has since been open sourced and is now know as Eclipse Memory Analyser. Check out the Getting Started page and especially the Finding Memory Leaks section (I'm pasting it below because I fixed some links):
Start by running the leak report to automatically check for memory leaks.
This blog details How to Find a Leaking Workbench Window.
The Memory Analyzer grew up at SAP. Back then, Krum blogged about Finding Memory Leaks with SAP Memory Analyzer. The content is still relevant!
This is probably the best tool you can get (even for money) for heap dump analysis (and memory leaks).
PS: I do not work for SAP/IBM/Eclipse, I'm just a very happy MAT user with positive feedback.
You need a memory profiler. I recommend trying the Netbeans profiler.
One approach would be to take heap dumps on a regular basis, then trend the instance counts of your classes to try to work out which objects are being consistently created but not collected.
Another would be to switch off parts of your app to try to narrow down where the problem is.
Look at tools like jmap and jhat.
You might look up JMX and the jconsole app that ships with Java. You can get some interesting statistics out-of-the-box, and adding some simple instrumentation to your classes can provide a whole lot more.
As already stated jvisualvm is a great way to get started, but once you know what is leaking you may need to find what is holding references to the objects in question for which I'd recommend jmap and jhat, e.g
jmap -dump:live,file=heap.dump.out,format=b <pid>
and
jhat heap.dump.out
where <pid> is easily found from jvisualvm. Then in a browser navigate to localhost:7000 and begin exploring.
You need to try and capture Java heap dump which is a memory print of the Java process.
It's a critical process for memory consumption optimisation and finding memory leaks.
Java heap dump is an essential object for diagnosing memory-linked issues including java.lang.OutOfMemoryError, Garbage Collection issues, and memory leaks which are all part of Java web development process
For clarity, a Heap dump contains information such as Java classes and objects in a heap during instant of taking the snapshot.
To do it, you need to run jmap -dump:file=myheap.bin <program pid>.
To learn more about how to capture Java heat dumps, check out: https://javatutorial.net/capture-java-heap-dump