I have a system in Scala, with a lot of simultaneous threads and system calls. This system has some problem, because memory usage is increasing over time.
The image bellow shows the memory usage for one day. When it gets to the limit, the process shuts down and I put a watch-dog to recover it again.
I periodically run the command
jcmd <pid> GC.run
And this makes the memory to increase slowly, but the leak still happens.
I analysed with jvisualvm, comparing to distinct moments in time, with 40 minutes delta. The image bellow shows the comparison between these two moments in time. Notice that there is an increase for instances of some classes like ConcurrentHashMap$HashEntry, SNode, WeakReference, char[] and String and many classes in the package scala.collection.concurrent.
What can be causing the memory leak?
Edit 1:
Investigating JVisualVM, I noticed object of CNode and INode classes that are in TriedMap, that is instanced inside sbt.TrapExit$App class. Here is the object hierarchy figure:
First capture a heap dump when your application crashes due to an out of memory issue. Add the following flags when starting the jvm
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/dump
Next you need to analyze the heap dump to figure out the source of the memory leak. I recommend using Eclipse MAT. The Leak Suspects report should give you a sense of what objects are actually causing the leak.
Without seeing the implementation its hard to say. The title of your post suggests that there is a memory leak in Scala, but did you check your implementation against problems with releasing objects?
Did you check following:
Do you limit number of actors at all?
Do you set timeouts for the system calls?
Do you allow the actors to be removed from Heap when they performed therir tasks?
Did you count how many actors can fit into your memory or you are just creating "hundreds of actors" with hope that jvm will know "what to do"
What I'm trying to say is that maybe you run out of memory because you simply create to many objects which are not later released, because either they are still performing their tasks (no timeout) or you have created to many of them.
Maybe you need to scale your application to many jvms? How many jvms do you use?
Related
My WebLogic server was configured with 16gb of heap space, but it was 90% used within 1 hour of production usage when most of the users started work. I observed there were several stuck threads whenever this happens.
I have captured the heap dump when the heap was approx 10% free. How do I inspect the heap dump to find out the memory leak, or process, codes which is causing this issue.
I have tried to understand the memory leak, running tools like JMap and Eclipse MAT, but it maybe due to lack of experience, I couldn't understand what these tools are trying to show. Or how/what should I look out for?
I have both the before/after GC heap dump to analyze.
I have reviewed the thread dumps, there were no "waiting to lock" objects threads, the threads were similar as shown below, with threads stuck with no obvious reasons.
According to your heap dump, your biggest memory issue is the int arrays, indeed it takes nearly 70 % of your heap (Yes sort the Size Column instead).
Select it in your heap dump, right click and select on Show in Instances View
Then browse the biggest objects and for each of them right click and select Show Nearest GC Root to see which Object has still an hard reference to the int array which prevents to be eligible for the GC.
It could help you to find your memory leak assuming that it is a memory leak.
See below an example of Nearest GC Root allowing to identify a leak that I added intentionally to my program just to show the idea. As you can see in the screenshot, I have an array of int which cannot be eligible for the GC because it is stored in an HashMap called leak in my class Application, so I know that my memory issue could be due to this particular HashMap especially if I have many other objects which lead to this HashMap.
NB: Be patient when you try to identify a leak as it is not always obvious, the ideal situation is where you have a huge object that takes the whole heap but obviously it is not your case there is nothing really obvious that is the reason why I propose to investigate the int arrays first. Don't forget that it could also be little int arrays but thousands of them with the same Nearest GC Root.
Another trick, If you have JProfiler you can simply follow this wonderful tutorial to find your leak.
Response Update:
One simple way to better identify the root cause of the memory leak is to take at least 2 heap dumps then compare them using a tool like jhat with the syntax
jhat -J-Xmx2G -baseline ${path-to-the-first-heap-dump} ${path-to-the-second-heap-dump}
It will launch a small HTTP sever on port 7000 so:
Launch http://localhost:7000/
Then click on Show instance counts for all classes (including platform)
You will then see the list of Classes ordered by total amount of new instances created. You can then use VisualVM to do what I described in the first part of my answer to find the root cause of your memory leak.
You can also use jhat
By selecting of the Top Classes then for each of them
click on one "Reference to this Object"
then click on Exclude weak refs
You will then see the GC root of each instances like the next screenshot:
Another way is to use Eclipse Memory Analyzer also called MAT.
Open the second snapshot with it
Select the view histogram
Then for each of the Top Classes right click
Choose Merge Shortest Paths To GC Roots/ Exclude All references
you will then see something like the next screenshot:
The JDK's "jmap -histo" command will dump object counts/bytes for all classes to a text file. If you capture/compare a few of these dumps over time, you will see which ones grow continually -- your memory leak. The overhead of -histo is much lower than that of capturing a full heap dump.
Comparing just a few dumps (like the python script detailed here) seems like too small of a sample, so I wrote an open-source tool (here) that runs this jmap -histo command in the background (at an interval). It has a live display and tracks the % of time that the byte count for each class is on the rise.
It seems you, probably, have a memory leak situation. Your best approach is to use Java Mission Control with Flight Recorder to get the class and method leaking.
You should setup your weblogic managed server with the following parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8999
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
When you set this up, follow the instructions here to detect the leak.
Hope it helps !!
I am one of the developers of the tool called Plumbr. Among other things we make an automatic analysis of heap contents in case of excessive memory usage. You may find it useful.
Per your comments: you have Java 7 with 16GB heap, no GC algorithm explicitly specified, so default for Java 7 is Throughput GC, which is not suitable for most web apps, for it leads to long GC pauses for big heaps.
Switch to ConcurrentMarkSweep GC, this way GC will not wait till your memory fills up and will try its best to collect garbage incrementally, so that you will have fewer Stop The World pauses.
Did you try yourkit profiler? It's not free, but you can evaluate it for 30 days. In this case if you dump contains all object (not only live), you will be able to check roots for them as well. Because it could be that you don't have memory leak, but too big memory footprint. Also it would be great to enable GC logs and parse how much FullGC pauses do you have:
grep "Full GC" jvm_gc.log | wc -l
In ideal world it should be 0 :)
Btw, whole this article could be helpful for you.
How can I determine the memory usage for referenced objects only in java? ie. exclude "dead" objects from memory usage during the running of a Java application.
I want to display this information and trigger an alert if it reaches a certain threahold. I also want to use this to measure how much memory is taken up when a file is imported into my application.
My application consists of many processes that all run at the same time of which one of them imports files into memory and then into a database. if I measure memory usage using the Runtime.getRuntime.freeMemory or MemoryPoolMXBean and with all the other processes running, memory usage goes up and up because of these processes and because the GC isn't running "in real time" the memory usage depicts dead objects as well as referenced ones. This is not a clear indication for me as to what is taking up memory at the time.
Is there a way to determine memory used by referenced objects only at any time?
You can look into JConsole and see if that suits your need.
There is also VisualVM.
They let you monitor the app but I am not sure how you can do that in your own application to trigger an alarm once your memory is low.
Also, you can use WeakReference and SoftReference if you want objects to be garbage-collected quicker.
I found a good article on how to query the size of a Java Object. It is slightly long so I cannot post any of it here. However, here is the link: http://www.javamex.com/tutorials/memory/instrumentation.shtml Here is a SO question on the same topic determining java memory usage
Click for:
JConsole
VisualVM
you can use eclipse memory analyzer MAT.
I am looking into how to use JConsole to detect memory leaks.
I see that in Memory Pool in my MBeans I can define UsageThreashold for my Tenured Generation.
So if my application exceeds this threashold the heap memory becomes red in the Memory tab.
Question: How does this help? I mean how am I supposed to use this setting to analyze my memory? How am I supposed to figure out this value?
In my opinion I don't think that UsageThreashold parameter is the most helpful for you to detect memory leaks (but if someone knows some tricks with it, please do share). In my experience that parameter is more helpful to visually understand if my application is getting way too near my max heap size and I'm in danger of getting an OutOfMemoryException.
Still regarding using JConsole to search for memory leaks, I don't think there's a silver bullet for the process. But what I usually do is the following:
If exists a memory leak, it means that the objects (the ones that are leaking) won't get collected, hence, your Tenured Generation won't fully recover after any amount of GCs.
With the application running I connect JConsole and try to spot a leak by observing the memory tab, if after several computations of my application and also after various GCs occurring (including pressing the Perform GC button, which will result in a full gc) the memory never goes below, or at least to the memory value, it started tracking there's a great possibility that something is leaking. When the leak is big, you can even see a "stair graph" pattern in your memory.
Keep in mind that if your application has long computations running, which may consume memory this analyzes must be done carefully. You must understand when those processes have finished. For example, just run one of those computations and track the total evolution of memory, before, during and afterwards.
Also, I suggest you to try visualVM instead, because it also allows you to create heap dumps, which you can use in order to understand which objects are still in memory and explore the references graph to understand why they are not being collected.
you can use JMAP to see the histogram and/or to create heap dumps and study your memory consumption with tools like Eclipse MAT or YourKit.
JConsole is used more for monitoring and running MBeans and less for analysis and in my expirence JVisualvm is better for that since you can use it for sampling your code and see what methods are CPU consuming.
If, on purpose, I create an application that crunches data while suffering from memory-leaks, I can notice that the memory as reported by, say:
Runtime.getRuntime().freeMemory()
starts oscillating between 1 and 2 MB of free memory.
The application then enters a loop that goes like this: GC, processing some data, GC, etc. but because the GC happens so often, the application basically isn't doing much else anymore. Even the GUI takes age to respond (and, no, I'm not talking about EDT issues here, it's really the VM basically stuck in some endless GC'ing mode).
And I was wondering: is there a way to programmatically detect that the JVM doesn't have enough memory anymore?
Note that I'm not talking about ouf-of-memory errors nor about detecting the memory leak itself.
I'm talking about detecting that an application is running so low on memory that it is basically calling the GC all the time, leaving hardly any time to do something else (in my hypothetical example: crunching data).
Would it work, for example, to repeatedly read how much memory is available during, say, one minute, and see that if the number has been "oscillating" between different values all below, say, 4 MB, conclude that there's been some leak and that the application has become unusable?
And I was wondering: is there a way to programmatically detect that the JVM doesn't have enough memory anymore?
I don't think so. You can find out roughly how much heap memory is free at any given instant, but AFAIK you cannot reliably determine when you are running out of memory. (Sure, you can do things like scraping the GC log files, or trying to pick patterns in the free memory oscillations. But these are likely to be unreliable and fragile in the face of JVM changes.)
However, there is another (and IMO better) approach.
In recent versions of Hotspot (version 1.6 and later, I believe), you can tune the JVM / GC so that it will give up and throw an OOME sooner. Specifically, the JVM can be configured to check that:
the ratio of free heap to total heap is greater than a given threshold after a full GC, and/or
the time spent running the GC is less than a certain percentage of the total.
The relevant JVM parameters are "UseGCOverheadLimit", "GCTimeLimit" and "GCHeapFreeLimit". Unfortunately, Hotspot's tuning parameters are not well documented on the public web, but these ones are all listed here.
Assuming that you want your application to do the sensible thing ... give up when it doesn't have enough memory to run properly anymore ... then just launch the JVM with a smaller "GCTimeLimitor" or "GCHeapFreeLimit" than the defaults.
EDIT
I've discovered that the MemoryPoolMXBean API allows you to look at the peak usage of individual memory pools (heaps), and set thresholds. However, I've never tried this, and the APIs have lots of hints that suggest that not all JVMs implement the full API. So, I would still recommend the HotSpot tuning option approach (see above) over this one.
You can use getHeapMemoryUsage.
I see two attack vectors.
Either monitor your memory consumption.
When you more or less constantly use lots of the available memory it is very likely that you have a memory leak (or are just using too much memory). The vm will constantly try to free some memory without much success => constant high memory usage.
You need to distinguish that from a large zigzag pattern which happens often without being an indicator of memory problem. Basically you use more an more memory, but when gc finds time to do its job it finds lots of garbage to bring out, so everything is fine.
The other attack vector is to monitor how often and what kind of success the gc runs. If it runs often with only small gains in memory, it is likely you have a problem.
I don't know if you can access this kind of information directly from your program. But if nothing else I think you can specify parameters on startup which makes the gc log information into a file which in turn could get parsed.
What you could do is spawn a thread that wakes up periodically and calculates the amount of used memory and records the result. Then you can do regression analysis on the result to estimate the rate of memory growth in your application. If you know the rate of growth, and the maximum amount of memory, you can predict (with some confidence) when your application will run out of memory.
You can pass arguments to your java virtual machine that gives you GC diagnostics such as
-verbose:gc This flag turns on the logging of GC information. Available
in all JVMs.
-XX:+PrintGCTimeStamps Prints the times at which the GCs happen
relative to the start of the
application.
If you capture that output in a file, in your application you can periodcly read that file and parse it to know when the GC has happened. So you can work out the average time between every GC
I think the JVM does exactly this for you and throws java.lang.OutOfMemoryError: GC overhead limit exceeded. So if you catch OutOfMemoryError and check for that message then you have what you want, don't you?
See this question for more details
i've been using plumbr for memory leak detection and it's been a great experience, though the licence is very expensive: http://plumbr.eu/
I have a standalone Java problem running in a linux server. I started the jvm with -Xmx256m. I attached a JMX monitor and can see that the heap never really passes 256Mb. However, on my linux system when I run the top command I can see that:
1) First of all, the RES memory usage of this process is around 350Mb. Why? I suppose this is because of memory outside of the heap?
2) Secondly, the VIRT memory usage of this process just keeps growing and growing. It never stops! It now shows at 2500Mb! So do I have a leak? But heap doesn't increase, it just cycles!
Ultimately this poses a problem because the swap of the system keeps growing and eventually the system dies.
Any ideas what is going on?
The important question I want to ask, what are some scenarios that this could be a result of my code and not the JVM, kernal, etc. For example, if the number of threads keeps growing, would that fit the description of my observations? Anything similar that you can suggest me to look out for?
A couple of potential problems:
Direct allocated buffers and memory mapped files are allocated outside of the Java heap, and can't conveniently be disposed.
An area of stack is reserved for each new thread.
Permanent generation (code and interned strings) is outside of the usual stack. It can be a problem is class loaders leak (usually when reloading webapps).
It's possible that the C heap is leaking.
pmap -x should show how your memory has disappeared.
Swap Sun vs IBM JVM to test
RES will include code + non-head data. Also, some things that you think would be stored in the heap aren't, such as the thread stack and "class data". (It's a matter of definition but code and class data are controlled by -XX:MaxPermSize=.)
This one sounds like a memory leak in either the JVM implementation, the linux kernel, or in library JNI code.
If using the Sun JVM, try IBM, or vice versa.
I'm not sure exactly how dlopen works, but code accessing system libraries might be remapping the same thing repeatedly, if that's possible.
Finally, you should use ulimit to make the system fail earlier, so you can repeat tests easily.
WRT #1, it's normal for your RSS to be larger than your heap. This is because system libraries and non-Java code are included in the RSS but not the heap size.
WRT #2, Yes, it sounds like you have a leak of some sort. If the system itself is crashing, you are likely consuming too much of a system resources, like sockets, threads, or files.
Try using lsof to see what files the JVM has open. Run this a few times as your memory increases. If the JVM is crashing, be sure to set the -XX:+HeapDumpOnOutOfMemoryError option.
In my experience, the most common cause of non-heap memory leak in Java is thread leak.
A tool you may find useful is jvmtop, which lets you monitor heap size, thread number and other metrics in real time.
Sounds like you have a leak. Can't you do profiling to see which function is driving the memory up? I am not sure though.
If I had to take a stab in the dark, I would say that the JVM you are using has a memory leak.