I'm trying to figure out the best way to analyse why an application suddenly allocates memory more often than before. For a better understanding what I mean, take a look at this memory graph.
As you can see it runs fine for a few days and then allocates memory way faster.
What tool can help me here? What is the best approach?
I have already taken several thread dumps but there is nothing noticeable. I dumped the heap twice and compared the objects but that didn't help either.
I don't think it is a memory problem because the full gc can always clean up the same amount of memory. It rather looks like some routine started to run which requires a lot of memory. It is not related to heavier usage as it turns back to normal as soon as I restart the application.
Related
I'm using Java persistence API to develop a standalone software. Recently I saw that the memory usage keep rising when I'm creating objects from entity classes, as well as JPAController classes. It seems that the objects stays at the memory since the memory allocation to the project won't decrease (Eg: 400mb ---> Create Object ---> 450mb ---> Stays at 450mb). Will this affect badly on performance? Should I call System.gc() method to remove these objects?
Generally System.gc() is not guarenteed to perform a garbage collection. Ultimately it is up to the JVM to decide. See the javadoc.
Have you observed what happens when you are approaching your memory limits of the JVM, does garbage collection happen then ? If not and you receive an OutOfMemoryError, you either are retaining something longer than you need to, or actually need extra heap allocated to your VM.
In anycase System.gc() I believe shouldn't be used to solve such problems.
In my opinion, the approach to the problem should be different. Actually the call to System.gc() is not a guarantee that it will free any memory at all; please see When does System.gc() do anything
If you can measure a problem in your memory allocation, either via jconsole, or making a post mortem analysis on the jvm dump, or whatever, then this is another problem. By gathering this information you will know what remains where in your memory regions, and then take actions in order to contain it.
The only way that this would negatively affect performance throughout the life of your program is if you want to keep these entities around forever but the size of your old generation in your heap is less than the 450MB you specified. Assuming that you are want to keep around between 1 and 2 times the 450MB you have you have specified forever, with the default ratios of the JVM, setting a parameter such as -Xmx2g will probably be fine. There are many more parameters to fine tune your performance much more than that, but that's probably all the complexity you're looking for for now. If you want to check out some more details on heap tuning and really get into performance, check out this doc on Garbage Collection Tuning by Oracle. Alternatively, something to eat lunch to is a great Youtube video on GC tuning by a guy named Gil Tene.
But calling System.gc() probably won't do anything useful.
Our JBoss 3.2.6 application server is having some performance issues and after turning on the verbose GC logging and analyzing these logs with GCViewer we've noticed that after a while (7 to 35 hours after a server restart) the GC going crazy. It seems that initially the GC is working fine and doing a GC every hour or so but at a certain point it starts going crazy and performing full GC's every minute. As this only happens in our production environment have not been able to try turning off explicit GCs (-XX:-DisableExplicitGC) or modify the RMI GC interval yet but as this happens after a few hours it does not seem to be caused by the know RMI GC issues.
Any ideas?
Update:
I'm not able to post the GCViewer output just yet but it does not seem to be hitting the max heap limitations at all. Before the GC goes crazy it is GC-ing just fine but when the GC goes crazy the heap doesn't get above 2GB (24GB max).
Besides RMI are there any other ways explicit GC can be triggered? (I checked our code and no calls to System.gc() are being made)
Is your heap filling up? Sometimes the VM will get stuck in a 'GC loop' when it can free up just enough memory to prevent a real OutOfMemoryError but not enough to actually keep the application running steadily.
Normally this would trigger an "OutOfMemoryError: GC overhead limit exceeded", but there is a certain threshold that must be crossed before this happens (98% CPU time spent on GC off the top of my head).
Have you tried enlarging heap size? Have you inspected your code / used a profiler to detect memory leaks?
You almost certainly have a memory leak and the if you let the application server continue to run it will eventually crash with an OutOfMemoryException. You need to use a memory analysis tool - one example would be VisualVM - and determine what is the source of the problem. Usually memory leaks are caused by some static or global objects that never release object references that they store.
Good luck!
Update:
Rereading your question it sounds like things are fine and then suddenly you get in this situation where GC is working much harder to reclaim space. That sounds like there is some specific operation that occurs that consumes (and doesn't release) a large amount of heap.
Perhaps, as #Tim suggests, your heap requirements are just at the threshold of max heap size, but in my experience, you'd need to pretty lucky to hit that exactly. At any rate some analysis should determine whether it is a leak or you just need to increase the size of the heap.
Apart from the more likely event of a memory leak in your application, there could be 1-2 other reasons for this.
On a Solaris environment, I've once had such an issue when I allocated almost all of the available 4GB of physical memory to the JVM, leaving only around 200-300MB to the operating system. This lead to the VM process suddenly swapping to the disk whenever the OS had some increased load. The solution was not to exceed 3.2GB. A real corner-case, but maybe it's the same issue as yours?
The reason why this lead to increased GC activity is the fact that heavy swapping slows down the JVM's memory management, which lead to many short-lived objects escaping the survivor space, ending up in the tenured space, which again filled up much more quickly.
I recommend when this happens that you do a stack dump.
More often or not I have seen this happen with a thread population explosion.
Anyway look at the stack dump file and see whats running. You could easily setup some cron jobs or monitoring scripts to run jstack periodically.
You can also compare the size of the stack dump. If it grows really big you have something thats making lots of threads.
If it doesn't get bigger you can at least see which objects (call stacks) are running.
You can use VisualVM or some fancy JMX crap later if that doesn't work but first start with jstack as its easy to use.
I have a problem with my program (JSF running on Glassfish). It's proceeding a lot of data (and inserting it to the database using hibernate). And the problem is that after about 2 hours of work it slows down. I don't get any exception (especially there is no OutOfMemory). Is it possible that it is a memory leak? I've checked the heap dump with Eclipse Memory Analyzer and there were some HashMap issues. I've repaired it where it was possible and now the tool doesn't show this problem. But my application still doesn't work properly.
It sounds like your problem is not a conventional memory leak at all.
If I was to guess, I'd say that you've got a poorly designed data structure, an ineffective cache, or maybe a concurrency bottleneck.
You should probably focus on performance profiling to see where the time is going and to look for signs of lock contention.
There is a chance, that you have some sort of memory leak and produce a lot of temporary objects so that after a decent time the garbage collector kills your performance. If this is the case, you could play with the -Xmx option: with less heap size your application should slow down earlier, a bigger heap should show an oppisite effect.
The effect could also be caused by growing internal datastructures. Operations on datastructures always have a time complexity ("Big-O-Notation") and if the complexity is polynomal or even worse, such operations can kill performance too. Have a look at those collections in your applications that grow over time and double-check, that you've chose the optimal collection type.
I've written a pretty complex java application that is doing a lot of calculations on price data from the markets in real time and from looking at the task manager in windows this sucker is taking close to 1MEG every 30 seconds and the performance is fine until it gets closer to the memory limit around 300MEG and then the g-collector really kicks in and spikes my CPU to around 50% and the UI performance rapidly degrades from all I've written so far it sounds like I have some bad code going on because the nature of my program is CPU intensive but by design stores very little data in memory.
I need some help on what might be some good next steps to take to see how I can figure out what the problem is, I think if I can see what objects are getting stored in memory that would help as maybe I have some lousy code but I am heart broken with Java as I thought these are problems I would not have to worry about. Thanks for any answers. - Duncan
Identify some reasonable performance targets (memory usage, throughput, latency).
Put together some repeatable performance tests, the closer you can get these to real life scenarios the better.
Get a hold of a good profiler. I've used YourKit with a lot of success, the Netbeans and Eclipse profilers are not bad either. Most decent profilers will be able to identify memory usage, GC and performance hotspots.
Identify the biggest culprits and start fixing the issues beginning at the TOP of the list.
Check out VisualVM. It's in the current JDK bin directory as jvisualvm. If you don't have a memory leak, the heap usage should go down when you run the garbage collector, and you can see which objects may be holding memory by calculating the retained sizes of objects in the heap.
http://download.oracle.com/javase/6/docs/technotes/guides/visualvm/intro.html
Like others say, use a profiler to find what is consuming the memory.
If you don't know already, the garbage collector can only release memory on objects that are out of scope. That is, don't have any references to them. Just make sure it goes out of scope when your done with it. It sounds like your locking it up in a way were it's still referenced some where.
Also, if you want to suggest to the GC that it cleans up, try this:
System.gc();
System.runFinalization();
Again, that is only a suggestion to the gc; but I've found it really helps if you run it after a lot of objects go out of scope.
Lastly, you can tweak your vm arguments.
There are settings for min/max heap size settings. If it's a critical application set them to the same and set it high (that way it doesn't have to keep allocating/deallocating - it just grabs one big chunk at startup). This isn't a fix, just a workaround.
I have a mobile application that is suffering from slow-down over time. My hunch, (In part fed by this article,) is that this is due to fragmentation of memory slowing the app down, but I'm not sure. Here's a pretty graph of the app's memory use over time:
fraggle rock http://kupio.com/image-dump/fragmented.png
The 4 peaks on the graph are 4 executions of the exact same task on the app. I start the task, it allocates a bunch of memory, it sits for a bit (The flat line on top) and then I stop the task. At that point it calls System.gc(); and the memory gets cleaned up.
As can be seen, each of the 4 runs of the exact same task take longer to execute. The low-points in the graph all return to the same level so there do not seem to be any memory leaks between task runs.
What I want to know is, is memory fragmentation a feasible explanation or should I look elsewhere first, bearing in mind that I've already done a lot of looking? The low-points on the graph are relatively low so my assumption is that in this state the memory would not be very fragmented since there can't be a lot of small memory holes to be causing problems.
I don't know how the j2me memory allocator works though, so I really don't know. Can anyone advise? Has anyone else had problems with this and recognises the memory profile of the app?
If you've got a little bit of time, you could test your theory by re-using the memory by using Memory Pool techniques: each run of the task uses the 'same' chunks of memory by getting them from the pool and returning them at release time.
If you're still seeing the degrading performance after doing this investigation, it's not memory fragmentation causing the problem. Let us all know your results and we can help troubleshoot further.
Memory fragmentation would account for it... what is not clear is whether the Apps use of memory is causing paging? this would also slow things up.... and could cause the same issues.
It the problem really is memory fragmentation, there is not much you can do about it.
But before you give up in despair, try running your app with a execution profiler to see if it is spending a lot of time executing in an unexpected place. It is possible that the slow down is actually due to a problem in your algorithms, and nothing to do with memory fragmentation. As people have already said, J2ME garbage collectors should not suffer from fragmentation issues.
Consider looking at garbage collection statistics. You should have a lot more on the last run than the first, if your theory is to hold. Another thought might be that something else eats your memory so your application has less.
In other words, profiler time :)
What OS are you running this on? I have some experience with Windows CE5 (or Windows Mobile) devices. CE5's operating system level memory architecture is quite broken and will fail soon for memory intensive applications. Your graph does not have any scales, but every process only gets 32MB of address space on CE5. The VM and shared libraries will take their fair share of that as well, leaving you with quite little left.
The only way around this is to re-use the memory you allocated instead of giving it back to the collector and re-allocating later. This is, of course, much more low-level programming than you would usually want to do in Java, but on this platform you might be out of luck.