The sun JVM spits out a LOT of extra noise when run under valgrind, which makes tracking memory problems in the application very challenging.
I'd like to find either a suppression file, or a VM runtime mode, that will strip out spurious memory errors in order to separate the wheat from the chaff in this situation. Any suggestions?
What about profiling this native code outside of the Java application? Usually the JNI code is a wrapper around some library that is not Java specific. Not sure if that is true for your specific case, but if it is then maybe the memory problems can be isolated by writing a plain C or C++ test framework around that library?
If your framework is in C++ then you might also be able to supply your own new and delete operators and track memory usage yourself. You'll have to collect statistics and process them with some scripts but it can work well.
I can't answer your posted question, but can you elaborate on what problem you're having?
In other words, can you tell us if it is...
In the JNI layer and not a JVM object scope issue?
A use of free'd memory?
A buffer underwrite/overwrite?
Other memory corruption?
I recently had to debug a Java/C that had issues (after 30+ minutes into its run), which it turns out was using memory after it had been free'd. I tried using dmalloc, a custom memory leak library of mine, Valgrind and none worked as I needed.
Eventually I created a simple set of wrappers around free, malloc, calloc, realloc that simply printed memory addresses and sizes to a file. After it aborted (within GDB) I could backtrack in time and figure out when the memory was free'd and where the references were that did not get removed.
IF your issue is in C/C++ and you can trap the error in a debugger this might work for you. Yes, it's tedious, but maybe no worse than sifting through megabytes of Valgrind output.
Hope that helps & good luck.
While not as spiffy as valgrind (based on what I've read), you might try jmap and jhat. These tools allow you to take a snapshot of the running process and see what's going on. I've used this technique for simple memory leaks and it's worked well. However, if the memory problems are caused by non-jvm allocations, this won't help.
Related
I have project written in java that using JNI uses a C++ library.
All the code was written by us, so I have all the source code.
After few hours the machine runs out of memory although my process just iterate over files and all the memory regarding the previous file deleted.
I'm sure that there is a memory leak, usually I use Valgrind, but it seems he can't cope with Java very well and believes that the JVM is leaking, even for "hello world" java project.
I've tested the C++ parts (the major flows) with unit-tests and used valgrind on the unit-tests, but couldn't find any leakage. It doesn't prove anything because there are many potential flows I could've missed.
My major question, how can I find my leak ?
It will be extremely helpful to know who is consuming the memory, the java or the native part ? they are in the same process.
Thanks.
You can use jemalloc to debug native memory leaks. This blog post has a detailed example of using jemalloc to debug a native memory leak in java applications.
From my experience, Valgrind is actually usable with the JVM, and it remains the best tool to hunt leaks in C/C++ code, even with JNI. So your question kinda contains the answer that you need ;)
If you fail to use JNI and Valgrind together, please refer to Valgrind and Java.
I am trying to detect the memory leaks in my java application using VisualVM. I am using the VisualVM 1.3.5.
I followed the steps which should be said in this tutorial http://rejeev.blogspot.in/2009/04/analyzing-memory-leak-in-java.html
After following those steps, I don't know where I have to start edit my code. Is there any way to find the java class and the line number where the memory was leaked to correct the code.
Or any one suggest me a good way to find memory leaks using the VisualVM.
Good answers are definitely appreciated .
No profiling tool will give you the line where a potential memory leak is concurring.
Profiling an application takes a bit more effort than that. Usually, a tool like VisualVM will, for instance, show you what type of Objects are being instantiated the most, and that can indicate where the problem is.
For instance, if a huge amount of byte[] objects are being created, perhaps you're not closing the Input/Output streams you are creating?
There is no silver bullet to find memory leaks, it takes effort and some practice, and is completely application dependent.
That being said, this link might help as well:
http://www.kdgregory.com/index.php?page=java.outOfMemory
I have been working on this project on Java with multiple modules. Since quite some time, I have been occasionally getting "java: Out Of Memory" error! I am pretty new to this 'popular' error and wanted to know the general approach to solve such errors.
Also, are there standard tools accepted by the industry to help figure out the cause of such errors?
The modules in my project include every minute polling from a third party (using web service), multi-threading among other things. However, this is just a pointer and I seek a general approach and not something very specific to my project.
Thanks.
Sometimes you just have an class that uses a lot of memory and you need to increase the heap size or make a more space-efficient algorithm. Other times it is a leak and you need to deference objects.
Run jvisualvm (it's included in the JDK).
Connect to your process and try if you can to recreate the
out-of-memory error while keeping an eye on the heap size.
Perform a heap dump when the memory grows large. Search for the
largest objects by size - often that will give you the culprit
class.
Look at the dependencies to see what is holding a references. If it is a memory leak make sure to dereference unneeded objects.
Also, are there standard tools accepted by the industry to help figure out the cause of such errors?
Yes, there are memory profilers such as VisualVM and YourKit. I use the latter extensively, for both CPU and memory profiling, and find it extremely useful. To get some idea of what it's capable of, take a look at this page: link.
If you can't increase the available memory you have to consume less.
Don't keep references to Objects that you don't need at the time of execution (like data you can reload dynamically) and if necessary redesign your flow (e.g. don't process all objects in parallel and do it sequentially) to require less memory at that time. The garbage collection should do the rest for you.
Especially if you load big data objects into memory consider to use a streaming approach if possible. E.g. you don't need to load a whole file into memory if you want to search through it. You can just step through it.
Besides architectural problems you can also have leaks: keeping unintentional references to objects you don't need anymore. Since they are referenced, the garbage collector can't free the memory and you run out of memory at some point. That is probably the #1 reason for OutOfMemoryExceptions and it usually has to do with static references since classes and therefore the statics are usually not unloaded after the first time you touch a class. The internet has many articles on finding / fixing those, e.g. How to Fix Memory Leaks in Java
one tool I know of is MAT
You likely have a memory leak. Finding it is a challenge. Netbeans has some tools to help you profile the VM . You can profile your project and view men usage while it runs. Apache JMeter is also available as a plug-in or you can run it on its own.
JMeter.apache.org
If you get OOM too often, then start java with correct options, get a heap dump and analyze it with jhat or with memory analyzer from eclipse (http://www.eclipse.org/mat/)
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path to dump file>
I have been developing a small Java utility that uses two frameworks: Encog and Jetty to provide neural network functionality for a website.
The code is 'finished' in that it does everything it needs to do, but I have some problems with memory usage. When running on my development machine the memory usage seems to fluctuate between about 4MB and 13MB when the application is doing things (training neural networks) and at most it uses about 18MB. This is very good usage and I think it is due to the fact that I call System.GC() fairly regularly. I do this because the processing time doesn't matter for me, but the memory usage does.
So it all works fine on my machine, but as soon as I put it online on our server (shared unix hosting with memory limits) it uses about 19MB to start with and rises to hundreds of MB of memory usage when doing things. These are the same things that I have been doing in testing. The only way, I believe, to reduce the memory usage, is to quit the application and restart it.
The only difference that I can tell is the Java Virtual Machine that it is being run on. I do not know about this and I have tried to find the reason why it is acting this way, but a lot of the documentation assumes a great knowledge of Java and Virtual Machines. Could someone please help m with some reasons why this may be happening and perhaps some things to try to stop it.
I have looked at using GCJ to compile the application, but I don't know if this is something I should be putting a lot of time in to and whether it will actually help.
Thanks for the help!
UPDATE: Developing on Mac OS 10.6.3 and server is on a unix OS but I don't know what. (Server is from WebFaction)
I think it is due to the fact that I
call System.GC() fairly regularly
You should not do that, it's almost never useful.
A garbage collector works most efficiently when it has lots of memory to play with, so it will tend to use a large part of what it can get. I think all you need to do is to set the max heap size to something like 32MB with an -Xmx32m command line parameter - the default depends on whether the JVM believes it's running on a "server class" system, in which case it assumes that you want the application to use as much memory as it can in order to give better throughput.
BTW, if you're running on a 64 bit JVM on the server, it will legitimately need more memory (usually about 30%) than on a 32bit JVM due to larger references.
Two points you might consider:
Calls of System.gc can be disabled by a commandline parameter (-XX:-DisableExplicitGC), I think the behaviour also depends on the gc algorithm the vm uses. Normally invoking the gc should be left to the jvm
As long as there is enough memory available for the jvm I don't see anything wrong in using this memory to increase application and gc performance. As Michael Borgwardt said you can restrict the amount of memory the vm uses at the command line.
Also you may want to look at what mode the JVM has been started when you deploy it online. My guess its a server VM.
Take a look at the differences between the two right here on stackoverflow. Also, see what garbage collector is actually running on the actual deployment. See if you can tweek the GC behaviour, or change the GC algorithm.See the -X options if its a Sun JVM.
Basically the JVM takes the amount of memory it is allowed to as needed, in order to make the "new" operation as fast as possible (this is a science in itself).
So if you have a lot of objects being used, and then discarded, you will slowly and surely fill up the available memory. Then you can ask for garbage collection, but it is just a hint, and the JVM may choose not to listen.
So, you need another mechanism to keep memory usage down. The typical approach is to limit the amount of memory with -Xoptions, but be careful since the JVM you use on your pc may be very different from the one you deploy on, and the memory need may therefore be different.
Is there a deliberate requirement for low memory usage? If not, then just let it run and see how the JVM behaves. Use jvisualvm to attach and monitor.
Perhaps the server uses more memory because there is a higher load on your app and so more threads are in use? Jetty will use a number of threads to spread out the load if there are a lot of requests. Its worth a look at the thread count on the server versus on your test machine.
I've just upgraded some old Java source which has been running on a Sun Java 1.4.2 VM to Sun Java (JRE) 6 VM. More or less the only thing I had to change was to add explicit datatypes for some abstract objects (Hashmap's, Vector's and so on). The code itself it quite memory intensive, using up to 1G of heap memory (using -Xmx1024m as a parameter to start the VM).
Since I read alot about better performance on newer Java VM's, this was one of the reasons I did this upgrade.
Can anyone think of a reason why the performance is worse in my case now (just in general, of course, since you can't take a look at the code)?
Has anyone some advice for a non Java guru what to look for if I wanted to optimize (speed wise) existing code? Any hints, recommended docs, tools?
Thanks.
Not much information here. But here are a couple of things you might want to explore:
Start the VM with Xmx and Xms as the same value (in your case 1024M)
Ensure that the server jvm dll is being used to start the virtual machine.
Run a profiler to see what objects are hogging memory or what objects are not being garbage collected
Hook up your VM with the jconsole and trace through the objects
If your application nearly runs out of free space, garbage collection time may dominate computation time.
Enable gc debugging to look for this. Or, even better, simply start jconsole and attach it to your program.
Theoretically it could be that you application consumes more memory, because there were changes to the way Strings share their internal char[]. Less sharing is done after 1.4.
Check my old blog at http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/5100 (new blog is here)
I would compare the Garbage Collector logs to see whether memory usage is really the problem.
If that doesn't help, us a profiler such as Yourkit to find the differences.
Definitely use a profiler on the app (YourKit is great)...it's easy to waste a lot of time guessing at the problem when most of the time you'll be able to narrow it down really quickly in the profiler.