The root of our problem is Singletons. But Singletons are hard to break and in the meantime we have a lot of unit tests that use Singletons without being careful to completely clear them in the tearDown() method. I figure that a good way to detect tests like these is to look for memory leaks. If the memory used after tearDown() and System.gc() is more than when the test started, either the test leaked or more classes were loaded by the classloader. Is there any way to automatically detect this sort of problem?
Could you introduce a subclass, between TestCase and your individual test classes, that did the cleanup? Then subclasses would only be responsible for calling super.teardown() - and only those that had a teardown() of their own.
I completely agree with other posters that monitoring the memory usage isn't a viable way to track this - System.gc() is not going to behave as you expect, or with enough precision to achieve your goal.
You're going to need a tool that lets you inspect the reference graph and show allocation call stacks.
I've used OptimizeIt from Borland and JProfiler from ej-technologies, both with success (a quick google reveals that OptimizeIt may now be dead.)
There's also the possiblity of using JVMTI to throw together a better monitor for this specific problem.
Edit: Wierd, but as I was reviewing this answer, I got a phone call from Embarcadero, who has apparently purchased OptimizeIt, done some updating and are now marketing under the name J Optimizer.
Just a thought: if you have two empty tests run right after one another, the second one should not have a different memory used after teardown(). If it does, you (probably) have a leak somewhere in your setup()/teardown() system.
I don't think this is a good approach. System.gc() is not guaranteed to fully clean up any unused objects as you think it will.
If your root problem is that you have unit tests which end up using global data (singletons) without properly cleaning them up, you should attack the root problem: these unit tests. It shouldn't be too too hard to find all tests that aren't using tearDown(), or to find all tests that use a particular singleton.
If your Singleton's are only intended to be initialized one time, you could have code that checks for reinitialization and logs the current stack when it detects that. Then if you check the stack, you will see which test got the ball rolling, and you can check the JUnit logs to see what the test run right before that was.
In terms of solving this problem more thoroughly, instead of detecting it I would recommend you have a singleton initializer that remembers what it initialized, and has one teardown method that tears down everything it initialized. That way tests can be made to only initialize via this class, and only has to do one thing in teardown.
I also think Carl Manaster's suggestion is a good one, but if you were using JUnit4, then you could have a teardown method that runs in the superclass without having to remember to call super. Unless you use the JUnit3 GUI, JUnit4 should be a drop in replacement. The only thing is to take advantage of its new features you have to migrate the whole test, you can't have both live in the same class. So tests that interact with these singletons would have to be migrated one whole test class at a time.
You could use the Eclipse Memory Analyzer to automate analyzing heap dumps taken after each test or probably better after all tests. MAT can find memory leaks fairly automatically.
Related
I have been assigned with testing a mongodb database from a java backend. I got told that I had to create a database completely utilizing a script for this task.
But I have difficulties understanding the benefit of creating a database from scratch with a script, instead of having a permanent test database. Where I imagine data should be inserted on startup and cleaned on teardown in both cases.
Why is it beneficial from a testing perspective to creating and deleting a database when testing?
Sometimes tests fail and therefore it may happen that the teardown phase will never be reached.
Furthermore, deleting a database is the fastest and most effective way to clean it, although perhaps not the most efficient way to do so. But it guarantees that you do not forget something in your cleanup routine.
And in particular for performance tests it is important that the database is in exact the same state for each run, otherwise the run times cannot be compared with each other: an improvement in a consecutive run could have been caused just because tablespaces were already increased or similar things, and not because the code optimisation worked …
most of the times test means a predefined environment and a expected reaction of that environment against our assumed states. so for verifying it we need a pure automated and repeatable process as much as possible without interference of manual setup or configuration.
In software development process we try to consider as many as possible test cases for QA of product. when we talk about too many test cases each one should be isolated from the others. if it's not isolated well the result may varies in each execution round and eventually invalid testing process.
They need not be. However:
You lose portability.
You don't have a known start state for your test.
When i start a JVM in debug mode things naturally slow down.
Is there a way to state that i am interested in only debugging a single application instead of the 15 (making up a number here) applications that run on this JVM.
An approach that facilitates this might make things faster particularly when we already know from the logs and other trace facilities that the likely issue with a single application
Appreciate thoughts and comments
Thanks
Manglu
I am going to make a lot of assumptions here, especially as your question is missing a lot of contextual information.
Is there a way to state that i am interested in only debugging a single application instead of the 15 (making up a number here) applications that run on this JVM.
Firstly, I will assume that you are attempting to do this in production. If so, step back and think what could go wrong. You might be putting a single breakpoint, but that will queue up all the requests arriving at that breakpoint, and by doing so you've thrown any SLA requirements out of the window. And, if your application is handling any sensitive data, you must have seen something that you were not supposed to be seeing.
Secondly, even if you were doing this on a shared development or testing environment this is a bad idea. Especially if are unsure of what you are looking for. If you are hunting a synchronization bug, then this is possibly the wrong way to do so; other threads will obviously be sharing data that you are reading and make it less likely to find the culprit.
The best alternative to this is to switch on trace logging in your application. This will, of course be useless, unless you have embedded the appropriate logger calls in your application (especially to trace method arguments and return values). With trace logs at your disposal, you should be able to create an integration or unit test that will reproduce the exact conditions of failure on your local developer installation; this is where you ought to be doing your debugging. Sometimes, even a functional test will suffice.
There is no faster approach in general, as it is simply not applicable to all situations. It is possible for you to establish a selected number of breakpoints in any of the other environments, but it simply isn't worth the trouble, unless you know that only your requests are being intercepted by the debuggee process.
All,
Recently I developed a code that supposedly is a thread-safe class. Now the reason I have said 'supposedly' is because even after using the sync'ed blocks, immutable data structures and concurrent classes, I was not able to test the code for some cases because of the thread scheduling environment of JVM. i.e. I only had test cases on paper but could not replicate the same test environment. Is there any specific guidelines or something the experienced members over here who can share about how to test a multi-threaded environment.
First thing is, you can't ensure only with testing that your class is fully thread-safe. Whatever tests you run on it, you still need to have your code reviewed by as many experienced eyes as you can get, to detect subtle concurrency issues.
That said, you can devise specific test scenarios to try to cover all possible inter-thread timing scenarios, as you did. For ideas on this (and for designing thread-safe classes in general), it is recommended to read Java Concurrency in Practice.
Moreover, you can run stress tests, executing many threads simultaneously over an extended period of time. The number of threads should be way over the reasonable limit to make sure that thread contention happens often - this raises the chances of potential concurrency bugs to manifest over time.
Also, another thing I would recomend is for you to use code coverage measuring tools and set a high standar as your goal. For example, set a high goal for modified condition/decision coverage.
We use GroboUtils to create multi threaded tests.
If you have code that you plan to test in order to make it reliable, then make it single threaded.
Threading should be reserved for code that either doesn't particularly need to work, or is simple enough to be statically analysed and proven correct without testing.
I am trying to reproduce java.lang.OutOfMemoryException in Jboss4, which one of our client got, presumably by running the J2EE applications over days/weeks.
I am trying to find a way for the webapp to spitout java.lang.OutOfMemoryException in a matter of minutes (instead of days/weeks).
One thing come into mind is to write a selenium script and has the script bombards the webapps.
One other thing that we can do is to reduce JVM heap size, but we would prefer not to do this, as we want to see the limit of our system.
Any suggestions?
ps: I don't have access to the source code, as we just provide a hosting service (of course I could decompile the class files...)
If you don't have access to the source code of the J2EE app in question, the options that come to mind are:
Reduce the amount of RAM available to the JVM. You've already identified this one and said you don't want to do it.
Create a J2EE app (it could probably just be a JSP) and configure it to run within the same JVM as the target app, and have that app allocate a ridiculous amount of memory. That will reduce the amount of memory available to the target app, hopefully such that it fails in the way you're trying to force.
Try to use some profiling tools to investigate memory leakage. Also good to investigate memory damps that was taken after OOM happens and logs. IMHO: reducing memory is not the rightest way to investigate cose you can get issues not connected with real production one.
Do both, but in a controlled fashion :
Reduce the available memory to the absolute minimum (using -Xms1M -Xmx2M, as an example, but I fear your app won't even load with such limitations)
Do controlled "nuclear irradiation" : do Selenium scripts or each of the known working urls before to attack the presumed guilty one.
Finally, unleash the power that shall not be raised : start VisualVM and any other monitoring software you can think of (DB execution is a usual suspect).
If you are using Sun Java 6, you may want to consider attaching to the application with jvisualvm in the JDK. This will allow you to do in-place profiling without needing to alter anything in your scenario, and may possibly immediately reveal the culprit.
If you don't have the source use decompile it, at least if you think the terms of usage allows this and you live in a free country. You can use:
Java Decompiler or JAD.
In addition to all the others I must say that even if you can reproduce an OutOfMemory error, and find out where it occurred, you probably haven't found out anything worth knowing.
The trouble is that an OOM occurs when an allocation can not take place. The real problem however is not that allocation, but the fact that other allocations, in other parts of the code, have not been de-allocated (de-referenced and garbage collected). The failed allocation here might have nothing to do with the source of the trouble (no pun intended).
This problem is larger in your case as it might take weeks before trouble starts, suggesting either a sparsely used application, or an abnormal code path, or a relatively HUGE amount of memory in relation to what would be necessary if the code was OK.
It might be a good idea to ask around why this amount of memory is configured for JBoss and not something different. If it's recommended by the supplier than maybe they already know about the leak and require this to mitigate the effects of the bug.
For these kind of errors it really pays to have some idea in which code path the problem occurs so you can do targeted tests. And test with a profiler so you can see during run-time which objects (Lists, Maps and such) are growing without shrinking.
That would give you a chance to decompile the correct classes and see what's wrong with them. (Closing or cleaning in a try block and not a finally block perhaps).
In any case, good luck. I think I'd prefer to find a needle in a haystack. When you find the needle you at least know you have found it:)
The root of the problem is most likely a memory leak in the webapp that the client is running. In order to track it down, you need to run the app with a representative workload with memory profiling enabled. Take some snapshots, and then use the profiler to compare the snapshots to see where objects are leaking. While source-code would be ideal, you should be able to at least figure out where the leaking objects are being allocated. Then you need to track down the cause.
However, if your customer won't release binaries so that you can run an identical system to what he is running, you are kind of stuck, and you'll need to get the customer to do the profiling and leak detection himself.
BTW - there is not a lot of point causing the webapp to throw an OutOfMemoryError. It won't tell you why it is happening, and without understanding "why" you cannot do much about it.
EDIT
There is not point "measuring the limits", if the root cause of the memory leak is in the client's code. Assuming that you are providing a servlet hosting service, the best thing to do is to provide the client with instructions on how to debug memory leaks ... and step out of the way. And if they have a support contract that requires you to (in effect) debug their code, they ought to provide you with the source code to do your job.
I recently began profiling an osgi java application that I am writing using VisualVM. One thing I have noticed is that when the application starts sending data to a client (over JMS), the number of loaded classes starts increasing at a steady rate. The Heap size and the PermGen size remains constant, however. The number of classes never falls, even after it stops sending data. Is this a memory leak? I think it is, because the loaded classes have to be stored somewhere, however the heap and permgen never increase even after I run the application for several hours.
For the screenshot of my profiling application go here
Are you dynamically creating new classes on the fly somehow?
Thanks for your help. I figured out what the problem is. In one of my classes, I was using Jaxb to create an XML string. In doing this, JAXB ueses reflection to create a new class.
JAXBContext context = JAXBContext.newInstance(this.getClass());
So although the JAXBContext wasn't saying around in the heap, the classes had been loaded.
I have run my program again, and I see a normal plateau as I would expect.
I'm willing to bet that your problem is related to bytecode generation.
Many libraries use CGLib, BCEL, Javasist or Janino to generate bytecode for new classes at runtime and then load them from controlled classloader. The only way to release these classes is to release all references to the classloader.
Since the classloader is held by each class, this also means that you should not release the references to all classes as well [1]. You can catch these with a decent profiler (I use Yourkit - search for multiple classloader instances with the same retained size)
One catch is that the JVM does not unload classes by default (the reason is backwards compatibility - that people assume (wrongly) that static initializers would be executed only once. The truth is that they get executed every time a class is loaded.) To enable unloading, you should pass some use the following options:
-XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
(tested with JDK 1.5)
Even then, excessive bytecode generation is not a good idea, so I suggest you look in your code to find the culprit and cache the generated classes. Frequent offenders are scripting languages, dynamic proxies (including the ones generated by application servers) or huge Hibernate model (in this case you can just increase your permgen).
See also:
http://blogs.oracle.com/watt/resource/jvm-options-list.html
http://blogs.oracle.com/jonthecollector/entry/presenting_the_permanent_generation
http://forums.sun.com/thread.jspa?messageID=2833028
You might find some hotspot flags to be of use in understanding this behavior like:
-XX:+TraceClassLoading
-XX:+TraceClassUnloading
This is a good reference:
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
Unless I misunderstand, we're looking here at loaded classes, not instances.
When your code first references a class, the JVM has the ClassLoader go out and fetch the information about the class from a .class file or the like.
I'm not sure under what conditions it would unload a class. Certainly it should never unload any class with static information.
So I would expect a pattern roughly like yours, where as your application runs it goes into areas and references new classes, so the number of loaded classes would go up and up.
However, two things seems strange to me:
Why is it so linear?
Why doesn't it plateau?
I would expect it to trend upwards, but in a wobbly line, and then taper off on the increase as the JVM has already loaded most of the classes your program references. I mean, there are a finite number of classes referenced in most applications.
Are you dynamically creating new classes on the fly somehow?
I would suggest running a simpler test app through the same debugger to get a baseline case. Then you could consider implementing your own ClassLoader that spits out some debug information, or maybe there is a tool to make it report.
You need to figure out what these classes being loaded are.
Yes, it's usually a memory leak (since we don't really deal with memory directly, it's more of a class instance leak). I've gone through this process before and usually it's some listener added to an old toolkit that didn't remove it self.
In older code, A listener relationship causes the "listener" object to remain around. I'd look at older toolkits or ones that haven't been through many revs. Any long-existing library running on a later JDK would know about reference objects which removes the requirement for "Remove Listener".
Also, call dispose on your windows if you recreate them each time. I don't think they ever go away if you don't (Actually there is also a dispose on close setting).
Don't worry about Swing or JDK listeners, they should all use references so you should be okay.
Use the Eclipse Memory Analyzer to check for duplicated classes and memory leaks. It might happen that the same class gets loaded more than once.
Regards,
Markus