I have a network threaded application that I run under Eclipse (Indigo) and Javd 1.7x. For quite a while I have have noticed that the first run of the application produced front and end loaded degradation in performance, for example if I was to load up the application and and then hit it (using a test harness) with say 100 network packets the first few responses would be heavily erratic and the last few. [edit] Without unloading the application, and just running the test harness again, the application performs normally.[end edit]
I decided to get to try and get to the bottom of it and loaded up VisualVM 1.3.5 to profile the behaviour. The CPU Usage has a distinct spike going from 10% to over 50% at the beginning of the run. After the spikes, everything appears normal, and as stated above subsequent runs do not have the leading spikes in CPU Utilisation and the profile of subsequent runs is identical to the profile between the spikes of the first run. There doesn't appear to be any evidence that the number of threads is causing it, but there is a small rise. Heap space increases from 100MB to 200MB, but other than that everything appears normal.
Any thoughts would be welcome.
Thanks
Its fairly typical for system performance to be erratic the first time you run a test. This is due to the operating system reading libraries, JAR files, and other data off of disk and storing it in cache. Once this has been done that first time all subsequent runs will be much faster and more consistent.
Also, keep in mind that the JVM will also tend to be slower right after it starts up. Due to its hotspot analysis and just-in-time compiling, the code will need to run a little while before the JVM optimizes the byte code for your particular workload.
This is typical for OSGi environments, where bundles may be initialize lazily upon first access of a bundles class or services.
You can figure out if this is the case in your scenario by starting eclipse with -console -consolelog arguments.
When the console opens and the application was loaded, issue the ss command and note which bundles are marked LAZY. Then, run your test, issue ss again, and see if one of the LAZY bundles now became ACTIVE. If so, you can force eager start of your bundles via the configuration/config.ini file. This can also be accomplished via the IStartup extension point.
Related
I am developing a Java 8 SE application in Netbeans. A new feature I added recently to the app was running too slowly (about a minute, until the calculations stopped). So I fired up the profiler to see what is the major bottleneck. To my surprise, the calculations completed in about 7 seconds.
Couldn't believe it at first, but the results were correct.
Tried it a few times again, but the app always ran 10 times faster with the profiler attached to it. I also tried to run the compiled .jar file directly from the Windows command line, but the computations took about a minute again and again.
How is it possible, that the attached profiler provides such a massive boost to the performance? What changes does it do to the JVM or application?
BTW, I am using native OpenCV in these calculations with provided Java wrapper, if it makes any difference.
//Edit - Additional info: I am using the built-in Netbeans 8.1 profiler, which I believe is basically VisualVM. As for a profiling method I chose to monitor "Methods" and their execution times and invocation counts. The performance bump happens both with instrumented and sampled profiling.
Unfortunately there probably isn't one single answer that will explain why this is the case. Of course, it will depend on what the program is doing as well as how the program is being launched. For example, if you're using the profiler to launch the application (as opposed to connecting afterwards) then it may be that the profiler is launching with different configuration (heap size, garbage collector etc.) and that is the cause of the difference.
If you run jcmd you should see a list of processes. You can then run jcmd <id> VM.flags to see what the JVM has been configured with, and verify that the same are for the application when under a profiler and when it isn't.
Another possibility is that your program is excessively locking, and this excessive locking is causing thrashing in your application when the profiler isn't attached. With it attached the locking may be slower, resulting in the application threads co-operating and ultimately making faster progress.
However these are just suggestions of how you can investigate further; it's quite likely that there is another as yet undiscovered problem that you're seeing which is completely different (e.g. it's defaulting to a different level of logging ...)
Imagine you have command-line application that takes input file and does something with it. Now imagine you want to sample/profile this application. If it were Visual Studio you would just select profiling method (sampling/instrumentation) and VS would run application for you and collect data while program completes. But as far as I can see there is no similar functionality in VisualVM. You have to run your application, then select it in VisualVM and then explicitly start sampling/profiling. The problem is that sometimes execution of program with certain input data takes less time than it is required to setup VisualVM. Also with such an approach there is no possibility to batch profile application. Someone has suggested to start application in debug mode from Eclipse and set breakpoint somewhere in the beginning of main() method. Then setup VisualVM and continue execution. But I have suspicion that running in Debug vs Release mode has performance implications on its own.
Suggestions?
There is a new Startup Profiler plugin for VisualVM 1.3.6, which allows you to profile your application from its startup. See this article for additional information.
If the program does I/O, the Visual Studio sampler will not see the I/O because it is a "CPU Sampler" (even if nearly all of the time is spent waiting for I/O).
If you use Instrumentation, you won't see any line-level information because it only summarizes at the function level.
I use this technique.
If the program runs too quickly to sample, just put a temporary outer loop around it of, say, 100 or 1000 iterations.
The difference between Debug and Release mode will be next to nothing unless you are spending a good fraction of time in tight loops, in your code, where the loops do not contain any function calls, OR if you are doing data structure operations that do a lot of validation in the libraries.
If you are, then your samples will show that you are, and you will know that Release will make a speed difference.
As far as batch profiling is concerned, I don't. I just keep an eye on the program's overall throughput rate. If there is some input that seems to make it take too long, then I do the sampling procedure on the program with that input, see what the problem is, and fix it.
I'm in the process of benchmarking an app i've written. I ran my app through the benchmark 10 times in a loop (to get 10 results instead of only 1). Each time, the first iteration seems to take some 50 - 100 milliseconds longer than rest of the iterations.
Is this related to the JIT compiler and is there anything one could do to "reset" the state so that you would get the initial "lag" included with all iterations?
To benchmark a long running application you should allow an initialization (1st pass), thats because classes have to be loaded, code has to be generated, in web-apps JSP compile to servlets etc. JIT of course plays its role also. Sometimes a pass could take longer if garbage collection occurs.
It is probably caused by the JIT kicking in, however you probably want to ignore the initial lag anyway. At least most benchmarks try to, because it heavily distorts the statistics.
You can't "uncompile" code that has been compiled but you can turn compiling off completely by using the -Xint command line switch.
The first pass will probably always be slower because of the JIT. I'd even expect to see differences when more runs are made because of possible incremental compilation or better branch prediction.
For benchmarking, follow the recommondations given in the other answers (except I wouldn't turn off the JIT because you'd have your app running with JIT in a production environment).
In any case use a profiler such as JVisualVM (included in JDK).
Is this related to the JIT compiler
Probably yes, though there are other potential sources of "lag":
Bootstrapping the JVM and creation of the initial classloader.
Reading and loading the application's classes, and the library classes that are used.
Initializing the classes.
JIT compilation.
Heap warmup effects; e.g. the overheads of having a heap that is initially too small. (This can result on the GC running more often than normal ... until the heap reaches a size that matches the application's peak working set size.)
Virtual memory warmup effects; e.g. the OS overheads incurred when the JVM grows the process address space and physical pages are allocated.
... and is there anything one could do to "reset" the state so that you would get the initial "lag" included with all iterations?
There is nothing you can do, apart from starting the JVM over again.
However, there are things that you can do to remove some of these sources of "lag"; e.g. turning of JIT compilation, using a large initial heap size, and running on an otherwise idle machine.
Also, the link that #Joachim contributed above is worth a thorough read.
There are certain structures you might have in your code, such as singletons which are initialized only once and consume system resources. If you're using a database connection pool for example, this might be the case. Moreover it is the time needed by Java classes to be initialized. For these reasons, I think you should discard that first value and keep only the rest.
I have been writing a small java application (my first!), that does only a few things at the moment. Currently, it runs the Main class which launches a gui class (a class I wrote that extends JFrame that only contains a JTextArea), a class that loads a local file through a BufferedInputStream that is approximately 40kb, and class that loads a entry from a Java properties file.
Everything works wonderfully, however, I was watching the Windows task manager and I noticed something that struck me as odd. When I launch the application, the RAM usage jumps to about 40MB while it loads the local file and pulls a few values from it to display in the JTextArea, which seems normal to me because of the JVM, Java base classes, etc. At this point, however, when the application has finished loading the file, itmerely sits idle, as I currently don't have it doing anything else. While it is sitting idle, as long as the window is active, the application's memory usage starts climbing by 10-20kb every second. This strikes me as odd. If I click on another program to make this one the inactive window, the memory still rises, but at a much slower rate (about 10kb every 3-5 seconds).
I have not tested to see how far it would go up, but this strikes me as very odd behavior. Is this normal Java behavior? I guess it is possible that my code could be leaking memory, but I'm not sure how. I did make sure to close the BufferedInputStream I am using, and I can't see what else would cause this.
I'm sorry if my explanation doesn't make sense, but I would appreciate any insight and/or pointers anyone may have.
UPDATE:
Upon suggestion, I basically stripped my application down to the Main class which simply calls the gui class. The gui class only extends JFrame and sets the window size, close operation, and visible properties. With these changes, the memory still grows at 10-20kb, but at a slower rate. This, in conjuction with other advice I have received leads me to believe that this is just Java. I will continue to play with it and let you all know if I find out anything else interesting.
Try monitoring the heap usage with jconsole instead of the Windows task manager:
Launch your app with the -Dcom.sun.management.jmxremote option e.g.
java -Dcom.sun.management.jmxremote -jar myapp.jar
Launch jconsole from the command line, and connect to the local pid of the java process you started in the last step.
Click over to memory and watch heap memory (the default display)
If you watch for a while, you'll probably get a "sawtooth" pattern as the memory climbs over time, but then has sharp drop-offs when the garbage collector runs. You can try to "suggest" garbage collection by clicking the so-labelled button.
When you do this, does the memory usage drop down to the same minimum level, or is the overall minimum increasing over the course of several minutes? If the minimum usage increases, then you have a memory leak. If it always returns to the same minimum level, then you're fine.
Congrats on your first app! Now, a couple things to think about. First, the Windows task manager is not a great resource to understand how quickly your vm is growing. Instead, you should monitor your garbage collection stats in the console (use the -verbose:gc commandline param). Second, if you are concerned about potential leaks and the growth of the vm, there are a bunch of great profilers out there that are easy to use and can help you diagnose memory issues. check out these two posts for some profiler options.
Congratulations for your first Java app!
Java applications run in a virtual machine. The virtual machine has been assigned a fixed amount of memory by the OS, typically 512 MB. As long as the application uses less than 512 MB the garbage collector won't kick in and start searching for "dead" memory blocks. The JVM memory limit can be modified in most OSes. Try switching the memory limit to 32 MB, for example.
Is this normal Java behavior?
No.
I guess it is possible that my code could be leaking memory
That is definitely the cause. Please post your source code, otherwise further diagnosis isn't possible.
I noticed you are using Swing, make sure you are launching your JFrame in the event dispatch thread, using the invokeLater(Runnable) method.
If your are using any sort of collections, make sure you clear them once done.
Since you are doing some file IO, make sure you close all of the classes involved in in the IO operations after you are done with them.
If you are using any event listeners, remember to explicitly remove event listeners when they are no longer necessary.
One thing you could try is experimenting. Take your application and remove the file IO, see what happens. Does the memory usage still climb as before? Now resotre your application to normal, and remove the text area - does the memory still climb as before? Etc, etc. This will help you to determine what the source is, and you can focus your efforts there. Most likely you will uncover what you are after by doing this.
Another useful diagnosis tool is to use System.gc() at particular points in time, usually after the heavy-lifting blocks of code. This will tell the JVM to perform a garbage collection at that point in the execution, rather than at another time determined by memory consumption. This will help you to take into account any periodic fluctuations in the memory usage of your application.
Failing which, you can always use a memory profiler. If you are using Netbeans IDE, there's one built right into it. For Eclipse, there're several plugins which can perform profiling.
it is normal. some background calc might leave dead objects around, which JVM isn't in a hurry to clean up. eventually they will be garbage collected, when max mem is approached.
leave your program running overnight, and your machine won't blow up.
Is there any Java profiler that allows profiling short-lived applications? The profilers I found so far seem to work with applications that keep running until user termination. However, I want to profile applications that work like command-line utilities, it runs and exits immediately. Tools like visualvm or NetBeans Profiler do not even recognize that the application was ran.
I am looking for something similar to Python's cProfile, in that the profiler result is returned when the application exits.
You can profile your application using the JVM builtin HPROF.
It provides two methods:
sampling the active methods on the stack
timing method execution times using injected bytecode (BCI, byte codee injection)
Sampling
This method reveals how often methods were found on top of the stack.
java -agentlib:hprof=cpu=samples,file=profile.txt ...
Timing
This method counts the actual invocations of a method. The instrumenting code has been injected by the JVM beforehand.
java -agentlib:hprof=cpu=times,file=profile.txt ...
Note: this method will slow down the execution time drastically.
For both methods, the default filename is java.hprof.txt if the file= option is not present.
Full help can be obtained using java -agentlib:hprof=help or can be found on Oracles documentation
Sun Java 6 has the java -Xprof switch that'll give you some profiling data.
-Xprof output cpu profiling data
A program running 30 seconds is not shortlived. What you want is a profiler which can start your program instead of you having to attach to a running system. I believe most profilers can do that, but you would most likely like one integrated in an IDE the best. Have a look at Netbeans.
Profiling a short running Java applications has a couple of technical difficulties:
Profiling tools typically work by sampling the processor's SP or PC register periodically to see where the application is currently executing. If your application is short-lived, insufficient samples may be taken to get an accurate picture.
You can address this by modifying the application to run a number of times in a loop, as suggested by #Mike. You'll have problems if your app calls System.exit(), but the main problem is ...
The performance characteristics of a short-lived Java application are likely to be distorted by JVM warm-up effects. A lot of time will be spent in loading the classes required by your app. Then your code (and library code) will be interpreted for a bit, until the JIT compiler has figured out what needs to be compiled to native code. Finally, the JIT compiler will spend time doing its work.
I don't know if profilers attempt to compensate to for JVM warmup effects. But even if they do, these effects influence your applications real behavior, and there is not a great deal that the application developer can do to mitigate them.
Returning to my previous point ... if you run a short lived app in a loop you are actually doing something that modifies its normal execution pattern and removes the JVM warmup component. So when you optimize the method that takes (say) 50% of the execution time in the modified app, that is really 50% of the time excluding JVM warmup. If JVM warmup is using (say) 80% of the execution time when the app is executed normally, you are actually optimizing 50% of 20% ... and that is not worth the effort.
If it doesn't take long enough, just wrap a loop around it, an infinite loop if you like. That will have no effect on the inclusive time percentages spent either in functions or in lines of code. Then, given that it's taking plenty of time, I just rely on this technique. That tells which lines of code, whether they are function calls or not, are costing the highest percentage of time and would therefore gain the most if they could be avoided.
start your application with profiling turned on, waiting for profiler to attach. Any profiler that conforms to Java profiling architecture should work. i've tried this with NetBeans's profiler.
basically, when your application starts, it waits for a profiler to be attached before execution. So, technically even line of code execution can be profiled.
with this approach, you can profile all kinds of things from threads, memory, cpu, method/class invocation times/duration...
http://profiler.netbeans.org/
The SD Java Profiler can capture statement block execution-count data no matter how short your run is. Relative execution counts will tell you where the time is spent.
You can use a measurement (metering) recording: http://www.jinspired.com/site/case-study-scala-compiler-part-9
You can also inspect the resulting snapshots: http://www.jinspired.com/site/case-study-scala-compiler-part-10
Disclaimer: I am the architect of JXInsight/OpenCore.
I suggest you try yourkit. It can profile from the start and dump the results when the program finishes. You have to pay for it but you can get an eval license or use the EAP version without one. (Time limited)
YourKit can take a snapshot of a profile session, which can be later analyzed in the YourKit GUI. I use this to feature to profile a command-line short-lived application I work on. See my answer to this question for details.