I need to optimise a Java application. It makes some 3rd party calls. I need some good tool to accurately measure the time taken by individual API calls.
To give an idea of complexity-
the application takes a data source file containing 1 million rows, and it takes around one hour to complete the processing. As a part of processing , it makes some 3rd party calls (including some network calls). I need to identify which calls are taking more time then others, and based on that, find out a way to optimise the application.
Any suggestions would be appreciated.
I can recommend JVisualVM. It's a great monitoring / profiling tool that is bundled with the Oracle/Sun JDK. Just fire it up, connect to your application and start the CPU-profiling. You should get great histograms over where the time is spent.
Getting Started with VisualVM has a great screen-cast showing you how to work with it.
Screen shot:
Another more rudimentary alternative is to go with the -Xprof command line option:
-Xprof
Profiles the running program, and sends profiling data to
standard output. This option is provided as a utility that is
useful in program development and is not intended to be be
used in production systems.
I've been using YourKit a few times and what quite happy with it. I've however never profiled a long-running operation.
Is the processing the same for each row? In which case the size of the input file doesn't really matter. You could profile a subset to figure out which calls are expensive.
Just wanted to mention the inspectIT tool. It recently became completely open source (https://github.com/inspectIT/inspectIT). It provides complete and detailed call graph with contextual information, there are many out-of the box sensor for database calls, http monitoring, exceptions, etc.
Seams perfect for your use-case..
Try OPNET's Panorama software product
It sounds like a normal profiler might not be the right tool in this case, since they're geared towards measuring the CPU time taken by the program being profiled rather than external APIs that it calls, and they tend to incur a high overhead of their own and collect a large amount of data that would probably overwhelm your system if left running for a long time.
If you really need to collect performance data over such a long time, and mainly for external calls, then Perf4J is probably a better tool.
In our office we use YourKit profiler on a day to day basis. It's really light weight and serves most of the performance related use cases we have had.
But I have also used Visual VM. It's free and fast. You may first want to give Visual VM a try before going towards YourKit (YourKit is not freeware).
visualvm (part of the SDK) and Java 7 can produce detailed profiling.
I use profiler in NetBeans (it is really brilliant and already built in, no need to install plugin) or JVisualVM when not using NetBeans.
Related
I have started to use visualvm for profiling my application which I launch in Eclipse. Then I launch visualvm which initially gives believable results.
After some time two processes appear in the monitor which consume huge amounts of time.
I have not deliberately invoked these. After a time they disappear. Are they an artefact of the profiling process and do I need to worry?
Very few of my routines appear in the profile, mainly the libraries they call. Is there a way of showing which routines call the most heavily used ones?
It is better to start with CPU sampling, if you don't know which part of the code is slow. Once you know better (based on the sampling results) what is going on, you can profile just part of your application, which is slow. You need to set profiling roots and instrumentation filter and don't forget to take the snapshot of collected results. See Profiling With VisualVM, Part 1 and Profiling With VisualVM, Part 2 to get more information about profiling and how to set profiling roots and instrumentation filter.
VisualVM uses Java to perform it's work. This means you will see some artefacts which relate to the RMI calls it makes. You can ignore them.
I use YourKit which doesn't do this, but it's not free ;)
VisualVM will track all methods being called by the java program it's monitoring, so either your program or one of its libraries is calling those methods. VisualVM is also connecting to it so there might be some small artifacts.
As for searching, probably the easiest way is to filter by your own packages. There is a space at the bottom where you can enter those so you can see which of your own methods is really taking time. Also you should pay attention to which thread you are in, usually you will want to look at the whatever is your "main" thread. The other threads are interesting but won't always give you the best idea of how your program behaves.
I am working on a Java Swing Based application using Hibernate. And i have used NetBeans IDE to build my application.
I am having around 13 sub-modules. I have used JInternalFrames to build those sub-modules. And 4 reports which use JasperViewer.
While loading my application it takes around 2 - 3 minutes. And when switching from one module to another modules it takes much time to load each modules.
Now i have to check following things
How to check Memory and CPU of my whole Swing application and which class takes much CPU and Memory usage?
How to test my application using simple approaches?
How to increase the performance of my swing application?
Startup times like these are not simply caused by having-a-big-gui-application. I think the problem is your database access. Questions come up:
Do you use the lazy loading feature of hibernate? This is a nice feature but can cause some issues, if not used responsible. Read this.
Do you cache data from the database in your application or are they fetched from the db on every action? A good way to reduce loading times is to store data in a HashMap. Hibernate has also it's own caches.
Do you fetch a large part of your whole database? This is not wanted, a better db schema design can prevent this, also lazy loading.
Is your application multithreaded? Loading the data off the event dispatch thread can by a big benefit.
To your questions:
The simpliest tool comes with every JDK: jVisualVM You can store .hprof files and also analyze them in an other tool. Memory Analyzer Tools or the Netbeans profiler are also a good choice. You can also read this for further information.
This depends on what you want to test. Just classes (JUnit), the GUI (UISpec4J), or do you mean profiling instead of testing?
You can increase the performance by caching data, using lazy loading, or making your application multithreaded as mentioned above. You can also read here how to achieve this.
As you are using NetBeans - use the internal profiler. That's the tool to get detailed information about where you lose the time.
For more information check the NetBeans site, e.g.
http://netbeans.org/kb/docs/java/profiler-intro.html
http://profiler.netbeans.org/
Regarding your questions:
for 3. First measure, then cut. In most cases some simple changes can gain you 80% performance improvement.
for 1. VisualVM is a great and free tool, which will help you measure both CPU and memory consumption. Than change exactly (and only) that code accordingly.
for 2. I would suggest to take a look at ReTest. It is a relatively new tool that implements an innovative approach to functional regression testing combined with ai-based monkey testing.
Disclaimer: I am one of the founders of the company behind ReTest.
How can we increase the performance of an application. My application is written using Java, Hibernate, Servlets, Wsdl i have used for web services. I have executed some of the tests on linux machine, so that i can get proper TPS of the execution.
but still , i am not satisfied by the performance.
So for this, what all steps i should try to increase the performance.
adding to above, i have executed code coverage and used find bugs in the code prominently for each and every test and every service i have written.
Individual suggestions are invited.
Thanks.
Profile your application, and remove all of your bottlenecks.
In addition, or better before, take a day or two and read as much from the Java Performance Tuning newsletters as you understand.
You should monitor your application with a tool like VisualVM, JProfiler etc. to determine the performance bottleneck(s). It is pointless to tune the application without knowing where the actual performance problems are located.
In a professional environment, I suggest dynaTrace that can show you performance bottlenecks along the execution path. The tool can show you exactly where the application spends its time.
Is the performance related to disk I/O or network I/O? In a high throughput system (from DB point of view) Hibernate might not be the best way to go. If you have a lot of writes I would recommend you use a different mechanism to write to database -- perhaps simply switching to simple JDBC might speed it up?
Secondly, is it the case that your webservices are taking too long to get back with results? SOAP is not the fastest protocols really -- have you looked at something like REST maybe coupled with JSON ?
I'm currently using VisualVM, but the problem I'm having is that I can't save the graphs it generates. I need to report some data about its memory usage and running time, though running time is easy to get with System.nanoTime(). I've also tried the NetBeans profiler but it isn't what I want, since I'm not looking for specific parts that would be slowing it down or anything, so that would be overkill. The biggest problem with it is that it eats up too much processing time. Also doesn't let me capture/transfer the data easily, like VisualVM, at least as far as I can tell.
Ideally the best way to go about it would be some method call because then I'd be able to get the information a lot more easily, but anything like VisualVM that actually lets me save the graph is fine. Performance with VisualVM is pretty good too, compared to the NetBeans profiler, though I suppose that's because I wasn't using its profiler.
I'm currently using Ubuntu, but Windows 7 is fine. I'd rather have a program that specializes in doing this though, since the information gotten by programs who don't is likely to include the JVM and other things that would be better left out.
Well, apparently, you can save snapshots of the current session and maximize the window in VisualVM, so you could make the charts bigger, take a snapshot and cut them... But that's kind of a hack. Better suggestions welcome.
Runtime.getRuntime().freeMemory();
Runtime.getRuntime().totalMemory();
Look at the Runtime class. It has freeMemory, maxMemory, and totalMemory. That's probably close enough for your purposes.
You may prefer graceful method to measure memory, rather than hack image.
JConsole is known to Monitor Applications by JMX,it provides program API. I guess it is what you need.
See: Using JConsole to Monitor Applications
Try JProfiler. Although its not free you can try evaluation version first.
The HPjmeter console is free. Run your Java process with -Xloggc:<file> and open the <file> with it. Not only can you save your sessions, but you can compare runs. Other options to consider including in your command line are:
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
I am writing a simple checkers game in Java. When I mouse over the board my processor ramps up to 50% (100% on a core).
I would like to find out what part of my code(assuming its my fault) is executing during this.
I have tried debugging, but step-through debugging doesn't work very well in this case.
Is there any tool that can tell me where my problem lies? I am currently using Eclipse.
This is called "profiling". Your IDE probably comes with one: see Open Source Profilers in Java.
Use a profiler (e.g yourkit )
Profiling? I don't know what IDE you are using, but Eclipse has a decent proflier and there is also a list of some open-source profilers at java-source.
In a nutshell, profilers will tell you which part of your program is being called how many often.
I don't profile my programs much, so I don't have too much experience, but I have played around with the NetBeans IDE profiler when I was testing it out. (I usually use Eclipse as well. I will also look into the profiling features in Eclipse.)
The NetBeans profiler will tell you which thread was executing for how long, and which methods have been called how long, and will give you bar graphs to show how much time each method has taken. This should give you a hint as to which method is causing problems. You can take a look at the Java profiler that the NetBeans IDE provides, if you are curious.
Profiling is a technique which is usually used to measure which parts of a program is taking up a lot of execution time, which in turn can be used to evaluate whether or not performing optimizations would be beneficial to increase the performance of a program.
Good luck!
1) It is your fault :)
2) If you're using eclipse or netbeans, try using the profiling features -- it should pretty quickly tell you where your code is spending a lot of time.
3) failing that, add console output where you think the inner loop is -- you should be able to find it quickly.
Yes, there are such tools: you have to profile the code. You can either try TPTP in eclipse or perhaps try JProfiler. That will let you see what is being called and how often.
Use a profiler. There are many. Here is a list: http://java-source.net/open-source/profilers.
For example you can use JIP, a java coded profiler.
Clover will give a nice report showing hit counts for each line and branch. For example, this line was executed 7 times.
Plugins for Eclipse, Maven, Ant and IDEA are available. It is free for open source, or you can get a 30 day evaluation license.
If you're using Sun Java 6, then the most recent JDK releases come with JVisualVM in the bin directory. This is a capable monitoring and profiling tool that will require very little effort to use - you don't even need to start your program with special parameters - JVisualVM simply lists all the currently running java processes and you choose the one you want to play with.
This tool will tell you which methods are using all the processor time.
There are plenty of more powerful tools out there, but have a play with a free one first. Then, when you read about what other features are available out there, you'll have an inking about how they might help you.
This is a typically 'High CPU' problem.
There are two kind of high CPU problems
a) Where on thread is using 100% CPU of one core (This is your scenario)
b) CPU usage is 'abnormally high' when we execute certain actions. In such cases CPU may not be 100% but will be abnormally high. Typically this happens when we have CPU intensive operations in the code like XML parsing, serialization de-serialization etc.
Case (a) is easy to analyze. When you experience 100% CPU 5-6 thread dumps in 30 sec interval. Look for a thread which is active (in "runnable" state) and which is inside the same method (you can infer that by monitoring the thread stack). Most probably that you will see a 'busy wait' (see code below for an example)
while(true){
if(status) break;
// Thread.sleep(60000); // such a statement would have avoided busy wait
}
Case (b) also can be analyzed using thread dumps taken in equal interval. If you are lucky you will be able to find out the problem code, If you are not able to identify the problem code by using thread dump. You need to resort to profilers. In my experience YourKit profiler is very good.
I always try with thread dumps first. Profilers will only be last resort. In 80% of the cases we will be able to identify using thread dumps.
Or use JUnit test cases and a code coverage tool for some common components of yours. If there are components that call other components, you'll quickly see those executed many more times.
I use Clover with JUnit test cases, but for open-source, I hear EMMA is pretty good.
In single-threaded code, I find adding some statements like this:
System.out.println("A: "+ System.currentTimeMillis());
is simpler and as effective as using a profiler. You can soon narrow down the part of the code causing the problem.