One day our java web application goes up to 100% CPU usage.
A restart solve the incident but not the problem because a few hours after the problem came back.
We suspected a infinite loop introduced by a new version but we didn't make any change on the code or on the server.
We managed to find the problem by making several thread dumps with kill -QUIT and by looking and comparing every thread details.
We found that one thread call stack appear in all the thread dumps.
After analysis, there was a while loop condition that never go false for some data that was regularly updated in the database.
The analysis of several thread dumps of web application is really tedious.
So do you know any better way or tools to find such issue in a production environment ?
After some queries, I found an answer in Monitoring and Managing Java SE 6 Platform Applications :
You can diagnose looping thread by using JDK’s provided tool called JTop that will show the CPU time each thread is using:
With the thread name, you can find the stack trace of this thread in the “Threads” tab of by making a thread dump with a kill -QUIT.
You can now focus on the code that cause the infinite loop.
PS.: It seems OK to answer my own question according to https://blog.stackoverflow.com/2008/07/stack-overflow-private-beta-begins/ :
[…]
“yes, it is OK and even encouraged to answer your own questions, if you find a good answer before anyone else.”
[…]
PS.: In case sun.com domain will no longer exists:
You can run JTop as a stand-alone GUI:
$ <JDK>/bin/java -jar <JDK>/demo/management/JTop/JTop.jar
Alternately, you can run it as a JConsole plug-in:
$ <JDK>/bin/jconsole -pluginpath <JDK>/demo/management/JTop/JTop.jar
Fix the problem before it occurs! Use a static analysis tool like FindBugs or PMD as part of your build system. It won't find everything, but it is a good first step.
Think of using coverage tools like Cobertura.
It would have shown you, that you didn't test these code-paths.
Testing sth. like this can become really cumbersome, so try to avoid this by introducing quality measurements.
Anyways tools like VisualVM will give you a nice overview of all threads, so it becomes relatively easy to identify threads which are working for an unexpectedly long time.
Related
We have recently installed a SonarQube instance to check our source code.
The codebase is pretty large, with more than 1 million lines of code.
We run sonar-runner automatically via Jenkins.
Now I get that the UI gets updates only after sonar-runner stores its results in the database.
But it seems to really take ages sometimes, up to an hour after the success of sonar-runner before we are able to see anything coming in the UI.
So I have a couple questions, all related :
Is there a way to see analysis that are still 'in the pipes'?
Where can I see whether the conversion from database to the UI has failed?
Is there a way to speed the process?
So if I summarize? How can I impact the sonar-runner to sonar UI latency?
I went through all the docs but couldn´t find much about this yet.
Thanks for the info,
Is there a way to see analysis that are still 'in the pipes'?
Yes, log in as admin and go to Settings > System > Analysis report
Where can I see whether the conversion from database to the UI has failed?
have a look at the content of the "Current Activity" and "Past Reports" tabs
Is there a way to speed the process?
This is a very broad question which implies tones of different answers. It all depends on where time is spent. You may be CPU bound, or memory bound or database bound, ...
Having a look at the queue of report processing might give you a hint.
1 MLoc is not so huge. I run SonarQube thru sonar-runner+Jenkins, and when Jenkins indicates in the log that the analysis has been successful, I am able to see it in SonarQube's dashboard. So I would say your 'latency' is not normal.
Could you please precise your environment? Physical/virtual? OS? DB? SQ release? etc.
After loads of searching around, I realized that for some reason sonarQube didn´t handle correctly the fact that I was running several sonar-runner analysis right after each other.
After the ´Store results in database´ message, there are a couple seconds for which starting a new analysis will cause SonarQube GUI to not see the analysis.
Running analysis with a bit more time between them reduced the latecny by a great deal.
Due to the fact that Seb gave a lot of insight about SonarQube itself, I will accept his answer. It is also probably more fit to a general public and less specific to my situation.
I'm currently attempting to debug a medium scale (in the 10's of thousands of lines ballpark) Java project which uses both JavaFX and Swing, and I'm hitting some odd exceptions every so often which I'm pretty sure are because of UI code not being called on the correct thread. The stack trace for these exceptions isn't really helpful at all, since they pretty much all originate from the UI drawing thread.
Now, sure I could sit down with a toothcomb and debug every UI call until I find one that's not being called on the correct thread, and keep doing that for the entire project, but that would be an incredibly long task. Is there some form of easier way to do this sort of debugging? For instance, somehow cause UI code to print out a debug message or throw an exception when it's not been called from its appropriate thread?
Running JavaFX and Swing on the same thread might help fix your threading issues.
There is an experimental feature in Java 8 to run JavaFX and Swing on the same thread:
https://javafx-jira.kenai.com/browse/RT-30694
http://bugs.sun.com/view_bug.do?bug_id=8015477
I think -Djavafx.embed.singleThread=true is the command line property setting to enable the experimental single threading system.
I am not sure if the experimental feature is available in the current Java 8 builds, but I think it might be now, so you may wish to try it.
If you need more information on the experimental single threading feature, you could ask the developers on the openjfx-dev mailing list.
Java 8 has better inbuilt reporting of when code is not run on the correct thread, it's not comprehensive, but it might assist you in locating the source of your error, even if you are not using the single threading option.
Some other users running large applications merging Swing and JavaFX reported similar hard to debug threading issues, so you could check those threads to see if your issues have the same cause.
You could turn on thread checks in glass by -Dglass.disableThreadChecks=false. This would switch on the thread checks in the lowest layer of JavaFX which is responsible for working with OS level APIs. In most cases those checks would be sufficient, because most of the calls are ending up in Glass. These checks would be enabled by default soon.
I have problem similar to my previous one presented here.
This time I want use program written in c/c++ to track execution of JAVA program. So as I stated before same code which track stdout printing for c/c++ and register syscall 4 haven't done it for JAVA. I assume it's because execlp which I trace is used just to run jvm. And later on there are created more processes (by inner mechanism of jvm) which I do not track. I found this topic which seems to be partial solution. If I got it right every child will be traced. But that's is a problem as well I want to track only that process which handles my application and not all others that jvm might create. Is there any chance to get know which jvm thread/process handles my program and track only it?
For make it a bit easier let's assume my JAVA program is one-thread.
If you start the binary through your tracer app, all threads will be traced.
But if you attach to a process, then you won't attach to all it's threads. You have to attach to all of its threads using the threadids, that you can found listed eg. in /proc/%d/task/.
Also, I suggest reading through strace's source code, I've learnt a lot from it. If you can use strace to successfully follow java threads as you want, you can get the logic from it.
I'm chasing a production bug that's intermittent enough to be a real bastich to diagnose properly but frequent enough to be a legitimate nuisance for our customers. While I'm waiting for it to happen again on a machine set to spam the logfile with trace output, I'm trying to come up with a theory on what it could be.
Is there any way for competing file read/writes to create what amounts to a deadlock condition? For instance, let's say I have Thread A that occasionally writes to config.xml, and Thread B that occasionally reads from it. Is there a set of circumstances that would cause Thread B to prevent Thread A from proceeding?
My thanks in advance to anybody who helps with this theoretical fishing expedition.
Edit: To answer Pyrolistical's questions: the code isn't using FileLock, and is running on a WinXP machine. Not asked, but probably worth noting: The production machines are running Java 1.5.
Temporarily setup your production process to startup with debugging support, add this to how you're starting your java program or to say the tomcat startup:
-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
Then attach to it:
jdb -connect com.sun.jdi.SocketAttach:hostname=localhost,port=8000
And take a look at your stack(s).
FileLock is an inter-process locking mechanism. It does nothing within the same JVM, so that isn't it. I would look at your synchronizations, and specifically at making sure you always acquire multiple locks in the same order.
I've gotten some useful tips for chasing the underlying bug, but based on the responses I've gotten, it would seem the correct answer to the actual question is:
No.
Damn. That was anti-climactic.
I know this is old, but to add some clarity on a "No" answer (for those of us who need to know why):
Deadlocking happens when exactly two distinct processes (transactions) update alternate dependent rows or records, but in reverse order. Basically both hang waiting for the other to complete an action which will never occur (as they are both waiting on the other). This is generally found in faulty database design.
If I recall, Wikipedia has a good definition here: http://en.wikipedia.org/wiki/Deadlock
Simple file access should not create dependencies like this. A more common issue would be your resource being used by another process and unavailable to the one trying to access it.
I am writing a simple checkers game in Java. When I mouse over the board my processor ramps up to 50% (100% on a core).
I would like to find out what part of my code(assuming its my fault) is executing during this.
I have tried debugging, but step-through debugging doesn't work very well in this case.
Is there any tool that can tell me where my problem lies? I am currently using Eclipse.
This is called "profiling". Your IDE probably comes with one: see Open Source Profilers in Java.
Use a profiler (e.g yourkit )
Profiling? I don't know what IDE you are using, but Eclipse has a decent proflier and there is also a list of some open-source profilers at java-source.
In a nutshell, profilers will tell you which part of your program is being called how many often.
I don't profile my programs much, so I don't have too much experience, but I have played around with the NetBeans IDE profiler when I was testing it out. (I usually use Eclipse as well. I will also look into the profiling features in Eclipse.)
The NetBeans profiler will tell you which thread was executing for how long, and which methods have been called how long, and will give you bar graphs to show how much time each method has taken. This should give you a hint as to which method is causing problems. You can take a look at the Java profiler that the NetBeans IDE provides, if you are curious.
Profiling is a technique which is usually used to measure which parts of a program is taking up a lot of execution time, which in turn can be used to evaluate whether or not performing optimizations would be beneficial to increase the performance of a program.
Good luck!
1) It is your fault :)
2) If you're using eclipse or netbeans, try using the profiling features -- it should pretty quickly tell you where your code is spending a lot of time.
3) failing that, add console output where you think the inner loop is -- you should be able to find it quickly.
Yes, there are such tools: you have to profile the code. You can either try TPTP in eclipse or perhaps try JProfiler. That will let you see what is being called and how often.
Use a profiler. There are many. Here is a list: http://java-source.net/open-source/profilers.
For example you can use JIP, a java coded profiler.
Clover will give a nice report showing hit counts for each line and branch. For example, this line was executed 7 times.
Plugins for Eclipse, Maven, Ant and IDEA are available. It is free for open source, or you can get a 30 day evaluation license.
If you're using Sun Java 6, then the most recent JDK releases come with JVisualVM in the bin directory. This is a capable monitoring and profiling tool that will require very little effort to use - you don't even need to start your program with special parameters - JVisualVM simply lists all the currently running java processes and you choose the one you want to play with.
This tool will tell you which methods are using all the processor time.
There are plenty of more powerful tools out there, but have a play with a free one first. Then, when you read about what other features are available out there, you'll have an inking about how they might help you.
This is a typically 'High CPU' problem.
There are two kind of high CPU problems
a) Where on thread is using 100% CPU of one core (This is your scenario)
b) CPU usage is 'abnormally high' when we execute certain actions. In such cases CPU may not be 100% but will be abnormally high. Typically this happens when we have CPU intensive operations in the code like XML parsing, serialization de-serialization etc.
Case (a) is easy to analyze. When you experience 100% CPU 5-6 thread dumps in 30 sec interval. Look for a thread which is active (in "runnable" state) and which is inside the same method (you can infer that by monitoring the thread stack). Most probably that you will see a 'busy wait' (see code below for an example)
while(true){
if(status) break;
// Thread.sleep(60000); // such a statement would have avoided busy wait
}
Case (b) also can be analyzed using thread dumps taken in equal interval. If you are lucky you will be able to find out the problem code, If you are not able to identify the problem code by using thread dump. You need to resort to profilers. In my experience YourKit profiler is very good.
I always try with thread dumps first. Profilers will only be last resort. In 80% of the cases we will be able to identify using thread dumps.
Or use JUnit test cases and a code coverage tool for some common components of yours. If there are components that call other components, you'll quickly see those executed many more times.
I use Clover with JUnit test cases, but for open-source, I hear EMMA is pretty good.
In single-threaded code, I find adding some statements like this:
System.out.println("A: "+ System.currentTimeMillis());
is simpler and as effective as using a profiler. You can soon narrow down the part of the code causing the problem.