It is really frustrating especially when I am working with sockets. Anyone know how to fix this? I constantly go into the task manager...
I think the most likely reason for this is a thread which does not terminate. This might be caused by the thread waiting for a time out, but a number of other reasons might prevent the thread from exiting as well.
I suggest you connect jvisualvm (part of the jdk, located in the bin folder) to your application and investigate what part of your application stays alive.
Edit: If your application runs in your systems default vm, you should see it in jvisualvm out of the box. But if you are using different vms, you have to start the application with appropriate parameters in order to connect jvisualvm to it.
This short guide explains the settings pretty well.
Related
We are trying to access an application from the tomcat which is on a different host, but it is not loading even though the tomcat is running. It was running fine for the past 3 months. We restarted the tomcat now it is working fine.
But, we could not able to zero in on what happened.
Any idea how to trace / what might have caused this?
The CPU usage was normal and the tomcat memory was 1205640.
the memory setting of tomcat are 1024- 2048(min-max)
We are using tomcat 7.
Help much appreciated....thanks in advance.....cheers!!
...also - not sure on Windows - you may be running out of file descriptors. This typically happens when streams are not properly closed in finally blocks.
In addition, check with netstat if you have a lot of sockets remaining open or accumulating in wait state.
Less likely, the application is creating threads and never releasing them.
The application is leaking something (memory, file descriptors, sockets, threads,...) and running over a limit.
There are different ways to track this. A profiler may help or more simply, running JVM dumps at regular intervals and checking what is accumulating. The excellent MAT will help you analyze the dumps.
Memory leak problems are not uncommon. If your Tomcat instance was running for three months and suddenly the contained application became unresponsive maybe that was the case. One solution (and if your resources allow you to do so) could be monitoring that Tomcat instance though JMX using jconsole to see how it behaves
I have been having occasional problems with a server I wrote. It's in Clojure, but I don't think that matters, and we can pretend it's in Java. Anyway, it works fine for hours at a time, but goes into fits where it behaves very badly: all activity stops, for around fifteen seconds, and then it works normally for a few seconds, then stops for fifteen seconds...and so on for (usually) about ten minutes or so, after which it goes back to behaving normally.
I've done a lot of profiling of it with YourKit, and I've ruled out a number of plausible suspects:
It's not a garbage collection issue: I'm running it with -XX:+UseConcMarkSweepGC, and I've verified that the server continues to run just fine during both minor and major collections, due to the concurrent nature of this garbage collector. And we're not thrashing as we run out of total memory or something: the current heap size is well below its max.
I don't think it's a locking/synchronization issue, but I'm not 100% sure on that. The YourKit profiler shows threads waiting sometimes, eg competing over the lock for System.out to produce log messages, but the only long waits are for worker threads in threadpools when there's nothing to do. And of course YourKit says it's never detected any deadlocks.
It's not something caused by having the profiler attached, because it still happens even if I boot the server up and then leave it alone without ever attaching the profiler.
It's not some other process on the system taking up all the CPU time: top shows CPU usage at 100% for my java process, and basically 0% for everything else.
My biggest problem is that I can't see what the server is doing during these strange funks, because the profiler stops receiving samples. Here's a graph of the CPU usage chart:
The left side of the graph is normal operation, during which we get profiler samples every second or so. The right side is "broken", and is very spiky because the profiler is only getting samples every ten seconds or so. In the samples it does get, the server seems to be doing its usual business: responding to requests and so on; and the logs confirm that it is doing normal stuff, but only at the times the profiler has samples for: during the upward-sloping "straight lines" on the graph, for which the profiler has no samples, the server is doing nothing at all.
So, does this graph look familiar to anyone? Have you had this problem before and fixed it? Or can you point me in the direction of a tool that can figure out what my server is doing during the time when YourKit can't? In case it matters, the server machine is running Ubuntu 10.04, and
$ java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.10) (rhel-1.28.1.10.10.el5_8-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
Okay, from the comments it seems clear to me we are not going to be able to figure this out with the information you've given so far. The best we can do is to give suggestions on how to debug it...
I would try to use jstack during one of the spikes and see if you can use that to figure out where it hangs.
If you have no chance to measure or debug in code try to look form the outside.
I would at first to try to reproduce the problem. In other words is there a external event that produce the behavior. Try to change the load on server. Switch every thing you can to reproduce the problem.
Maybe it's also a good idea to sniff the network traffic (tcpdump) to find something interesting around the time when you server hangs.
You can also run it on another operating system to check if it depends from your installation environment.
If you can't reproduce a situation where the problem occurs, try to find situations where you don't get the problem. For instance remove the server from net. Shutdown all other services.
If you can't find with that any change of behavior of your program try to reduce the complexity of your working code and see if you can find a internal module that seems to be related with the problem.
Have you had this problem before and fixed it? Or can you point me in
the direction of a tool that can figure out what my server is doing
during the time when YourKit can't?
If you have shell access on the server and can see stdout, try taking a thread dump when the server becomes unresponsive. Not sure if this will give you anything different than what jstack (mentioned in the other answer) would give you or not.
On Ubuntu: kill -QUIT <java-pid> (will not actually kill the Java process).
http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx
I am in the process of development of a stand-alone (Swing) Java EE client. The application server is JBoss 6.1.0. Some Windows workstations freeze while the client is running, some do not. Both 32-bit and 64-bit Windows workstations freeze, and again, some 32-bit and 64-bit Windows workstations work perfectly. The client runs well on Linux without any problems.
I have followed instructions from StackOverflow where people suggested to disable direct draw with -Dsun.java2d.noddraw=true. It did not help.
What puzzles me the most is that some workstations are almost identical - the same Windows version, the same graphics drivers, the same JRE, yet some work well, some do not.
The application uses the Preferences API a lot to store various positions of dockables (from the Docking Frames project), it uses JBoss client classes.
I am in the process of investigating if perhaps concurrent access to the Windows registry is causing this problem (if you had similar problems please let me know)...
First guess was that a race condition occurs somewhere in the GUI thread, and the GUI freezes. But that would freeze only the GUI, it should not freeze the whole Windows.
The machine, once frozen, responds to pings, however no Windows service works.
I would appreciate any hint that can help me solve the problem.
Edit:
CPU usage is always around 10%.
Number of threads never goes above 30, however not all threads are daemon threads (this includes AWT threads, RMI threads, etc).
Try out the EventQueue with Watchdog. You install an alternative EventQueue in the Application. The Watchdog tells you which Events are locking up your GUI.
It turned out to be ESET. On every Windows machine that had ESET installed I had this problem. Apparently ESET has this "protocol tracking" feature that is ON by default. Once turned off we never had this problem.
Trying to diagnose some bizarre Tomcat (7.0.21) and/or JVM errors on a 64-bit linux (CentOS) machine.
I'm load testing our server application and tried hitting it with 100K messages. Launched jvisualvm and kept my eye on the heap the whole time. Everything was looking great* (see below) until I got to about 93K processed messages and then Tomcat just died. Ran a ps on Tomcat's PID number to confirm it was dead.
Up until this crash:
Load test had been running for about 90 minutes; should have finished shortly thereafter since we were at 93K/100K)
CPU was holding strong around 45%
Used heap was around 2GB (plus or minus a bunch after GCs) but heap size grew from 4GB to MAX_HEAP after about 30 minutes
Class loading/unloading was cycling normally
Thread dumps were normal
Nowhere in the server code are any calls to System.exit() - so we can rule that right out (and yes I've double-checked!!!).
I'm not sure if this is Tomcat crashing or the JVM (how do I tell?). And even if I did know, I can't seem to find any indication of what went wrong:
All of the server app's logs just stop without any ERROR messages (even though we have logging universally set to DEBUG and higher)
Tomcat's catalina.out and respect localhost_access_* files just stop without any info
I've heard it is possible to have Tomcat log a coredump when it does but not sure how to do that and online examples aren't helping much.
How would SO go about diagnosing this? What steps should I take to start ruling out all of the possible factors?
Thanks in advance!
If the JVM crashes, you should have a hs_err_pidNNN.log file; you don't have to do anything to enable this. Its location depends on your OS and how you are running Tomcat. On Windows, they can show up on your desktop, unless you are running as a service. Otherwise, they should be in the current working directory of the crashed process.
Your operating system probably provides additional tools for process monitoring; you could describe your environment more, or perhaps ask at serverfault.com.
It's also possible that jvisualvm is actually causing the crash.
I'd try reproducing the problem, and progressively simplify the scenario to help isolate the cause.
Another possibility is that the OS is running out of memory and the OOM Killer is killing your process. In this case, the JVM wouldn't get an opportunity to write a heap dump, or an hs_err_pid file.
You can use the option java -XX:+HeapDumpOnOutOfMemoryError to create a heap dump for jvm crash due to out of memory error.
More details here Using HeapDumpOnOutOfMemoryError parameter for heap dump for JBoss.
Sorry I had to remove the green check from #erickson. I finally figured out what was killing Tomcat.
It looks like a profiler plugin is not configured correctly with VisualVM and attempting to run a profile on the Tomcat process killed it.
Investigating why right now, and will update this answer once I know more.
I have a java application that I run from eclipse 3.5.
My OS is WinXP(SP2) and the JRE version is 6.05.
I run the application on two identical computers (or so I think) but the application behaves differently on each computer.
The computers are the same Dell Optiplex model with the same amount of memory and have the same GPU.
On the first computer, the application runs flawlessly. However, on the second one the application freezes for a couple of minutes and then returns to run normally.
The strange thing is that the CPU usage on the second computer is not high at all. It seems as though my application does not receive any CPU for no apparent reason.
Computers should be deterministic so I assume there must be some difference between the machines but I don't know where to look.
I would love some ideas on where the problem might be.
Thanks,
Yoav.
I've found the problem.
The application that was unresponsive was run in debug mode.
Sorry to have wasted your time...
It may help you to get a Thread Dump when the app freezes. This will hopefully tell you exactly what is holding you up (i.e. waiting for IO somewhere).
Well, I would first update your JRE version as there are newer versions now.
As for both computers being identical, are they really identical? I find it difficult to believe that both have the same exact software and setup and that anything you have done to one, you have always done to the other. If this is indeed the case, you may want to try to debug your application on the second machine (the one that hangs) and find out specifically where it hangs.
It may also help us if you give more information about your application. The problem may not be your computer at all if the application is doing things like web access, network access, etc.
So both computers have nearly identical hardware. A few other things to check
Do they both have Eclipse 3.5, WinXP(SP2) and JRE 6.05 installed?
And behave differently when run from within Eclipse (on both machines or on one run from command-line)?
Is this reproducible? If yes When does it happen? On startup? Or on some specific action?
Does the program have a GUI?
Is there maybe some kind of virus scanner or another comparable software installed on one of the machines which could delay the program
Is networking, file acccess, multithreading involved?
I can think of two non-application possibilities:
Memory Paging. There's something extra happening on the slow machine, so your JVM is not getting a fair share of CPU time. A large daemon process or some such.
Network access. Your app is making some kind of network call and it's glitching or timeing out. Perhaps going and fetching some XML schema, perhaps a disk acesss to a mounted drive.
I've seen all manner of weirdness when apps attempt to access hosts by name and DNS is not well. One machine has an etc/host entry the other does not. Even each machine might want to resolve itself.