I have ran the performance test of my java application for two days and monitored through JConsole. However After the test ran for a day; JConsole output window reports OutOfMemoryError and JConsole hangup.
Q1) Is it becasue of my application OR Jconsole problem?
Q2) How to resolve this ? Is there any way to increase the HEAP memory of JConsole?
Q3) If it is due to long run then how can i set time range to disconnect the remote connection after sometime (not closing/killing)?
Please help.
That stack trace makes it pretty obvious that JConsole is running out of memory when setting text to the window.
Unfortunately there's no command line options to configure that. There may be an MBean in JConsole that allows you to configure it, but I'd guess that there isn't.
Your alternatives are
to close the console output, and just open it when you need it, or
to redirect your application's stdout and stderr to files, so that the console output never has anything in it.
Related
I have a problem and my job depend on of that.
There are some java apps with tomcat under Linux that crash ramdonly (the apps are not mine and it can not be modified).
Every day we find in the morning some app broken.
I want to see the java stack just when it crashed the app for seeing the message of the JVM (outofmemory, nullpointer etc). For if i can see an advice for fixing the problem.
I do not know nothing about to do this.
I saw searching in internet visualvm and jconsole for this. Is enough for what i want to do?.
I want to see the messages of the java stack of JVM just when crashes.
I need help. Thank you very much.
looks like you have memory leak issue, does the app works after restart for a particular period of time?
You might want to see what happening inside java heap , for that you can take heap dump.. Use jcmd utility for this , you can find this utility within JDK installed on your server.
jcmd <process id/main class> GC.heap_dump filename=filename
NOTE: This will do a GC every time this runs.
To schedule this you need to set the cronjob.
Alternatively if you specify -XX:+HeapDumpOnOutOfMemoryError command-line option while running your application, then when an OutOfMemoryError exception is thrown, the JVM will generate a heap dump(in the logs).
Hope this helps. :)
I have tomcat server running. and it crashes suddenly for some reason... however I am trying to find the error.
Is there a function in tomcat or java like beforeExit() or ifCrashed() which I can override and write some code there like notifying myself if server crashes for some reason.
You could try using shut down hooks which are executed on system exit. It's not guaranteed though that it will be executed on a hard crash like SIGKILL but this may be an option in some cases:
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
// Implementation goes here..;
}
});
I don't think that crashed server can manage something. You can consider this steps:
use servletContextListeners in order to run code before servlet context will shutdown. It will work if shutdown is done gracefully
if it is JVM that is crushing you can try to find tools that provide JVM monitoring services and reporting. Also I recommend to provide flag in order to get reports from OutOfMemoryError and open JMX ports for monitoring
you can right some mock rest endpoint and run some external job that will call that endpoint periodically and if it returns HTTPStatus.OK then it working, if not - then you can report yourself someway.
It could be better if you can provide logs and some information about what is really happening with server. Maybe here people will help but I gues this is topic of another questions.
In case of a fatal crash, you will get a jvm log. You can specify the location by giving tomcat a -XX:ErrorFile on the Oracle JVM
http://www.oracle.com/technetwork/java/javase/felog-138657.html
If it is not that fatal, you can adjust the log settings in tomcat to display better logging levels, such as specified on their doc site
https://tomcat.apache.org/tomcat-6.0-doc/logging.html
If it dies because of out of memory errors, you can get a heap dump on OOM, and analyse it with i.e. jhat
http://docs.oracle.com/javase/6/docs/technotes/tools/share/jhat.html
If profiling is an option for you JProfiler has an offline profiling mode where you can capture information about the running program and log it for viewing at a later time. Check out the docs
JPRofiler Offline Profiling
This might give you some more insight into whats happening in the application. According to this documentation you can also manage the profiling agent from you application.
I am facing with a strange case. I'd be glad if you could share your comments.
We have solution running on Java 1.6.085 and sometimes Java process is getting hang in production. The solution is running on Linux server.
I investigated GC logs, there is no Full GC. Pause times also look reasonable.
Then we tried to take a thread dump when case happens however kill -3, ./jstack or ./jstack -F do not work. No thread dump could be taken. What could be the reason for that ? Any ideas on investigating the issue ?
BR
-emre
After a while it is understood that the issue occured due to pstack and qdb commands which are executed on java process for operational purposes. Somehow pstack and qdb suspends the java process. Therefore we couldnt be able to take thread or heap dump
We're using jConsole with the topthreads plugin to analyze such cases. The plugin uses JMX to check the thread runtimes and displays their CPU usage since start of the tracking procedure as well as the current stack trace for each thread.
To connect our servers from a local machine we use tunnels in putty, i.e. we first connect to the server via putty and then connect jConsole to a local port which is tunneled to the server.
Trying to diagnose some bizarre Tomcat (7.0.21) and/or JVM errors on a 64-bit linux (CentOS) machine.
I'm load testing our server application and tried hitting it with 100K messages. Launched jvisualvm and kept my eye on the heap the whole time. Everything was looking great* (see below) until I got to about 93K processed messages and then Tomcat just died. Ran a ps on Tomcat's PID number to confirm it was dead.
Up until this crash:
Load test had been running for about 90 minutes; should have finished shortly thereafter since we were at 93K/100K)
CPU was holding strong around 45%
Used heap was around 2GB (plus or minus a bunch after GCs) but heap size grew from 4GB to MAX_HEAP after about 30 minutes
Class loading/unloading was cycling normally
Thread dumps were normal
Nowhere in the server code are any calls to System.exit() - so we can rule that right out (and yes I've double-checked!!!).
I'm not sure if this is Tomcat crashing or the JVM (how do I tell?). And even if I did know, I can't seem to find any indication of what went wrong:
All of the server app's logs just stop without any ERROR messages (even though we have logging universally set to DEBUG and higher)
Tomcat's catalina.out and respect localhost_access_* files just stop without any info
I've heard it is possible to have Tomcat log a coredump when it does but not sure how to do that and online examples aren't helping much.
How would SO go about diagnosing this? What steps should I take to start ruling out all of the possible factors?
Thanks in advance!
If the JVM crashes, you should have a hs_err_pidNNN.log file; you don't have to do anything to enable this. Its location depends on your OS and how you are running Tomcat. On Windows, they can show up on your desktop, unless you are running as a service. Otherwise, they should be in the current working directory of the crashed process.
Your operating system probably provides additional tools for process monitoring; you could describe your environment more, or perhaps ask at serverfault.com.
It's also possible that jvisualvm is actually causing the crash.
I'd try reproducing the problem, and progressively simplify the scenario to help isolate the cause.
Another possibility is that the OS is running out of memory and the OOM Killer is killing your process. In this case, the JVM wouldn't get an opportunity to write a heap dump, or an hs_err_pid file.
You can use the option java -XX:+HeapDumpOnOutOfMemoryError to create a heap dump for jvm crash due to out of memory error.
More details here Using HeapDumpOnOutOfMemoryError parameter for heap dump for JBoss.
Sorry I had to remove the green check from #erickson. I finally figured out what was killing Tomcat.
It looks like a profiler plugin is not configured correctly with VisualVM and attempting to run a profile on the Tomcat process killed it.
Investigating why right now, and will update this answer once I know more.
I'm trying to run jstack command on my java application. Application is rather big, running inside jboss AS occupying about 4gb of memory. OS is Windows Server 2003 Standard edition. Every time i get an error "Not enough storage is available to process this command". There is enough ram, 16gb, and disk space. So, any ideas?
I ran into this recently on Win2008r2 and thought I'd share my solution since it took a while to figure out. Rob's comment about psexec -s is what did it for me.
It appears that on Vista and later jstack doesn't work against services because of the user context. It has nothing to do with memory. I suspect this is the same reason people have seen this problem on 2003 via remote desktop, unless you use the /admin or /console switch on mstsc. As of Vista the tightened security is probably what broke it.
Starting my app from a cmd window worked fine, but that doesn't help me debug our standard install. Enabling the java debug port (for VisualVM, Eclipse or most any Java debugger) requires an app restart, so you lose the state you're probably trying to capture if you don't already have debugging enabled. Starting the service under my user credentials did not work - I was a little surprised at that. But psexec -s runs jstack from the system context, which worked like a charm. Oh, and you'll need to run psexec from an elevated cmd prompt, if UAC is on.
In the past I have seen this when the JVM is running as a Windows Service on Windows 2003.
First, check to see if this is an issue with the TMP directory.
Second, jstack (or the other utilities like jconsole) will not connect to the local process unless it is running in the same session. If the service is running as a specific user, you may be able to connect by logging into the same session. If you are using Remote Desktop, you can connect using "mstsc /admin" (used to be /console) and try to run jstack again. Definitely check to make sure the TMP directory is set properly if this doesn't fix the problem.
If the service is running as LocalSystem, the above procedure probably will not help much. I don't know if there is a way to log into the same session as LocalSystem.
Some other alternatives may be to set the process up for remote monitoring and use jvisualvm (from the server itself or another machine) to connect over a port and do a thread dump.
We had problems running JStack on a Windows machine with even a modest application (1GB). We ended up doing our stack and heap analysis using Netbeans. This seemed to cope with the parsing of dump files a lot better. YMMV.
Give Netbeans a try for profiling - its very good. Note that VisualVM is a cutdown NB profiler and comes with 6u7.
psexec -s jstack PID >> c:\jstack.log perfectly works on the same machine. For the first time it took some time but again I executed with the redirect to file option, it completed with in few seconds.
This is an error message from the underlying O/S. There's not much you can do in your code to deal with this other than catch the exception which is thrown. Boo to Windows for being so limited.
http://technet.microsoft.com/en-us/library/cc978735.aspx