I have a situation where Tomcat is starting with my java application, but then hangs for several minutes before continuing. Startup logs do not seem to point to anything specific.
Is there a way that I could debug this?
I encountered once a problem where starting Tomcat on a dedicated server also took ages. My problem was that the entropy source that is used to initialize SecureRandom was short of entropy.
It's a known bug: https://wiki.apache.org/tomcat/HowTo/FasterStartUp#Entropy_Source
If this is your problem, known workarounds are:
Set the following system property: -Djava.security.egd=file:/dev/./urandom
or
Use 'new SecureRandom()' instead of 'SecureRandom.getInstance("SHA1PRNG")' (if you have this in your code [source]).
Your question is very broad. We have nothing to work with here in providing an answer.
All I can give you is generic debugging advice.
Add printouts into your web application's startup sequence. Watch
them and see where it hangs.
Try commenting out part for part of the start-up until you notice that
start-up is suddenly sped up, the last bit you commented out should
be the faulty one.
Related
When a Java VM crashes with an EXCEPTION_ACCESS_VIOLATION and produces an hs_err_pidXXX.log file, what does that indicate? The error itself is basically a null pointer exception. Is it always caused by a bug in the JVM, or are there other causes like malfunctioning hardware or software conflicts?
Edit: there is a native component, this is an SWT application on win32.
Most of the times this is a bug in the VM.
But it can be caused by any native code (e.g. JNI calls).
The hs_err_pidXXX.log file should contain some information about where the problem happened.
You can also check the "Heap" section inside the file. Many of the VM bugs are caused by the garbage collection (expecially in older VMs). This section should show you if the garbage was running at the time of the crash. Also this section shows, if some sections of the heap are filled (the percentage numbers).
The VM is also much more likely to crash in a low memory situation than otherwise.
Answer found!
I had the same error and noticed that others who provided the contents of the pid log file were running 64 bit Windows. Just like me. At the end log file, it included the PATH statement. There I could see C:\Windows\SysWOW64 was incorrectly listed ahead of: %SystemRoot%\system32. Once I corrected it, the exception disappeared.
First thing you should do is upgrade your JVM to the latest you can.
Can you repeat the issue? Or does it seem to happen randomly? We recently had a problem where our JVM was crashing all over the place, at random times. Turns out it was a hardware problem. We put the drives in a new server and it completely went away.
Bottom line, the JVM should never crash, as the poster above mentioned if your not doing any JNI then my gut is that you have a hardware problem.
The cause of the problem will be documented in the hs_err* file, if you know what to look for. Take a look, and if it still isn't clear, consider posting the first 5 or 10 lines of the stack trace and other pertinent info (don't post the whole thing, there's tons of info in there that won't help - but you have to figure out which 1% is important :-) )
Are you using a Browser widget and executing javascript in the Browser widget? If so, then there are bugs in some versions of SWT that causes the JVM to crash in native code, in various Windows libraries.
Two examples (that I opened) are bug 217306 and bug 127960. These two bug reports are not the only bug reports of the JVM crashing in SWT, however.
If you aren't using the Browser widget then these suggestions won't help you. In that case, you can search for a list of SWT bugs causing a JVM crash. If none of those are your issue, then I highly recommend that you open a bug report with SWT.
I have the same problem with a JNLP application that I have been using for a long time and is pretty reliable. The problem started immediately after I upgraded from Windows 7 to Windows 10. According to my investigation, it is most likely a bug in Win 10.
The following is not a solution, but an ugly workaround. In jre/bin directory, there is javaws.exe. If I right-clicked /Properties/Compatibility and ticked Run this program as an administrator, the JNLP app started to work.
Please, be aware that this approach could cause security issues and use it only if you have no other option and 100% know what you are doing.
I'm having trouble with a Jetty 9 server application that seems to go into some kind of resting state after a longer period of idleness. Normally the memory usage of the Java process is ~500 MB, but after being idle for some time it seems to drop down to less than 50MB. The first request that comes takes up to several seconds to respond whereas requests are normally on the scale of tens of milliseconds. But after one or two requests it seems like the application is back to it's normal responsive state.
I'm running on the 32-bit Oracle Java 8 JVM. My JVM configuration is very basic:
java -server -jar start.jar
I was hoping that this issue might be solvable through JVM configuration. Does anyone know if there's any particular parameter to disable this type of behavior?
edit: Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. See my own answer below for a description of my solution.
Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. This was clearly visible when comparing the private working set to the commit size in the task manager.
My solution to this was two-fold. First, I made a simple scheduled job inside my server app that runs every minute and does a simple test run to make sure that the important services never go inactive for long periods. I'm hoping this should ensure that Windows doesn't regard the related pages as inactive.
Afterwards, I also noticed that the process was executing with "Below normal" priority. So I changed the script that starts the server to ensure that it's running with "High" priority going forward. This seems likely to affect swapping behavior and may very well also have been enough to resolve the issue on it's own, but I only found it after already deploying my first solution so that remains unclear. In any case, everything seems to be working as it should now.
When a Java VM crashes with an EXCEPTION_ACCESS_VIOLATION and produces an hs_err_pidXXX.log file, what does that indicate? The error itself is basically a null pointer exception. Is it always caused by a bug in the JVM, or are there other causes like malfunctioning hardware or software conflicts?
Edit: there is a native component, this is an SWT application on win32.
Most of the times this is a bug in the VM.
But it can be caused by any native code (e.g. JNI calls).
The hs_err_pidXXX.log file should contain some information about where the problem happened.
You can also check the "Heap" section inside the file. Many of the VM bugs are caused by the garbage collection (expecially in older VMs). This section should show you if the garbage was running at the time of the crash. Also this section shows, if some sections of the heap are filled (the percentage numbers).
The VM is also much more likely to crash in a low memory situation than otherwise.
Answer found!
I had the same error and noticed that others who provided the contents of the pid log file were running 64 bit Windows. Just like me. At the end log file, it included the PATH statement. There I could see C:\Windows\SysWOW64 was incorrectly listed ahead of: %SystemRoot%\system32. Once I corrected it, the exception disappeared.
First thing you should do is upgrade your JVM to the latest you can.
Can you repeat the issue? Or does it seem to happen randomly? We recently had a problem where our JVM was crashing all over the place, at random times. Turns out it was a hardware problem. We put the drives in a new server and it completely went away.
Bottom line, the JVM should never crash, as the poster above mentioned if your not doing any JNI then my gut is that you have a hardware problem.
The cause of the problem will be documented in the hs_err* file, if you know what to look for. Take a look, and if it still isn't clear, consider posting the first 5 or 10 lines of the stack trace and other pertinent info (don't post the whole thing, there's tons of info in there that won't help - but you have to figure out which 1% is important :-) )
Are you using a Browser widget and executing javascript in the Browser widget? If so, then there are bugs in some versions of SWT that causes the JVM to crash in native code, in various Windows libraries.
Two examples (that I opened) are bug 217306 and bug 127960. These two bug reports are not the only bug reports of the JVM crashing in SWT, however.
If you aren't using the Browser widget then these suggestions won't help you. In that case, you can search for a list of SWT bugs causing a JVM crash. If none of those are your issue, then I highly recommend that you open a bug report with SWT.
I have the same problem with a JNLP application that I have been using for a long time and is pretty reliable. The problem started immediately after I upgraded from Windows 7 to Windows 10. According to my investigation, it is most likely a bug in Win 10.
The following is not a solution, but an ugly workaround. In jre/bin directory, there is javaws.exe. If I right-clicked /Properties/Compatibility and ticked Run this program as an administrator, the JNLP app started to work.
Please, be aware that this approach could cause security issues and use it only if you have no other option and 100% know what you are doing.
As soon I click to start (literally immediately) the tomcat server with my web app, it displays the following error:
http://i67.photobucket.com/albums/h282/jamesekki/error.jpg
What's weird is the application still builds successfully and works fine. The problem is it slows down development time since I get this error every time I need to deploy any changes.
Here is my eclipse ini:
http://i67.photobucket.com/albums/h282/jamesekki/Capture.jpg
Any ideas?
Things I have tried:
1. Increased xms/xmx in increments of 128 megs all the way up to 4 gigs (still fail)
2. Increased permsize/maxpermsize similar to increments above (still fail)
Maybe these will help explain:
http://www.mulesoft.com/tomcat-oome-out-of-memory-error
http://baskarfelix.wordpress.com/2008/08/06/out-of-memory-errorjava-heap-space/
I'd also be curious to know if you have the same issue if you leave Eclipse out of the picture. Create a WAR file, add it to the Tomcat /webapps directory, and start Tomcat using a command shell and the start script in the /bin directory. If you still have a problem, I'd say you should look at how Tomcat and your JVM are configured. If you don't, it tells me that Eclipse is the problem.
Thanks everyone for your inputs. I was able to narrow it down to what was the root cause of the whole issue. I had an external api jar file in the src/main/resources/lib of my services layer and Eclipse seems to not like that for whatever reason. As soon as I moved that out of that location, it resolved the issue. Can anyone explain why this would potentially cause an issue? Just for learning purposes. Thanks!
Your settings look good to me. You could try the long way of profiling [1] tomcat to see what happens on startup, or the crude way of deleting your local server and runtime, to see if it helps with a fresh one. If that fails, re-download latest Eclipse Java EE edition, and set up from scratch.
While it's not the best solution (in case the problem resurfaces, you still have no proper solution), it is sometimes worth it if it takes too long otherwise.
[1] You could profile with JProfiler or YourKit Profiler. Both have trial versions. JVisualVM is also an alternative that comes with the JDK. However it may be an issue if Tomcat crashes immediately.
I wanted to get ideas from the SO community about this issue.
Here is the problem:
We have a user on the other side of the world launching our app through WebStart. The user, however, is complaining that her whole application freezes up and becomes unresponsive. Usually, the client is doing a lot of database queries to a distributed database.
Questions:
If we ask her to do a CTRL-Break on her application, where would the JVM write the stack trace to?
Would it be enough just to use JConsole?
Would implementing JMX beans on the client be overkill? Would it actually help in troubleshooting issues in production?
Right now the users are running on JRE 1.5.0-b08, but we do plan on migrating to JRE 6 in a couple of months.
What do you think?
José, you can get a lot of information from the JVM in a number of ways.
The best might be to enable debugging in the remote JVM. You can set them using the j2se element in the descriptor XML, as shown here. Since you can set -Xdebug you have a good start; I've never tried to do remote debugging on a web start app, so it may be a little bit of an issue setting up the remote part.
You could also set some things up yourself by adding a separate thread to talk to you remotely and send debugging messages.
You could use a native java or log4j remote logger.
If it's hanging the way you describe, though, the odds are very high that what's happening is a network hangup of some sort. Can you put some tracing/debugging onto your end of the conversation?
Instead of these debugging suggestions, why don't you install an exception handler for your threads? See java.lang.Thread.
void setDefaultUncaughtExceptionHandler(Thread.UncaughtExceptionHandler eh)
Here's the relevant javadoc:
http://java.sun.com/javase/6/docs/api/java/lang/Thread.html#setDefaultUncaughtExceptionHandler(java.lang.Thread.UncaughtExceptionHandler)
If you install that in your code, and once inside Swing's EDT, then just write some java code to e-mail it to yourself, save it on a server, show it to the user, etc.
You need to have the Java Console displayed (run javaws from the command line, and select this from the Preferences dialog), then hit "v"