Today in the company where I work we received a report about one of our webapps not working.
The first thing we did was look at the hardware utilization:
Processor: 5%;
Memory: 68%;
Disk IO capacity: 4%;
Network: 1Mbps/1Gbps;
After some tests we saw that, indeed, the webapp pages weren't loading, some time after the webapp timeouts.
Other webapps in the exactly same tomcat instance were working fine and fast as ever, no problem with them. We tried to restart/reload the webapp but it still didn't get it working. Finally we restarted tomcat which corrected the issue for now.
There were no restarts or redeploys since yesterday when it was working fine. We believe this may be a periodic bug of some sort so we want to correct it soon.
Does anyone know of any steps we may take to investigate what it might have been?
It seems related to lock on IO.
If you are using linux, the first thing is do is to check open files by using the command lsof . If you see a lot of opened files by your app, you have to check in the code that every InputStream/OutputStream are closed (even in exception handling code).
An other common source of issues is related to thread starvation, see http://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Stuck_Thread_Detection_Valve
Related
I'm having trouble with a Jetty 9 server application that seems to go into some kind of resting state after a longer period of idleness. Normally the memory usage of the Java process is ~500 MB, but after being idle for some time it seems to drop down to less than 50MB. The first request that comes takes up to several seconds to respond whereas requests are normally on the scale of tens of milliseconds. But after one or two requests it seems like the application is back to it's normal responsive state.
I'm running on the 32-bit Oracle Java 8 JVM. My JVM configuration is very basic:
java -server -jar start.jar
I was hoping that this issue might be solvable through JVM configuration. Does anyone know if there's any particular parameter to disable this type of behavior?
edit: Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. See my own answer below for a description of my solution.
Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. This was clearly visible when comparing the private working set to the commit size in the task manager.
My solution to this was two-fold. First, I made a simple scheduled job inside my server app that runs every minute and does a simple test run to make sure that the important services never go inactive for long periods. I'm hoping this should ensure that Windows doesn't regard the related pages as inactive.
Afterwards, I also noticed that the process was executing with "Below normal" priority. So I changed the script that starts the server to ensure that it's running with "High" priority going forward. This seems likely to affect swapping behavior and may very well also have been enough to resolve the issue on it's own, but I only found it after already deploying my first solution so that remains unclear. In any case, everything seems to be working as it should now.
I am running Apache Tomcat 7.0.42 on Ubuntu 12.04. When I start up Apache initially it's all fine and I can connect. However after using it for a while I can no longer browse to any site hosted on Apache, not even the root site. The browser just waits for a connection forever. If I go on the machine I can see lots of connections that have been closed and are CLOSE_WAIT on netstat, but nothing open. The Apache log file doesn't tell me anything either.
If I restart Apache the problem doesn't go away, even rebooting the server hasn't solved the problem. After struggling for a while it starts working again only to stop shortly afterwards.
I don't have any network issues between the machines.
I had a look at the known bugs on 7, but don't notice anything that sounds related. I have also reinstalled Apache on the machine, but the problem returned soon after.
I feel like I'm missing something silly, but I'm just not getting it.
I am not sure if this will help, but you can give a try for below options.
The problem may not be for Tomcat7, but for your own site. I have used tomcat for so many days and didnt notice such kind of issue single time.
May be your one of the site is having memory leaks and after running for sometime, it makes server stop because of low memory.
If there are many sites then you can try removing all and adding one by one to check which site is giving the issue.
You can try with simple page first by removing all other sites to check if the issue is with tomcat or your site.
We are trying to access an application from the tomcat which is on a different host, but it is not loading even though the tomcat is running. It was running fine for the past 3 months. We restarted the tomcat now it is working fine.
But, we could not able to zero in on what happened.
Any idea how to trace / what might have caused this?
The CPU usage was normal and the tomcat memory was 1205640.
the memory setting of tomcat are 1024- 2048(min-max)
We are using tomcat 7.
Help much appreciated....thanks in advance.....cheers!!
...also - not sure on Windows - you may be running out of file descriptors. This typically happens when streams are not properly closed in finally blocks.
In addition, check with netstat if you have a lot of sockets remaining open or accumulating in wait state.
Less likely, the application is creating threads and never releasing them.
The application is leaking something (memory, file descriptors, sockets, threads,...) and running over a limit.
There are different ways to track this. A profiler may help or more simply, running JVM dumps at regular intervals and checking what is accumulating. The excellent MAT will help you analyze the dumps.
Memory leak problems are not uncommon. If your Tomcat instance was running for three months and suddenly the contained application became unresponsive maybe that was the case. One solution (and if your resources allow you to do so) could be monitoring that Tomcat instance though JMX using jconsole to see how it behaves
As soon I click to start (literally immediately) the tomcat server with my web app, it displays the following error:
http://i67.photobucket.com/albums/h282/jamesekki/error.jpg
What's weird is the application still builds successfully and works fine. The problem is it slows down development time since I get this error every time I need to deploy any changes.
Here is my eclipse ini:
http://i67.photobucket.com/albums/h282/jamesekki/Capture.jpg
Any ideas?
Things I have tried:
1. Increased xms/xmx in increments of 128 megs all the way up to 4 gigs (still fail)
2. Increased permsize/maxpermsize similar to increments above (still fail)
Maybe these will help explain:
http://www.mulesoft.com/tomcat-oome-out-of-memory-error
http://baskarfelix.wordpress.com/2008/08/06/out-of-memory-errorjava-heap-space/
I'd also be curious to know if you have the same issue if you leave Eclipse out of the picture. Create a WAR file, add it to the Tomcat /webapps directory, and start Tomcat using a command shell and the start script in the /bin directory. If you still have a problem, I'd say you should look at how Tomcat and your JVM are configured. If you don't, it tells me that Eclipse is the problem.
Thanks everyone for your inputs. I was able to narrow it down to what was the root cause of the whole issue. I had an external api jar file in the src/main/resources/lib of my services layer and Eclipse seems to not like that for whatever reason. As soon as I moved that out of that location, it resolved the issue. Can anyone explain why this would potentially cause an issue? Just for learning purposes. Thanks!
Your settings look good to me. You could try the long way of profiling [1] tomcat to see what happens on startup, or the crude way of deleting your local server and runtime, to see if it helps with a fresh one. If that fails, re-download latest Eclipse Java EE edition, and set up from scratch.
While it's not the best solution (in case the problem resurfaces, you still have no proper solution), it is sometimes worth it if it takes too long otherwise.
[1] You could profile with JProfiler or YourKit Profiler. Both have trial versions. JVisualVM is also an alternative that comes with the JDK. However it may be an issue if Tomcat crashes immediately.
[UPDATE: I forgot to add that this 30 sec. freezing problem only happens the first time I try to load a file from the server. Subsequent loads are very quick. Maybe some strange reverse DNS lookup? I am hosting on Google's appengine.]
I started a little project recently called http://www.chartle.net which is build around an applet.
Startup time is an important factor in the user's experience of an applet. I collect statistics and am shocked that I find often very long startup times (factor 50 to 100 higher then necessary)
The applet starts in 1-3 seconds depending on the speed of your computer and connection. Still for some users it takes up to 100 sec.
I have mixed results from my own tests. Mostly it is very fast but sometimes freezes the browser for a long time and the Java console doesn't tell me why. Best guess is, that it stalls when loading a saved chart.
Please help me figuring this out - best test by opening an already saved chart (click on one of the 'create' links at http://www.chartle.net/gallery)
Cheers,
Dieter
This is generic help rather than specific for your demo (which loaded fine for me in a few attempts).
Freezing applets
In the JDK bin directory there is a very handy programme called jstack. Refresh your browser window until it crashes and then run:
jstack *process_id*
This will give you the stack trace of any frozen Java process. If Java is not a separate process then you can use the browser's process (eg for Opera).
The following few problems were/are common for me:
I reccommend you use invokeLater rather than invokeAndWait on the init method (although you can't do this if you use start/stop methods)
Opera's custom java plugin acts very poorly...
Deadlocks caused by synch blocks and invokeAndWait's
Slow applets
Possibly the browser is fetching resources from the server, unable to use the jar file?
It may be that only the old plugin causes these problems. That means basically all people running on OSX and other users with Java prior to 1.6_update_10.
So, I would really appreciate people with such setups to watch their Java console and describe the first startup behaviour.
Cheers,
Dieter