In our confluence server (tomcat 9 based), we added interruptThreshold to 30mins to kill long running threads (usually these are stuck threads).
However, interrupts for certain threads are not succesful (and it is okay, as it's mentioned in documentation that it doesn't work in all cases), but these specific threads gets killed after 8 hours.
These specific threads are related to a custom plugin and our developers say they don't have such 8 hours setting anywhere and I couldn't find anything in tomcat or in confluence.
We will be able to replicate the stuck threads, but how to findout what's killing them after 8 hours?
Such actions on Tomcat should be logged in the {CONFLUENCE_HOME}/{DATA_FOLDER}/logs folder, please find these logs files and look for this operation/timestamp.
If for some reason, logs are not fully reflected, you can set it as described in
https://confluence.atlassian.com/doc/configuring-logging-181535215.html.
Also, you can control loggng in -X... Java / Tomcat statements when running start.bin/start.sh.
Related
Today in the company where I work we received a report about one of our webapps not working.
The first thing we did was look at the hardware utilization:
Processor: 5%;
Memory: 68%;
Disk IO capacity: 4%;
Network: 1Mbps/1Gbps;
After some tests we saw that, indeed, the webapp pages weren't loading, some time after the webapp timeouts.
Other webapps in the exactly same tomcat instance were working fine and fast as ever, no problem with them. We tried to restart/reload the webapp but it still didn't get it working. Finally we restarted tomcat which corrected the issue for now.
There were no restarts or redeploys since yesterday when it was working fine. We believe this may be a periodic bug of some sort so we want to correct it soon.
Does anyone know of any steps we may take to investigate what it might have been?
It seems related to lock on IO.
If you are using linux, the first thing is do is to check open files by using the command lsof . If you see a lot of opened files by your app, you have to check in the code that every InputStream/OutputStream are closed (even in exception handling code).
An other common source of issues is related to thread starvation, see http://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Stuck_Thread_Detection_Valve
I'm having trouble with a Jetty 9 server application that seems to go into some kind of resting state after a longer period of idleness. Normally the memory usage of the Java process is ~500 MB, but after being idle for some time it seems to drop down to less than 50MB. The first request that comes takes up to several seconds to respond whereas requests are normally on the scale of tens of milliseconds. But after one or two requests it seems like the application is back to it's normal responsive state.
I'm running on the 32-bit Oracle Java 8 JVM. My JVM configuration is very basic:
java -server -jar start.jar
I was hoping that this issue might be solvable through JVM configuration. Does anyone know if there's any particular parameter to disable this type of behavior?
edit: Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. See my own answer below for a description of my solution.
Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. This was clearly visible when comparing the private working set to the commit size in the task manager.
My solution to this was two-fold. First, I made a simple scheduled job inside my server app that runs every minute and does a simple test run to make sure that the important services never go inactive for long periods. I'm hoping this should ensure that Windows doesn't regard the related pages as inactive.
Afterwards, I also noticed that the process was executing with "Below normal" priority. So I changed the script that starts the server to ensure that it's running with "High" priority going forward. This seems likely to affect swapping behavior and may very well also have been enough to resolve the issue on it's own, but I only found it after already deploying my first solution so that remains unclear. In any case, everything seems to be working as it should now.
I have a web service running on tomcat. I have configured 30 request threads. The service is working fine. The unusual thing is that the number of threads JMX is displaying are more than 150. My application is not forking any threads in the application java code. I am unable to know what could be the reason for this? I couldn't find anything on google and SO as well..
Using a tool like VisualVM (https://visualvm.java.net/) will give you a complete list of all running threads in your current process.
We are trying to access an application from the tomcat which is on a different host, but it is not loading even though the tomcat is running. It was running fine for the past 3 months. We restarted the tomcat now it is working fine.
But, we could not able to zero in on what happened.
Any idea how to trace / what might have caused this?
The CPU usage was normal and the tomcat memory was 1205640.
the memory setting of tomcat are 1024- 2048(min-max)
We are using tomcat 7.
Help much appreciated....thanks in advance.....cheers!!
...also - not sure on Windows - you may be running out of file descriptors. This typically happens when streams are not properly closed in finally blocks.
In addition, check with netstat if you have a lot of sockets remaining open or accumulating in wait state.
Less likely, the application is creating threads and never releasing them.
The application is leaking something (memory, file descriptors, sockets, threads,...) and running over a limit.
There are different ways to track this. A profiler may help or more simply, running JVM dumps at regular intervals and checking what is accumulating. The excellent MAT will help you analyze the dumps.
Memory leak problems are not uncommon. If your Tomcat instance was running for three months and suddenly the contained application became unresponsive maybe that was the case. One solution (and if your resources allow you to do so) could be monitoring that Tomcat instance though JMX using jconsole to see how it behaves
[UPDATE: I forgot to add that this 30 sec. freezing problem only happens the first time I try to load a file from the server. Subsequent loads are very quick. Maybe some strange reverse DNS lookup? I am hosting on Google's appengine.]
I started a little project recently called http://www.chartle.net which is build around an applet.
Startup time is an important factor in the user's experience of an applet. I collect statistics and am shocked that I find often very long startup times (factor 50 to 100 higher then necessary)
The applet starts in 1-3 seconds depending on the speed of your computer and connection. Still for some users it takes up to 100 sec.
I have mixed results from my own tests. Mostly it is very fast but sometimes freezes the browser for a long time and the Java console doesn't tell me why. Best guess is, that it stalls when loading a saved chart.
Please help me figuring this out - best test by opening an already saved chart (click on one of the 'create' links at http://www.chartle.net/gallery)
Cheers,
Dieter
This is generic help rather than specific for your demo (which loaded fine for me in a few attempts).
Freezing applets
In the JDK bin directory there is a very handy programme called jstack. Refresh your browser window until it crashes and then run:
jstack *process_id*
This will give you the stack trace of any frozen Java process. If Java is not a separate process then you can use the browser's process (eg for Opera).
The following few problems were/are common for me:
I reccommend you use invokeLater rather than invokeAndWait on the init method (although you can't do this if you use start/stop methods)
Opera's custom java plugin acts very poorly...
Deadlocks caused by synch blocks and invokeAndWait's
Slow applets
Possibly the browser is fetching resources from the server, unable to use the jar file?
It may be that only the old plugin causes these problems. That means basically all people running on OSX and other users with Java prior to 1.6_update_10.
So, I would really appreciate people with such setups to watch their Java console and describe the first startup behaviour.
Cheers,
Dieter