This question already has an answer here:
tomcat8 at 100% cpu when trying https on port 80
(1 answer)
Closed 5 years ago.
I have started experiencing a strange issue with my tomcat8 server. I use it to exclusively run two applications - libresonic (a music streaming app) and guacamole (remote desktop gateway)
I am experiencing the tomcat process taking 100% of available CPU after the server has been running for a few hours with either application deployed. In order to troubleshoot I have done the following:
Spun up a vanilla Debian 8.6 Virtual Machine using KVM and installed:
Tomcat8
jdk-8 - 1.8.0_111
If I leave the tomcat instance running with no applications deployed the server and CPU usage remain inactive
If I deploy one of the applications (it doesn't matter which one), after a few hours the CPU usage climbs to 100%. Killing and restarting the tomcat server causes the CPU usage to drop, and then climb back to 100% after a few hours
Note that memory usage remains steady with plenty of free memory, so I don't believe this is a GC issue. Nothing related to memory is reported in the logs.
Catalina.out does not report any errors
I have taken threaddumps during the period of high CPU when each application is deployed. Other than being able to identify the threads that are in runnable state and consuming CPU, I cannot establish the root cause or ideas to rectify/fix the issue.
Can someone help? Threaddumps are linked below
Download threaddumps
Perhaps it's related to this case (https://bz.apache.org/bugzilla/show_bug.cgi?id=57544). I have actually the same symptoms with tomcat 8.0.14. What is your tomcat version?
Related
I have a j2ee application running on windows, the application is running fantastic for first 5 min, then after 5 min I see a severe performance degradation.
On the java flight controller I see some time spent on WAIT and Socket Read
But I am not sure how to analyze this further,
This is just a guess but Is there any limitation on windows on sockets/connections like on linux?
in linux max number of sockets/ files is configured in /etc/security/limits.conf
I have a simple 3 tier setup of an Apache server which sends requests to a Tomcat server, which queries a (MySQL) database to generate HTML results. I'm finding that as soon as Tomcat is started, there are threads in the java process that are spinning away making futex() calls. After a few dozens web requests, the threads trying to serve requests get caught in the same futex() loops, and it stops answering all requests -- they time out on the client side.
I have tried this in Tomcat 6 and Tomcat 7. I have tried it with Oracle's Java 1.7.0_45 and OpenJDK 1.6.0. This VM is a 64 bit Redhat 6 system, and I have tried with their released 2.6.32-358.23.2 kernel and their 2.6.32-431.3.1 kernel, and all combinations are showing these system calls in strace, and eventually locking up.
futex(an addr, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {a timestamp}, ffffffff)
= -1 ETIMEDOUT (Connection timed out)
futex(an addr, FUTEX_WAKE_PRIVATE, 1) = 0
The JVM does this with the default memory, or even if I increase the available mem to 3GB (of the 4GB on the machine). I ran with a GC logger, and GC printed a few minor collections, and was not doing one when the lockup occurred. This machine was created in Jan 2014, so is not in any "leap second" situations.
So my questions would be: why is Java making all of these futex() calls in a fast loop even when the JVM should be "idle"? are they normal? should they be getting the timeout? and is there a known fix?
Thank you for info insights.
i have the same problem,i suspect its the "leap second" caused so. my java processes have a high cpu load for a long time.i get a file "leap-a-day.c" from http://marc.info/?t=134138331900001&r=1&w=2,and run as "./leap-a-day -s",and cpu load suddenly become low,i dont know why....my os is redhat as6.5
I have a strange behaviour and maybe you can help me with it.
The environment is
jdk_7u40 (Tried with jdk_7u51 with same behaviour)
debian 6.0 (on windows I have never had this problem)
jboss 7.1.1
Geoserver 2.4.x (tried .3 and .4 with same result) which is based on spring framework
other war modules (not spring-based, but geoserver has some dependencies on them)
The problem is that after a couple of hours that jboss is running, when I try to login to the web interface of geoserver (a POST to the j_spring_security servlet) it took A LOT (4-5 minutes) to land to the welcome page of the application.
Using jstack, I found that there is a thread that consumes 100% of a core for all the time, with and the process keep working here
at sun.security.provider.SHA2.lf_S(SHA2.java:162)
at sun.security.provider.SHA2.lf_sigma0(SHA2.java:171)
at sun.security.provider.SHA2.implCompress(SHA2.java:225)
at sun.security.provider.SHA2.implDigest(SHA2.java:118)
at sun.security.provider.DigestBase.engineDigest(DigestBase.java:186)
at sun.security.provider.DigestBase.engineDigest(DigestBase.java:165)
at java.security.MessageDigest$Delegate.engineDigest(MessageDigest.java:576)
at java.security.MessageDigest.digest(MessageDigest.java:353)
at java.security.MessageDigest.digest(MessageDigest.java:399)
at org.jasypt.digest.StandardByteDigester.digest(StandardByteDigester.java:979)
- locked <0x00000006f8c30bb0> (a java.security.MessageDigest$Delegate)
at org.jasypt.digest.StandardByteDigester.matches(StandardByteDigester.java:1099)
at org.jasypt.digest.StandardStringDigester.matches(StandardStringDigester.java:1052)
at org.jasypt.util.password.StrongPasswordEncryptor.checkPassword(StrongPasswordEncryptor.java:99)
at org.jasypt.spring.security3.PasswordEncoder.isPasswordValid(PasswordEncoder.java:204)
at org.geoserver.security.password.AbstractGeoserverPasswordEncoder.isPasswordValid(AbstractGeoserverPasswordEncoder.java:138)
at org.geoserver.security.password.GeoServerMultiplexingPasswordEncoder.isPasswordValid(GeoServerMultiplexingPasswordEncoder.java:75)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.additionalAuthenticationChecks(DaoAuthenticationProvider.java:64)
Some of you had a similar issue?
EDIT (with workaround)
I find out that the problem is related to the CMS garbage collector and to the increase of the permgen space.
Environment
The application server is JBoss 7.1.1 with 5 war deployed in it (Geoserver and others). There are shared dependencies among all the wars (with Geoserver too); Java is running with -XX:+UseParallelOldGC -XX:SoftRefLRUPolicyMSPerMB=36000
What happens
When a full gc is performed, the permgen space is increased a lot above the used. After that, the computation of methods in sun.security.provider.SHA2.* became very slow.
How did I solve
Moving to G1GC garbage collector solved the problem for me (currently I'm using the following options -XX:+UseG1GC -XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=1 -XX:NewRatio=1 -XX:MaxTenuringThreshold=15 -XX:G1HeapRegionSize=32m )
I haven not. Can you report the issue on GeoServer own bug tracker? http://jira.codehaus.org/browse/GEOS
We are trying to access an application from the tomcat which is on a different host, but it is not loading even though the tomcat is running. It was running fine for the past 3 months. We restarted the tomcat now it is working fine.
But, we could not able to zero in on what happened.
Any idea how to trace / what might have caused this?
The CPU usage was normal and the tomcat memory was 1205640.
the memory setting of tomcat are 1024- 2048(min-max)
We are using tomcat 7.
Help much appreciated....thanks in advance.....cheers!!
...also - not sure on Windows - you may be running out of file descriptors. This typically happens when streams are not properly closed in finally blocks.
In addition, check with netstat if you have a lot of sockets remaining open or accumulating in wait state.
Less likely, the application is creating threads and never releasing them.
The application is leaking something (memory, file descriptors, sockets, threads,...) and running over a limit.
There are different ways to track this. A profiler may help or more simply, running JVM dumps at regular intervals and checking what is accumulating. The excellent MAT will help you analyze the dumps.
Memory leak problems are not uncommon. If your Tomcat instance was running for three months and suddenly the contained application became unresponsive maybe that was the case. One solution (and if your resources allow you to do so) could be monitoring that Tomcat instance though JMX using jconsole to see how it behaves
I have a standalone application that is running in IBM Websphere 7.0.0.19. It is running in Java 6 and we pack an Axis2 JAR in our EAR. We have 'parent last' style class loading and we have disabled the Axis service that is packed with WAS7 by default.
Recently, after 6+ weeks of continuous functioning, the application experienced an OOM. Perplexing point is, the application is deployed seperately on 2 different machines. But only one machine went down. Second machine is still up.
We checked OS, server configuration like classloader policy using WAS console and they are similar in both machines.
When the application crashed, it generated a .phd file which we analysed using Eclipse Memory Analyser Tool (MAT). The analysis is shown in the screenshot.
If I'm correct the bootstrap class loader is repeatedly loading and holding on to references of AxisConfiguraiton and so GC is unable to collect them when it runs. But, if that is the case, then both servers must have come down. But only one server experienced an OOM. Memory allocated to JVM is same in both machines.
We are not sure whether the issue is with WAS 7 or with axis2-kernel-1.4.1.jar or with something else.
http://www.slideshare.net/leefs/axis2-client-memory-leak
https://issues.apache.org/jira/browse/AXIS2-3870
http://java.dzone.com/articles/12-year-old-bug-jdk-still-out
(Links may not refer to the current issue. But they are just pointers)
Has anyone experienced something similar ?
We saw memory growth and sockets left open on WebSphere 6.1 with Axis 2 1.4 in the past. It's been a long time, but my notes suggest it might be worth considering an upgrade to at least Axis 2 1.5.1 to fix this bug with the open sockets and also to ensure you're not creating new objects repeatedly where a singleton exists (e.g. the Service object).