I have a j2ee application running on windows, the application is running fantastic for first 5 min, then after 5 min I see a severe performance degradation.
On the java flight controller I see some time spent on WAIT and Socket Read
But I am not sure how to analyze this further,
This is just a guess but Is there any limitation on windows on sockets/connections like on linux?
in linux max number of sockets/ files is configured in /etc/security/limits.conf
Related
I am using Tomcat 9.0.0.M22 with jdk1.8.0_131 on Windows Server 2012 R2 and I have a Spring Boot web application deployed on it, the issue is that every 10 seconds the commons daemon service runner spikes the cpu to 50% although my deployed web application is idle then decreases to 0% and this behavior continue to happen every 10 seconds.
In my application I don't have any job that runs every 10 seconds, and also when I run my web application on Tomcat from Eclipse I didn't notice the same behavior, so I am guessing that this is a Tomcat built in thread.
do a series of thread dumps as soon as cpu load spikes
you can use jdk/bin/jvisualvm to connect to your tomcat and repeatedly press thread dump button on upper right of threads tab or if you prefer command line (e.g. via script), you can also use jdk/bin/jcmd <pid-of-your-tomcat> Thread.Print >> dumps.txt
each dump shows all threads existing at that moment and a stack trace for each thread showing what is being executed
this should give you some hints what's involved creating that load
Without more information this is just guessing but this could be the garbage collector trying to do it's job every ten seconds but not being able to evict any items because they are all still needed. You could try to increase the memory for Tomcat (-Xmx).
With that much info it's pretty tough, couple of points you can think of:
As #jorg pointed out, you can take thread dumps that will give you
insights about any blocking threads.
You said it's working fine on local system, it doesn't necessarily mean that code is optimal for the server platform. Double check configurations, maxThreads etc.
Optimize Server JVM by eliminating any excessive garbage collection. Starting the JVM with a higher maximum heap memory (-Xmx) will decrease the frequency with which garbage collection occurs.
Existing monitoring tools can leverage your analysis (jVisualVM) .
I was able to stop this behavior completely by setting reloadable="false" in context.xml.
This question already has an answer here:
tomcat8 at 100% cpu when trying https on port 80
(1 answer)
Closed 5 years ago.
I have started experiencing a strange issue with my tomcat8 server. I use it to exclusively run two applications - libresonic (a music streaming app) and guacamole (remote desktop gateway)
I am experiencing the tomcat process taking 100% of available CPU after the server has been running for a few hours with either application deployed. In order to troubleshoot I have done the following:
Spun up a vanilla Debian 8.6 Virtual Machine using KVM and installed:
Tomcat8
jdk-8 - 1.8.0_111
If I leave the tomcat instance running with no applications deployed the server and CPU usage remain inactive
If I deploy one of the applications (it doesn't matter which one), after a few hours the CPU usage climbs to 100%. Killing and restarting the tomcat server causes the CPU usage to drop, and then climb back to 100% after a few hours
Note that memory usage remains steady with plenty of free memory, so I don't believe this is a GC issue. Nothing related to memory is reported in the logs.
Catalina.out does not report any errors
I have taken threaddumps during the period of high CPU when each application is deployed. Other than being able to identify the threads that are in runnable state and consuming CPU, I cannot establish the root cause or ideas to rectify/fix the issue.
Can someone help? Threaddumps are linked below
Download threaddumps
Perhaps it's related to this case (https://bz.apache.org/bugzilla/show_bug.cgi?id=57544). I have actually the same symptoms with tomcat 8.0.14. What is your tomcat version?
Need some help from the experts!
We have a project here (still on dev) that needs to run 50 java processes (for now and it will probably doubled or tripled in the future) at the same time every 5 minutes. I set Xmx50m for every process and our server has only 4gb of RAM, I know that would really slow our server. What I have in mind is to upgrade our RAM. My question is that do I have other options to prevent our server from being slow when running that amount of java processes?
Since you have 50 process and as per your assumption your processes need about 2.5 Gb to run .
To prevent your server from being slow you can follow some best practices to set java memory parameters e.g. set -Xmin and -Xmx the same values and determine a proper values based on your process usage, Also you can profile your process on runtime to ensure that everything is ok.
I have a simple 3 tier setup of an Apache server which sends requests to a Tomcat server, which queries a (MySQL) database to generate HTML results. I'm finding that as soon as Tomcat is started, there are threads in the java process that are spinning away making futex() calls. After a few dozens web requests, the threads trying to serve requests get caught in the same futex() loops, and it stops answering all requests -- they time out on the client side.
I have tried this in Tomcat 6 and Tomcat 7. I have tried it with Oracle's Java 1.7.0_45 and OpenJDK 1.6.0. This VM is a 64 bit Redhat 6 system, and I have tried with their released 2.6.32-358.23.2 kernel and their 2.6.32-431.3.1 kernel, and all combinations are showing these system calls in strace, and eventually locking up.
futex(an addr, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {a timestamp}, ffffffff)
= -1 ETIMEDOUT (Connection timed out)
futex(an addr, FUTEX_WAKE_PRIVATE, 1) = 0
The JVM does this with the default memory, or even if I increase the available mem to 3GB (of the 4GB on the machine). I ran with a GC logger, and GC printed a few minor collections, and was not doing one when the lockup occurred. This machine was created in Jan 2014, so is not in any "leap second" situations.
So my questions would be: why is Java making all of these futex() calls in a fast loop even when the JVM should be "idle"? are they normal? should they be getting the timeout? and is there a known fix?
Thank you for info insights.
i have the same problem,i suspect its the "leap second" caused so. my java processes have a high cpu load for a long time.i get a file "leap-a-day.c" from http://marc.info/?t=134138331900001&r=1&w=2,and run as "./leap-a-day -s",and cpu load suddenly become low,i dont know why....my os is redhat as6.5
I've installed weblogic 10.3.3, configured simple domain with default configuration. And after starting weblogic I cannot use it via admin console because it is starting too slow (~10 mins). It seems that admin console application is being deployed incorrectly. Even when I enable 'staging' mode in AdminConsole deployment options, it does not help. Does anybody have a solution for this problem? It really bugs me out.
Configuration properties:
Server: Dell PowerEdge r410 server (Six-core Intel® Xeon® x64 CPU, 8GB RAM)
OS: Ubuntu Maverick 10.10 x86_64
Weblogic: 10.3.3 x64 (used wls1033_generic.jar file for installation)
Java: 1.6.0_17_i586 (tried with different jdk's including x64, but it does not work either)
Turns out weblogic uses random number generator during startup. Because of the bug in java it reads random bits from /dev/random. There are almost no problems with /dev/random except that it is extremely slow. It takes sometimes 10 minutes or more to generate one number. Simple solution exists – using /dev/urandom instead. It is not that good as /dev/random, but at least it is instant.
Java somehow maps /dev/urandom file to /dev/random. That’s why default settings in $JAVA_HOME/jre/lib/security/java.security are useless, they just do not make any sense.
Problem fix is very simple – adding string export JAVA_OPTIONS="-Djava.security.egd=file:/dev/./urandom" to the /etc/bash.bashrc file. Usage of /dev/./urandom instead of simple /dev/urandom is another hack. JVM does not understand the option value otherwise.
Be aware of this problem if you try to setup weblogic under UNIX-based OS.
Without looking at the startup logs we could only speculate. Weblogic typically is a beast. It typically takes anywhere fromo 6-8 minutes to start up for me on my development machines, but that seems like a nice server for it to take 10 minutes for you.
The Admin Console should not be this slow, nor should it take 6-8 minute to start a server or deploy the Admin Console. The deployment of the admin console on my machine takes less than 30 seconds, and startup is under 60 seconds. Startup time for WebLogic is a common complaint, but chances are that something is misconfigured if there is a very long startup or admin console deployment time.