Short description of my problem: I start up Tomcat with my deployed Wicket application. When I want to shut down tomcat I get this error message:
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.lang.ref.Reference.<clinit>(Reference.java:145)
I am running the following setup:
Ubuntu Linux: 10.04 (lucid) with a 2.6.18-028stab094.3 kernel
Java Version: "1.6.0_26" Java HotSpot(TM) 64-Bit Server VM
Tomcat Version: 7.0.23
jvm_args: -Xms512m -Xmx512m -XX:MaxPermSize=205m (these are added via CATALINA_OPTS, nothing else)
Wicket 1.5.1
Tomcat is configured with two virtual hosts on subdomains with ModProxy
My application is deployed as ROOT.war in the appbase directory (it makes no difference if I deploy one or both applications)
'''No application deployed does not result in OOM on shutdown''', unless I mess around with the jvm args
The size of the war is about 500k, all libraries are deployed in tomcat/common/lib (directory which I added to common.loader in conf/catalina.properties)
ulimit -u -> unlimited
When I check the Tomcat manager app it says the following about the JVM memory:
Free memory: 470.70 MB Total memory: 490.68 MB Max memory: 490.68 MB
(http connector) Max threads: 200 Current thread count: 6 Current thread busy: 1
'top' or 'free -m' is similar:
Mem: 2097152k total, 1326772k used, 770380k free, 0k buffers
20029 myuser 18 0 805m 240m 11m S 0 11.7 0:19.24 java
I tried to start jmap to get a dump of the heap, it also fails with an OutOfMemoryError. Actually as long as one or both of my applications are deployed any other java process fails with the same OOM Error (see top).
The problem occurs while the application is deployed. So something is seriously wrong with it. However the application is actually running smoothly for quite a while. But I have seen OOMs in the application as well, so I don't trust the calm.
My application is using a custom filter class? Could that be it?
For completeness (hopefully), here's the list of libraries in my common/lib:
activation-1.1.jar
antlr-2.7.6.jar
antlr-runtime-3.3.jar
asm-3.1.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
c3p0-0.9.1.1.jar
commons-collections-3.1.jar
commons-email-1.2.jar
dependencies-provided.tgz
dom4j-1.6.1.jar
ejb3-persistence-1.0.2.GA.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jta_1.1_spec-1.1.1.jar
hibernate-annotations-3.4.0.GA.jar
hibernate-commons-annotations-3.1.0.GA.jar
hibernate-core-3.3.0.SP1.jar
hibernate-entitymanager-3.4.0.GA.jar
hibernate-search-3.1.0.GA.jar
javassist-3.4.GA.jar
joda-time-1.6.2.jar
jta-1.1.jar
log4j-1.2.16.jar
lombok-0.9.3.jar
lucene-core-2.4.0.jar
mail-1.4.1.jar
mysql-connector-java-5.1.14.jar
persistence-api-1.0.jar
quartz-2.1.1.jar
servlet-api-2.5.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
stringtemplate-4.0.2.jar
wicket-auth-roles-1.5.1.jar
wicket-core-1.5.1.jar
wicket-datetime-1.5.1.jar
wicket-extensions-1.5.1.jar
wicket-request-1.5.1.jar
wicket-util-1.5.1.jar
xml-apis-1.0.b2.jar
I appreciate any hint or even speculation that gives me additional ideas what to try.
Update: I tested some more and found that this behaviour only occurs while one or both of my applications are deployed. The behaviour does not occur on "empty" tomcat (that was a mistake on my part messing with jvm args)
Update2: I am currently experimenting trying to reproduce this behaviour in a virtual box, I want to debug this with a profiler. I am still not convinved that it should be impossible to run my setup on 2GB RAM.
Update3 (10/01/12): I am trying to run jenkins instead of my own application. Same behaviour, so it is definitely server configuration issues. Jenkins jobs fail when maven is called, so I need not even try the shutdown hack suggested below because I need a second java process while running Jenkins. It was suggested to me that because this is a Virtual Server ulimits may be imposed from outside and I would not be able to see them. I think I'll ask a new question regarding this. Thx all.
Update4 (02/05/12): see below for the answer that contains the hint. I'll clarify some more up here: I am now 95% sure that the errors occur because I am reaching my thread limit. However because this is a virtual server the method described below would not work to check this value because it is not visible with ulimit, that was what was confusing me and only today I found out that this is the "numproc" value that I can see in the Parallels Power Panel that I can log into for my virtual server. There were Resource Alerts for numproc but I did not see those either until just now. The value has a hard limit of 96 which I cannot change of course. The current value of numproc corresponds to the number of processes I see with "top" after toggling "H" to see threads. I had a very hard time finding this because this numproc value is hidden deep inside the panel. Sadly 96 is a rather low number if you want to run a tomcat with apache and mysql. I am also very sad that I cannot even find this value in the small print of my hosting contract and it is rather relevant to my application I dare say. So I guess I'll need a server upgrade.
Thanks all for your helpful answers in the end everyone helped me a bit to find out what the problem was.
The tomcat shutdown procedure consits of sending an command/word via a tcp port to the running tomcat VM. This port is configured in the server.xml (if I remember corretly, writting on my phone right now). So far so good.
Unfortunately, the shutdown script does this by starting a 2. VM using the same java options used for the tomcat. Your system simply has not enough memory for this.
As a sollution you could write your own stop script using telnet or something.
I could help with later if needed.
Hope that helps.
Viele grüsse Bert
Seems you have too many threads open.
Use this command :
ulimit -u
What is the result ?
Should be something like :
max user processes (-u) 100
If this is correct, you can edit this file :
/etc/security/limits.conf
and the the following modifications :
#<domain> <type> <item> <value>
user soft nproc 10000
user hard nproc 10000
You can probably survive for a while like this. All you need to do is kill the tomcat process whenever you need to restart it. It is not a nice approach, but the main concern is that your application runs correctly.
It seems to me though, that on the long run, you might need to order a hosting plan with more RAM available.
I was having a similar problem with a tomcat installation just last week. I managed to fix it by giving tomcat a smaller heap. Something like this:
export CATALINA_OPTS=”-Xms256m -Xmx512m”
Before starting Tomcat may help. In the meantime you'll have to kill it the old fashioned way, with a kill -9 ;)
EDIT: you could also take look here, it appears tomcat automatically creates a bunch of "spare" threads, but you can limit those as well as your max thread count in the config. Hope it helps.
Related
I have Tomcat 9 installed as service on Windows 7 64 bit. I want to:
see what heap size is currently configured and active
update the service configuration persistently to use a different heap size
verify that Tomcat is actually using the new heap size.
do all that from the command line.
For 1 and 3: I naively tried to use jconsole, but I don't find the process there because Tomcat is running as Local System Service. While I found out how to run jconsole as Local System Account, it seems that JMX is deactivated when Tomcat is installed as service.
So finding out the currently used memory sizes by JMX seems to be at least very complicated (possibly enabling JMX remote, which should be over TLS...).
For 2: I suppose this is the corresponding place of Tomcat's documentation, which reads:
To update the service parameters, you need to use the //US// parameter.
Update the service named 'Tomcat9'
C:\> tomcat9 //US//Tomcat9 --Description="Apache Tomcat Server - http://tomcat.apache.org/ " ^
--Startup=auto --Classpath=%JAVA_HOME%\lib\tools.jar;%CATALINA_HOME%\bin\bootstrap.jar
But I don't understand that text sufficiently to apply it to my problem. Especially, I don't want to change other parameters (like description, startup etc.).
As far as I understand, when run as a service, configuration is stored in the Windows registry, so the usual configuration in tomcat/conf does not or at least does only partly apply.
Please note that this question is not about installing Tomcat, but about modifying an existing installation. Also I am not interested in some hacky way to get the desired result (somehow), but in the best practice to do that; and it would be perfect to have links to reference documentation for that.
Given the documentation that you link to
--JvmMx
Maximum memory pool size in MB. (Not used in exe mode.)
should help to control the heap size.
For fetching the current values, tools such as jvmtop might be easiest answer.
For 1 and 3 there is jmap. You just need to know the process Id of the tomcat instance running.
jmap -heap 7082
Here is the output from a running jvm I have right now(the relevant lines):
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 1073741824 (1024.0MB) // that is -Xmx flag
....
NewSize = 357564416 (341.0MB) // 1
MaxNewSize = 357564416 (341.0MB)
OldSize = 716177408 (683.0MB) // 2
1 + 2 = -Xms flag
Unfortunately I can't answer 2, since I've never started tomcat on windows - as a service (I hardly know what that means for windows). But assuming this is a process that is started by windows as a script...
Shouldn't : tomcat9 -Xms512M -Xmx2G... work? Again, just a hint, not sure. The last thing to notice is that the heap can be changed only at start-up of the jvm, you can't do that at runtime while tomcat is running obviously (just in case...).
I'm new in glassfish.
I have a vps in digitalocean with Ram 512 mb.
I just running 1 domain and 1 simple web service application.
But glassfish stop without my command.
Any suggest?
I using glassfish 4.1.1
Edit for Trevor. I forget to say about error log.
I checked the error log. No log that say error or why the glassfish stop. It's running okay when i restart, but after few hours it happen again
There is a good chance you are running out of memory.
By default, the server has -Xmx512m set, which means the heap size can increase to 512MB. Since that is all you have available on your DigitalOcean machine, it will start with a lower amount, and increase as you deploy your application to it. Once GlassFish tries to use more memory than your DigitalOcean machine can spare, it will die.
Decreasing this to something like -Xmx256m will probably give you more stability. For simple apps, you should be OK with that.
Note: you may want to also decrease the -XX:MaxPermSize=192m to -XX:MaxPermSize=128m. If you are on Java 8, then this doesn't matter any more and the value isn't used. For Java 7, decreasing this will help.
Either change this value through the admin console at http://[$HOSTNAME]:4848 and go to Configurations -> server-config -> JVM Settings, then click the JVM Options tab (you will need to change the value, then click save, then restart GlassFish).
Or change it in the domain.xml directly (being careful to get it right)
glassfish41/glassfish/domains/domain1/config/domain.xml
You will notice that that value is in that file twice. One is the actual server-config, used for the server itself, and the other is the "default-config" which is a template used for creating new configurations. Make sure you change the correct one! If you're unsure, just change both.
I have searched and searched and did not help me much hence posting the new question.
Platform
Ubuntu 11.10 server 64 bit
JVM 1.7.0_03
Tomcat 7
There is nothing special in the configuration - front end server is apache using ajp connector. Tomcat runs as ubuntu service.
On our server, tomcat7 is dying and could not figure out the reason. I have checked all the log files (syslog, catalina.out, even auth.log) to see if there is something getting logged.
As per top command server still has around 4gb of memory free and cpu usage is averaging around 35% most of the times.
In order to isolate the problem, is there any way to get the exit status code of tomcat process that terminated?
I read some reports where jvm logging error log in case of jvm crash. I am not seeing it either.
It seems like I need to set ulimit to get the core dump, but not sure how to do it for tomcat service or is the setting valid for all the users.
It seems like I need to set ulimit to get the core dump, but not sure how to do it for tomcat service or is the setting valid for all the users.
One way to do that without interfering with anything else would be to add a ulimit command to the catalina.sh script. (It is a bit hacky ... but it sounds like you are at the point where hackiness might give happiness.)
Trying to diagnose some bizarre Tomcat (7.0.21) and/or JVM errors on a 64-bit linux (CentOS) machine.
I'm load testing our server application and tried hitting it with 100K messages. Launched jvisualvm and kept my eye on the heap the whole time. Everything was looking great* (see below) until I got to about 93K processed messages and then Tomcat just died. Ran a ps on Tomcat's PID number to confirm it was dead.
Up until this crash:
Load test had been running for about 90 minutes; should have finished shortly thereafter since we were at 93K/100K)
CPU was holding strong around 45%
Used heap was around 2GB (plus or minus a bunch after GCs) but heap size grew from 4GB to MAX_HEAP after about 30 minutes
Class loading/unloading was cycling normally
Thread dumps were normal
Nowhere in the server code are any calls to System.exit() - so we can rule that right out (and yes I've double-checked!!!).
I'm not sure if this is Tomcat crashing or the JVM (how do I tell?). And even if I did know, I can't seem to find any indication of what went wrong:
All of the server app's logs just stop without any ERROR messages (even though we have logging universally set to DEBUG and higher)
Tomcat's catalina.out and respect localhost_access_* files just stop without any info
I've heard it is possible to have Tomcat log a coredump when it does but not sure how to do that and online examples aren't helping much.
How would SO go about diagnosing this? What steps should I take to start ruling out all of the possible factors?
Thanks in advance!
If the JVM crashes, you should have a hs_err_pidNNN.log file; you don't have to do anything to enable this. Its location depends on your OS and how you are running Tomcat. On Windows, they can show up on your desktop, unless you are running as a service. Otherwise, they should be in the current working directory of the crashed process.
Your operating system probably provides additional tools for process monitoring; you could describe your environment more, or perhaps ask at serverfault.com.
It's also possible that jvisualvm is actually causing the crash.
I'd try reproducing the problem, and progressively simplify the scenario to help isolate the cause.
Another possibility is that the OS is running out of memory and the OOM Killer is killing your process. In this case, the JVM wouldn't get an opportunity to write a heap dump, or an hs_err_pid file.
You can use the option java -XX:+HeapDumpOnOutOfMemoryError to create a heap dump for jvm crash due to out of memory error.
More details here Using HeapDumpOnOutOfMemoryError parameter for heap dump for JBoss.
Sorry I had to remove the green check from #erickson. I finally figured out what was killing Tomcat.
It looks like a profiler plugin is not configured correctly with VisualVM and attempting to run a profile on the Tomcat process killed it.
Investigating why right now, and will update this answer once I know more.
When I run PowerShell in a remote session (etsn {servername}), I sometimes can't seem to run Java processes, even the most simple:
[chi-queuing]: PS C:\temp> java -cp .\hello.jar Hello
Error occurred during initialization of VM
Could not reserve enough space for object heap
Hello.jar is an "Hello, world!" application that should just print "Hello" to standard output.
So, the question is, is there something special about running processes on the other side of a PowerShell session? Is there something special about how the Java VM works that might not allow treatment like this? The memory is allocated on the remote computer, right? Here is a readout on the physical memory available:
[chi-queuing]: PS C:\temp> $mem = Get-wmiobject -class Win32_OperatingSystem
[chi-queuing]: PS C:\temp> $mem.FreePhysicalMemory
1013000
But, when I remote desktop to the server and ask the OS how much free memory there is, it says 270 MB physical memory free. Let me know what you think!
According to this:
http://msdn.microsoft.com/en-us/library/aa384372(VS.85).aspx
MaxMemoryPerShellMB
Specifies the maximum amount of memory allocated per shell, including the shell's child processes. The default is 150 MB.
Increase Max Memory Per Shell MB
winrm set winrm/config/winrs '#{MaxMemoryPerShellMB="1000"}'
I have a different answer to share with you guys. I found myself in the same situation and increasing memory min/max for Java.exe or using winrm did NOT solve my issue.
I compared two servers: one working and one not working.
I used this link https://technet.microsoft.com/en-us/library/ff520073%28v=ws.10%29.aspx to check my Windows Management Foundation wich is needed to run WINRS and also remote powershell.
the result: Both servers running Windows Server 2008 R2. One server running WMF 2.0, one running WMF 3.0.
To my surprise, the server running 2.0 was working and the one running 3.0 was NOT!
My solution: I upgraded the 3.0 WMF to 4.0!
Just a fyi: we suffered the same symptoms, and had an endless investigation based on the other two answers.
The actual solution for us was changing jdk1.8.0_31 to jdk1.8.0_51.