I am currently facing an issue with my java webapp running on Jetty 7.4.5.v20110725 on Linux. My Webapp serving static content is running out of file descriptors after a few days from its start timestamp. I am starting the jetty server with useFileMappedBuffer = true( in webDefaults.xml ) . I am using jdk1.6.0_30 . Please let me know if you'll have any suggestions on how to fix this issue.
Please note that this issue does not occur when useFileMappedBuffer = false (in webDefaults.xml).
If your application has run out of timestamps next time, please try to find out which files are open and if open connections are causing a problem.
Try to list open files by calling (I think it's lsof -p, but try or look at the lsof manpage as I'm writing this out of my mind):
lsof -p <jettypid>
This will show you what files are opened by the jetty process. Look for files which probably should have been closed already, etc.
Then do a:
netstat -an
This will show you established network connections. Look if there's lots of connections in CLOSE_WAIT state or similar indicating that connections are not properly closed.
Also have a look at your system wide OS limits with:
ulimit -a
It'll show you how many file descriptors can be opened by a single process. If you've a site with high traffic and the pretty common default value of 1024 max fd, you might need to raise that. If you think that traffic is the problem have a look at this guide: http://wiki.eclipse.org/Jetty/Howto/High_Load
However you've stated that the problem does occur only after a couple of days. This usually indicates not properly closed connections, file resources, etc.
If unsure what to do with the output of the commands above, feel free to paste them.
Independent of the problem I'd recommend you to upgrade to the latest jetty 7.x.
Related
I am running Apache Tomcat 7.0.42 on Ubuntu 12.04. When I start up Apache initially it's all fine and I can connect. However after using it for a while I can no longer browse to any site hosted on Apache, not even the root site. The browser just waits for a connection forever. If I go on the machine I can see lots of connections that have been closed and are CLOSE_WAIT on netstat, but nothing open. The Apache log file doesn't tell me anything either.
If I restart Apache the problem doesn't go away, even rebooting the server hasn't solved the problem. After struggling for a while it starts working again only to stop shortly afterwards.
I don't have any network issues between the machines.
I had a look at the known bugs on 7, but don't notice anything that sounds related. I have also reinstalled Apache on the machine, but the problem returned soon after.
I feel like I'm missing something silly, but I'm just not getting it.
I am not sure if this will help, but you can give a try for below options.
The problem may not be for Tomcat7, but for your own site. I have used tomcat for so many days and didnt notice such kind of issue single time.
May be your one of the site is having memory leaks and after running for sometime, it makes server stop because of low memory.
If there are many sites then you can try removing all and adding one by one to check which site is giving the issue.
You can try with simple page first by removing all other sites to check if the issue is with tomcat or your site.
I have a java EE app running in Jboss in Unix and I have noticed that after a couple of days I get "too many open files"
I've identified what is leaking but I have no idea what it actually is or how to find which part of my code that is leaking
my analysis of lsof shows that these rows are slowly increasing:
java 25222 user 220u unix 105,208835 0t465959 55050244 /devices/pseudo/tl#0:ticots->(socketpair: 0x32fc3) (0x6009c7d41f0)
the closest thing to a similar problem is http://tinyurl.com/csmy7ay
I use JMS so the problem might lie there.
But my main question is how should I proceed in my quest to fix this? I'm having a hard time finding anything tangible in the lsof file that might help me determine what kind of traffic this is and where in my code they might be created
I found what the problem was, there was a maintenance script running which, among other things, ran jstack which together with this bug: http://tinyurl.com/c5fdl43 (java attach API in solaris causes too make open files). Caused the file leak
Short description of my problem: I start up Tomcat with my deployed Wicket application. When I want to shut down tomcat I get this error message:
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.lang.ref.Reference.<clinit>(Reference.java:145)
I am running the following setup:
Ubuntu Linux: 10.04 (lucid) with a 2.6.18-028stab094.3 kernel
Java Version: "1.6.0_26" Java HotSpot(TM) 64-Bit Server VM
Tomcat Version: 7.0.23
jvm_args: -Xms512m -Xmx512m -XX:MaxPermSize=205m (these are added via CATALINA_OPTS, nothing else)
Wicket 1.5.1
Tomcat is configured with two virtual hosts on subdomains with ModProxy
My application is deployed as ROOT.war in the appbase directory (it makes no difference if I deploy one or both applications)
'''No application deployed does not result in OOM on shutdown''', unless I mess around with the jvm args
The size of the war is about 500k, all libraries are deployed in tomcat/common/lib (directory which I added to common.loader in conf/catalina.properties)
ulimit -u -> unlimited
When I check the Tomcat manager app it says the following about the JVM memory:
Free memory: 470.70 MB Total memory: 490.68 MB Max memory: 490.68 MB
(http connector) Max threads: 200 Current thread count: 6 Current thread busy: 1
'top' or 'free -m' is similar:
Mem: 2097152k total, 1326772k used, 770380k free, 0k buffers
20029 myuser 18 0 805m 240m 11m S 0 11.7 0:19.24 java
I tried to start jmap to get a dump of the heap, it also fails with an OutOfMemoryError. Actually as long as one or both of my applications are deployed any other java process fails with the same OOM Error (see top).
The problem occurs while the application is deployed. So something is seriously wrong with it. However the application is actually running smoothly for quite a while. But I have seen OOMs in the application as well, so I don't trust the calm.
My application is using a custom filter class? Could that be it?
For completeness (hopefully), here's the list of libraries in my common/lib:
activation-1.1.jar
antlr-2.7.6.jar
antlr-runtime-3.3.jar
asm-3.1.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
c3p0-0.9.1.1.jar
commons-collections-3.1.jar
commons-email-1.2.jar
dependencies-provided.tgz
dom4j-1.6.1.jar
ejb3-persistence-1.0.2.GA.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jta_1.1_spec-1.1.1.jar
hibernate-annotations-3.4.0.GA.jar
hibernate-commons-annotations-3.1.0.GA.jar
hibernate-core-3.3.0.SP1.jar
hibernate-entitymanager-3.4.0.GA.jar
hibernate-search-3.1.0.GA.jar
javassist-3.4.GA.jar
joda-time-1.6.2.jar
jta-1.1.jar
log4j-1.2.16.jar
lombok-0.9.3.jar
lucene-core-2.4.0.jar
mail-1.4.1.jar
mysql-connector-java-5.1.14.jar
persistence-api-1.0.jar
quartz-2.1.1.jar
servlet-api-2.5.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
stringtemplate-4.0.2.jar
wicket-auth-roles-1.5.1.jar
wicket-core-1.5.1.jar
wicket-datetime-1.5.1.jar
wicket-extensions-1.5.1.jar
wicket-request-1.5.1.jar
wicket-util-1.5.1.jar
xml-apis-1.0.b2.jar
I appreciate any hint or even speculation that gives me additional ideas what to try.
Update: I tested some more and found that this behaviour only occurs while one or both of my applications are deployed. The behaviour does not occur on "empty" tomcat (that was a mistake on my part messing with jvm args)
Update2: I am currently experimenting trying to reproduce this behaviour in a virtual box, I want to debug this with a profiler. I am still not convinved that it should be impossible to run my setup on 2GB RAM.
Update3 (10/01/12): I am trying to run jenkins instead of my own application. Same behaviour, so it is definitely server configuration issues. Jenkins jobs fail when maven is called, so I need not even try the shutdown hack suggested below because I need a second java process while running Jenkins. It was suggested to me that because this is a Virtual Server ulimits may be imposed from outside and I would not be able to see them. I think I'll ask a new question regarding this. Thx all.
Update4 (02/05/12): see below for the answer that contains the hint. I'll clarify some more up here: I am now 95% sure that the errors occur because I am reaching my thread limit. However because this is a virtual server the method described below would not work to check this value because it is not visible with ulimit, that was what was confusing me and only today I found out that this is the "numproc" value that I can see in the Parallels Power Panel that I can log into for my virtual server. There were Resource Alerts for numproc but I did not see those either until just now. The value has a hard limit of 96 which I cannot change of course. The current value of numproc corresponds to the number of processes I see with "top" after toggling "H" to see threads. I had a very hard time finding this because this numproc value is hidden deep inside the panel. Sadly 96 is a rather low number if you want to run a tomcat with apache and mysql. I am also very sad that I cannot even find this value in the small print of my hosting contract and it is rather relevant to my application I dare say. So I guess I'll need a server upgrade.
Thanks all for your helpful answers in the end everyone helped me a bit to find out what the problem was.
The tomcat shutdown procedure consits of sending an command/word via a tcp port to the running tomcat VM. This port is configured in the server.xml (if I remember corretly, writting on my phone right now). So far so good.
Unfortunately, the shutdown script does this by starting a 2. VM using the same java options used for the tomcat. Your system simply has not enough memory for this.
As a sollution you could write your own stop script using telnet or something.
I could help with later if needed.
Hope that helps.
Viele grüsse Bert
Seems you have too many threads open.
Use this command :
ulimit -u
What is the result ?
Should be something like :
max user processes (-u) 100
If this is correct, you can edit this file :
/etc/security/limits.conf
and the the following modifications :
#<domain> <type> <item> <value>
user soft nproc 10000
user hard nproc 10000
You can probably survive for a while like this. All you need to do is kill the tomcat process whenever you need to restart it. It is not a nice approach, but the main concern is that your application runs correctly.
It seems to me though, that on the long run, you might need to order a hosting plan with more RAM available.
I was having a similar problem with a tomcat installation just last week. I managed to fix it by giving tomcat a smaller heap. Something like this:
export CATALINA_OPTS=”-Xms256m -Xmx512m”
Before starting Tomcat may help. In the meantime you'll have to kill it the old fashioned way, with a kill -9 ;)
EDIT: you could also take look here, it appears tomcat automatically creates a bunch of "spare" threads, but you can limit those as well as your max thread count in the config. Hope it helps.
I use eclipse to develop a web based java application. My normal course of business is grab the next task tracking ticket. If there is a problem that needs to correcting, I run the application locally, which loads of a Jetty webserver, and binds to port 8080. After verifying the problem, I fix the problem, rebuild, and the re-run the application. The problem is, I far too often forget to close the jetty server before re-running. This generates the Java bind error:
WARNING: failed SocketConnector#0.0.0.0:8080: java.net.BindException: Address already in use: JVM_Bind
I work in Windows, and was looking to see if there is command I could run to un-bind the port, but couldn't find an answer there. Does anyone here have a good idea of how to fix my problem, other than remember to shut down the old jetty instance before starting a new one?
Thanks,
Jay
Use netstat -ap | grep 8080 to find which process is using that port. (What, you don't have a suitable grep installed on Windows? Everyone should!)
Use -Djetty.port=<port number> to change the default port number used by Jetty.
To unbind the port cleanly, you typically need to stop the process that is holding it open (otherwise, the process would probably die with strange communication errors anyway).
You could right a short batch file to detect if the port is in use and kill the Jetty process. Assuming you have Sysinternal's pskill.exe available in your path (e.g. under C:\Windows), here's an example that will kill the process which is holding port 8080 open:
FOR /F "skip=4 tokens=2,5" %%G IN ('netstat -ano') DO (
IF "%%G"=="127.0.0.1:8080" pskill %%H
)
You sortof answered your own question there... since you can't run two applications on the same port you'll need to remember to shut down the old one before starting a new one.
You mention the application starts a Jetty instance, could that launching application check if it's already running before doing so? I guess I'm not quite on the same page as you...
However, there are some nifty ways out there to "hot deploy" your changes -- once you have the appserver running you can make changes to JSP files or Java classes and it will automatically deploy those changes.
Are you using some integration like run-jetty-run? If not, you might want to give that a try.
MyEclipse also comes to mind, though I'm not a huge fan of it, some people really like it.
But for the most part, just remember to stop and restart.
I'm trying to run jstack command on my java application. Application is rather big, running inside jboss AS occupying about 4gb of memory. OS is Windows Server 2003 Standard edition. Every time i get an error "Not enough storage is available to process this command". There is enough ram, 16gb, and disk space. So, any ideas?
I ran into this recently on Win2008r2 and thought I'd share my solution since it took a while to figure out. Rob's comment about psexec -s is what did it for me.
It appears that on Vista and later jstack doesn't work against services because of the user context. It has nothing to do with memory. I suspect this is the same reason people have seen this problem on 2003 via remote desktop, unless you use the /admin or /console switch on mstsc. As of Vista the tightened security is probably what broke it.
Starting my app from a cmd window worked fine, but that doesn't help me debug our standard install. Enabling the java debug port (for VisualVM, Eclipse or most any Java debugger) requires an app restart, so you lose the state you're probably trying to capture if you don't already have debugging enabled. Starting the service under my user credentials did not work - I was a little surprised at that. But psexec -s runs jstack from the system context, which worked like a charm. Oh, and you'll need to run psexec from an elevated cmd prompt, if UAC is on.
In the past I have seen this when the JVM is running as a Windows Service on Windows 2003.
First, check to see if this is an issue with the TMP directory.
Second, jstack (or the other utilities like jconsole) will not connect to the local process unless it is running in the same session. If the service is running as a specific user, you may be able to connect by logging into the same session. If you are using Remote Desktop, you can connect using "mstsc /admin" (used to be /console) and try to run jstack again. Definitely check to make sure the TMP directory is set properly if this doesn't fix the problem.
If the service is running as LocalSystem, the above procedure probably will not help much. I don't know if there is a way to log into the same session as LocalSystem.
Some other alternatives may be to set the process up for remote monitoring and use jvisualvm (from the server itself or another machine) to connect over a port and do a thread dump.
We had problems running JStack on a Windows machine with even a modest application (1GB). We ended up doing our stack and heap analysis using Netbeans. This seemed to cope with the parsing of dump files a lot better. YMMV.
Give Netbeans a try for profiling - its very good. Note that VisualVM is a cutdown NB profiler and comes with 6u7.
psexec -s jstack PID >> c:\jstack.log perfectly works on the same machine. For the first time it took some time but again I executed with the redirect to file option, it completed with in few seconds.
This is an error message from the underlying O/S. There's not much you can do in your code to deal with this other than catch the exception which is thrown. Boo to Windows for being so limited.
http://technet.microsoft.com/en-us/library/cc978735.aspx