I'm using Tomcat 7 to deploy my app.
Tomcat goes down randomly and requires me to restart the server time and again to keep it up and running.
Sometimes when i restart ,i check the logs to see waiting for instances to be deallocated.
Other times when i restart ,i get the ERROR 1053.
I tried deleting my temp/work folders.I checked my database connections through hibernate ,i made sure that i have all the required jars in lib under tomcat.
The way i deploy is copy my war file and its respective Applicationfolder into Webapps under Tomcat.Please direct me as to what needs to be done to keep the server stable.
Another important note is that the exception in the logs below,changes from Nullpointer to Illegal state Exception
Any help to identify the root cause will be truly appreciated as this has been an impending and critical issue that has been haunting,inspite of trying different solutions.
My Logs are as follows
INFO: Waiting for 3 instance(s) to be deallocated
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/cep4] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/cep4] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/cep4] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/cep4] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Locale context]) and a value of type [org.springframework.web.servlet.DispatcherServlet$1] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/cep4] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#1655d52d]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [org.springframework.security.core.context.SecurityContextImpl#388100c: Authentication: org.springframework.security.cas.authentication.CasAuthenticationToken#388100c: Principal: com.java.service.MyUserDetails#9154e84; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails#ffff4c9c: RemoteIpAddress: 128.32.111.117; SessionId: CF331A8C27CEA68FAC797ED7E0E654B5; Granted Authorities: com.java.service.MyUserDetails$1#1aca2c25 Assertion: org.jasig.cas.client.validation.AssertionImpl#1a6d280b Credentials (Service/Proxy Ticket): ST-89206-bOamLKnOoLTLDJoIxYqs-cas-p3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/cep4] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Request attributes]) and a value of type [org.springframework.web.context.request.ServletRequestAttributes] (value [SecurityContextHolderAwareRequestWrapper[ org.springframework.security.web.savedrequest.SavedRequestAwareWrapper#5d1e88cf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 27, 2014 2:17:27 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-80"]
Jan 27, 2014 2:17:27 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8443"]
Jan 27, 2014 2:17:27 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load org.hibernate.cfg.ImprovedNamingStrategy. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:258)
at org.springframework.beans.factory.support.AbstractBeanDefinition.resolveBeanClass(AbstractBeanDefinition.java:417)
at org.springframework.beans.factory.support.AbstractBeanFactory.doResolveBeanClass(AbstractBeanFactory.java:1283)
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1254)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:433)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:270)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:120)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Jan 27, 2014 2:17:29 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'maxSpareThreads' to '75' did not find a matching property.
Jan 27, 2014 2:17:29 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'source' to 'org.eclipse.jst.jee.server:cep4' did not find a matching property.
Jan 27, 2014 2:17:29 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-80"]
Jan 27, 2014 2:17:29 PM org.apache.coyote.AbstractProtocol init
Really my colleagues and I faced with a similar issue with nearly the same error logs when using Apache Tomcat 7 as application server. Finally we could solve the problem using helps of contents at Stack Overflow and Apache Tomcat websites. Here I would like to share our experiences in a step by step form of debugging:
Try to find out crashing time due to server time.
Goto log folder of Tomcat and check errors just before the crashing time to the end.
Try to find out the reason of each error and try to solve the problems which caused the errors.
Start to monitor Tomcat behaviour using available tools. We used jConsole, but do not forget to link jConsole to Tomcat.
Try to control important parameters such as changes in Heap Memory and Threads especially in peak time of using your system. Maybe you have to increase specified or default values of Heap Memory or Threads.
P.S.: Do not panic on PS Survivor space! That's full most of time. ;-)
Final Word
Finally in our case by monitoring Tomcat with jConsole, we found out the problem was
insufficient Heap Memory in peak time. The solution was increasing the amount of allocated heap memory of Tomcat.
Related
I developed a webapp with Vaadin and kept it running on an embedded Jetty Server. Now, in order to deploy it on the productive server, I need it to run on a Tomcat v7 server.
I took the WAR-file (built with Maven) and deployed it to Tomcat.
Now, when I call the Tomcat URL in the browser, the webapp starts and shows the Vaadin loading circle. After a few seconds, the circle disappears and Tomcat stops.
The Catalina Logfile says the following:
Nov 17, 2015 11:37:55 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 5830 ms
Nov 17, 2015 11:37:59 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8080"]
Nov 17, 2015 11:37:59 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
Nov 17, 2015 11:37:59 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 17, 2015 11:38:01 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SCHWERWIEGEND: The web application [/myWebapp] is still processing a request that has yet to finish. This is very likely to create memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
Nov 17, 2015 11:38:01 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SCHWERWIEGEND: The web application [/myWebapp] created a ThreadLocal with key of type [com.vaadin.util.CurrentInstance$1] (value [com.vaadin.util.CurrentInstance$1#d8b2a4cd]) and a value of type [java.util.HashMap] (value [{interface com.vaadin.server.VaadinRequest=com.vaadin.util.CurrentInstance#d59e9a8b, class com.vaadin.server.VaadinSession=com.vaadin.util.CurrentInstance#bb1b7c4b, class com.vaadin.server.VaadinService=com.vaadin.util.CurrentInstance#5b5dec40, interface com.vaadin.server.VaadinResponse=com.vaadin.util.CurrentInstance#ea949e2b, class com.vaadin.ui.UI=com.vaadin.util.CurrentInstance#174bb6a7}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Nov 17, 2015 11:38:01 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8080"]
Which fault is likely to cause this behaviour? Do I need to perform certain steps in order to deploy a Jetty webapp on Tomcat?
I recently started working in a project which uses OpenAM for single sign on top of tomcat 7 and server running on Apache 2.4.12.
Installed Java version details:
java version "1.7.0_65"
OpenJDK Runtime Environment (rhel-2.5.1.2.el6_5-x86_64 u65-b17)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
OpenAM 11.0.0 (2013-November-08 10:40)
Most of the time 95% of CPU utilization is by java. I can't find the reason for this high CPU utilization.
We had an issue today with OpenAM, OpenAM Tomcat processes seems to have hung in two servers out of 8 and not listening to the port 8180. After restart, the service is up and running fine with ports listening.
However, I could see below error in openAM logs, I tried to find the root cause but unfortunately I can't, could anyone please help?
Server 1:
Jul 28, 2015 8:33:36 AM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextDestroyed
INFO: WSSERVLET13: JAX-WS context listener destroyed
Jul 28, 2015 8:33:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/sso] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak.
Jul 28, 2015 8:33:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/sso] appears to have started a thread named [Monitor Provider State Updater] but has failed to stop it. This is very likely to create a memory leak.
Jul 28, 2015 8:33:43 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/sso] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#4ee2edc3]) and a value of type [org.glassfish.grizzly.ThreadCache.ObjectCache] (value [org.glassfish.grizzly.ThreadCache$ObjectCache#1467c5ab]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8180"]
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol destroy
Server 2:
Jul 28, 2015 8:34:50 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/sso] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak.
Jul 28, 2015 8:34:50 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/sso] appears to have started a thread named [Monitor Provider State Updater] but has failed to stop it. This is very likely to create a memory leak.
Jul 28, 2015 8:33:43 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/sso] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#4ee2edc3]) and a value of type [org.glassfish.grizzly.ThreadCache.ObjectCache] (value [org.glassfish.grizzly.ThreadCache$ObjectCache#1467c5ab]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8180"]
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8180"]
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8109"]
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-8109"]
Jul 28, 2015 8:33:50 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jul 28, 2015 8:33:50 AM org.apache.coyote.AbstractProtocol init
Other WARNING in Server 2:
Jul 28, 2015 8:35:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl newResourceClass
WARNING: A sub-resource method, public javax.ws.rs.core.Response org.forgerock.openam.forgerockrest.authn.AuthenticationRestService.getMethodNotSupported(), with URI template, "/", is treated as a resource method
Jul 28, 2015 8:35:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl newResourceClass
WARNING: A sub-resource method, public javax.ws.rs.core.Response org.forgerock.openam.forgerockrest.authn.AuthenticationRestService.authenticate(javax.ws.rs.core.HttpHeaders,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.String,java.lang.String,java.lang.String,java.lang.String), with URI template, "/", is treated as a resource method
Jul 28, 2015 8:35:23 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/tomcat7/openam/webapps/manager
I've deployed two instances of the same Grails app in Tomcat7. I've used the following to create wars, using environments in config files:
grails -Dgrails.env=instance1 war
grails -Dgrails.env=instance2 war
I use setenv.sh for extra configurations and settings for JAVA_OPTS.
First instance runs correctly. The second gives me the following error:
Error 500: Internal Server Error
URI
/instance2/
Class
java.lang.IllegalStateException
Message
No thread-bound request found: Are you referring to request attributes outside of an actual web request, or processing a request outside of the originally receiving thread? If you are actually operating within a web request and still receive this message, your code is probably running outside of DispatcherServlet/DispatcherPortlet: In this case, use RequestContextListener or RequestContextFilter to expose the current request.
Trace
Line | Method
->> 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 615 | run in java.util.concurrent.ThreadPoolExecutor$Worker
^ 722 | run . . . in java.lang.Thread
Any suggestion?
EDIT:
I've tried to deploy a third instance with a different name. It works correctly. So I've undeployed and redeployed it into Tomcat, but it does not work...
I'm going crazy, because it seems that the app name has an influence...
I've deployed the application using the following:
grails -Dgrails.env=nameofapp war
where nameofapp is the name of the application
I'm using Grails 2.2.1 version
EDIT 2:
I've understood that the problem occurs when an app is deployed for the first time. Infact, when first deploy into Tomcat occurs, I see the following in catalina.out:
java.lang.NullPointerException
at medicalofficemanager.SecUserSecRole.create(SecUserSecRole.groovy:32)
at BootStrap$_closure1.doCall(BootStrap.groovy:61)
at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:308)
at grails.util.Environment.executeForEnvironment(Environment.java:301)
at grails.util.Environment.executeForCurrentEnvironment(Environment.java:277)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Jan 18, 2015 5:49:49 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Error listenerStart
Jan 18, 2015 5:49:49 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/MyAppName] startup failed due to previous errors
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/MyAppName] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [net.sf.ehcache.CacheManager#d5381] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [myapp.SecRole.data] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [org.hibernate.cache.UpdateTimestampsCache.data] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [org.hibernate.cache.StandardQueryCache.data] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [net.sf.ehcache.CacheManager#13e45] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [aclCache.data] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:51:13 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load net.sf.ehcache.store.compound.CompoundStore$KeySet. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1587)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1546)
at net.sf.ehcache.store.compound.CompoundStore.keySet(CompoundStore.java:216)
at net.sf.ehcache.store.compound.factories.DiskStorageFactory$DiskExpiryTask.run(DiskStorageFactory.java:670)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
I think that first exception (about SecUserSecRole) it is not relevant...
If I undeploy and redeploy the same war, without cleaning the database, the application starts correctly.
What about this stacktrace?
There is a problem in this line: grails -Dgrails.env=instance1 war
-Dgrails.env should be set to the environment name eg: dev, test, prod or myCustomEnv, not the name of the application.
This setting tells Grails to look up the settings in the environment section of DataSource.groovy and Config.groovy under the name supplied and apply them. If they do not exist who knows what you will end up with.
As you can see in your stacktrace you are getting a null pointer exception when Grails is trying to execute the environment block. This is because the value you have given it does not exist.
Your problem appears to be in SecUserSecRole.groovy:32. I'd be willing to bet that is the spot where your app first tries to do DB access. Can you post the relevant section of SecUserSecRole.groovy as well as details on your DB configuration?
I have a grails (v2.1.5) app in production, it works fine and i can redeploy without problems, but i have updated the app (a few changes), when I deploy the new war, i get a memory leak error, the server starts but the app doesn´t work (the page in the browser is only a withe page), I tried to reboot the instance, stop all services,....
I can redeploy only the old war. How could i fix this problem?
Thanks.
The errors:
Mar 09, 2014 4:02:14 PM org.apache.catalina.core.StandardContext startInternal SEVERE: Error listenerStart
Mar 09, 2014 4:07:17 PM org.apache.catalina.core.StandardContext startInternal SEVERE: Context [] startup failed due to previous errors
Mar 09, 2014 4:07:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 09, 2014 4:07:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
Mar 09, 2014 4:07:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [net.sf.ehcache.CacheManager#61783b69] but has failed to stop it. This is very likely to create a memory leak.
Mar 09, 2014 4:07:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [com.padelsix.SecRole.data] but has failed to stop it. This is very likely to create a memory leak.
The web application [/services] appears to have started a thread named [net.sf.ehcache.CacheManager#37ce6a10] but has failed to stop it. This is very likely to create a memory leak.
The solution, in our groovy based spring web app that uses annotation based configuration, was to register the net.sf.ehcache.constructs.web.ShutdownListener class with the context in our class that extends WebApplicationInitializer via an overridden onStartup() method, something like this:
import net.sf.ehcache.constructs.web.ShutdownListener
import org.springframework.web.WebApplicationInitializer
...
public class CustomWebApplicationInitializer implements WebApplicationInitializer {
...
#Override
public void onStartup(ServletContext servletContext) throws ServletException {
...
createContext(...)
...
}
private AnnotationConfigWebApplicationContext createContext(final Class... annotatedClasses) {
AnnotationConfigWebApplicationContext context = new AnnotationConfigWebApplicationContext()
context.register(annotatedClasses)
context.register(ShutdownListener)
return context
}
}
The main problem is not the potential memory leaks, it is the first part of the log:
Mar 09, 2014 4:02:14 PM org.apache.catalina.core.StandardContext
startInternal SEVERE: Error listenerStart
Mar 09, 2014 4:07:17 PM org.apache.catalina.core.StandardContext
startInternal SEVERE: Context [] startup failed due to previous errors
The error "SEVERE: Error listenerStart" usually means that there was an error in a servlet context listener (run on startup) that prevented the webapp to start.
Look for errors both in catalina.out and localhost.<date>.log. You may need to enable additional logging to find the root cause of this error. Google for "SEVERE: Error listenerStart" and look for specific instructions for your version of Tomcat.
I'm pretty used to crashing my tomcat server. But I don't think I made any changes at all and seem to have broken it completely. Was working one min then not the next.
I really hope someone can help here is the Catalina.out:
Dec 20, 2012 1:35:56 AM org.apache.catalina.startup.Embedded initDirs
SEVERE: Cannot find specified temporary folder at
Dec 20, 2012 1:35:57 AM org.apache.catalina.core.AprLifecycleListener init
INFO: Loaded APR based Apache Tomcat Native library 1.1.23.
Dec 20, 2012 1:35:57 AM org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
Dec 20, 2012 1:35:57 AM org.apache.coyote.http11.Http11AprProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-8040
Dec 20, 2012 1:35:57 AM org.apache.coyote.ajp.AjpAprProtocol init
INFO: Initializing Coyote AJP/1.3 on ajp-8042
Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 587 ms
Dec 20, 2012 1:35:57 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Dec 20, 2012 1:35:57 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.33
Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory docs
Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory myApp
Dec 20, 2012 1:35:57 AM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/lhome/me/tomcat6/webapps/myApp/WEB-INF/lib/javax.servlet.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Dec 20, 2012 1:35:58 AM org.apache.coyote.http11.Http11AprProtocol start
INFO: Starting Coyote HTTP/1.1 on http-8040
Dec 20, 2012 1:35:58 AM org.apache.coyote.ajp.AjpAprProtocol start
INFO: Starting Coyote AJP/1.3 on ajp-8042
Dec 20, 2012 1:35:58 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 880 ms
Error Reading TLE line 1: java.lang.Exception: TLE line 1 not valid first line
Error Creating SGP4 Satellite
Dec 20, 2012 1:36:26 AM org.apache.coyote.http11.Http11AprProtocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8040
Dec 20, 2012 1:36:26 AM org.apache.coyote.ajp.AjpAprProtocol pause
INFO: Pausing Coyote AJP/1.3 on ajp-8042
Dec 20, 2012 1:36:27 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Dec 20, 2012 1:36:30 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/myApp] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Dec 20, 2012 1:36:30 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/myApp] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
Dec 20, 2012 1:36:30 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/myApp] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
Dec 20, 2012 1:36:30 AM org.apache.coyote.http11.Http11AprProtocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8040
Dec 20, 2012 1:36:30 AM org.apache.coyote.ajp.AjpAprProtocol destroy
INFO: Stopping Coyote AJP/1.3 on ajp-8042
Dec 20, 2012 1:37:12 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:338)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:416)
And I have a new log that has been created, I've never seen it before but it seems to think its something severe. A long one but here it is:
Massive Error Log I've never seen before
Your JVM's got SIGBUS. This is either a hardware problem, or a low-level JVM code bug.
siginfo:si_signo=SIGBUS: si_errno=0,
si_code=2 (BUS_ADRERR), si_addr=0x00007f0a9bb06fe5
One suspicious entry is -Djava.io.tmpdir= in your startup (i.e. temp dir is empty string). Could do funny things to JVM, though I did not test it. May be it is OK to have it like that.
jvm_args: -Dcatalina.base=/lhome/me/tomcat6
-Dcatalina.home=/usr/share/tomcat6
-Djava.endorsed.dirs=
-Djava.io.tmpdir=
-Djava.util.logging.config.file=/lhome/me/tomcat6/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
Bottom line: I believe your hardware goes down. Backup everything.
The bomb seems to have occurred in the native method java.util.zip.ZipFile.getZipMessage() which is called by getEntry(). ZipFile had experienced an error and was trying to construct an exception message. This occurs while stopping the running webapp as part of reload.
Did you move the .war file? That can cause the problem.
These kinds of problems are hard to diagnose and fix. However, the problem is very likely to be java.io.tmpdir not being set properly. http://efod.se/blog/archive/2011/05/02/java-sigbus is a similar issue.
If that doesn't work, use apt-get to upgrade zlib and OpenJDK, and force ZipFile.getEntry() to be interpreted:
-X:CompileCommand=exclude,java/util/zip/ZipFile,getEntry
As OSB Stats said, don't trust your hardware (although it's not likely a hardware problem).
Thanks to all who gave help here.
However I fell I should answer my own question. Although I still don't fully understand why it caused such a severe exception. The problem seemed to lie with me having dropped a table on my database that the program was reading. This table was full of data that the program used to compile other data. Once I re-populated this table it was fine.
I don't know why something so simple caused such a lot of errors and never showed up in the logs as a table not found error. However I can only assume that the program was trying to run with blank data and this someone caused a severe error.
So thanks again to those who offered help!