I have a grails (v2.1.5) app in production, it works fine and i can redeploy without problems, but i have updated the app (a few changes), when I deploy the new war, i get a memory leak error, the server starts but the app doesn´t work (the page in the browser is only a withe page), I tried to reboot the instance, stop all services,....
I can redeploy only the old war. How could i fix this problem?
Thanks.
The errors:
Mar 09, 2014 4:02:14 PM org.apache.catalina.core.StandardContext startInternal SEVERE: Error listenerStart
Mar 09, 2014 4:07:17 PM org.apache.catalina.core.StandardContext startInternal SEVERE: Context [] startup failed due to previous errors
Mar 09, 2014 4:07:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 09, 2014 4:07:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
Mar 09, 2014 4:07:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [net.sf.ehcache.CacheManager#61783b69] but has failed to stop it. This is very likely to create a memory leak.
Mar 09, 2014 4:07:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [com.padelsix.SecRole.data] but has failed to stop it. This is very likely to create a memory leak.
The web application [/services] appears to have started a thread named [net.sf.ehcache.CacheManager#37ce6a10] but has failed to stop it. This is very likely to create a memory leak.
The solution, in our groovy based spring web app that uses annotation based configuration, was to register the net.sf.ehcache.constructs.web.ShutdownListener class with the context in our class that extends WebApplicationInitializer via an overridden onStartup() method, something like this:
import net.sf.ehcache.constructs.web.ShutdownListener
import org.springframework.web.WebApplicationInitializer
...
public class CustomWebApplicationInitializer implements WebApplicationInitializer {
...
#Override
public void onStartup(ServletContext servletContext) throws ServletException {
...
createContext(...)
...
}
private AnnotationConfigWebApplicationContext createContext(final Class... annotatedClasses) {
AnnotationConfigWebApplicationContext context = new AnnotationConfigWebApplicationContext()
context.register(annotatedClasses)
context.register(ShutdownListener)
return context
}
}
The main problem is not the potential memory leaks, it is the first part of the log:
Mar 09, 2014 4:02:14 PM org.apache.catalina.core.StandardContext
startInternal SEVERE: Error listenerStart
Mar 09, 2014 4:07:17 PM org.apache.catalina.core.StandardContext
startInternal SEVERE: Context [] startup failed due to previous errors
The error "SEVERE: Error listenerStart" usually means that there was an error in a servlet context listener (run on startup) that prevented the webapp to start.
Look for errors both in catalina.out and localhost.<date>.log. You may need to enable additional logging to find the root cause of this error. Google for "SEVERE: Error listenerStart" and look for specific instructions for your version of Tomcat.
Related
I have a jar that deploys and runs in a standalone tomcat 8.0.28 server with Java 8.0.66
I wish to run it from an embedded Tomcat to run integration tests.
The war deploys but fails to start, I ran out of ideas, I am getting the following error:
Dec 07, 2015 6:11:46 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-9191"]
Dec 07, 2015 6:11:46 PM org.apache.tomcat.util.net.NioSelectorPool
getSharedSelector
INFO: Using a shared selector for servlet write/read
Dec 07, 2015 6:11:46 PM org.apache.catalina.core.StandardService
startInternal
INFO: Starting service Tomcat
Dec 07, 2015 6:11:46 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/8.0.28
Dec 07, 2015 5:43:47 PM org.apache.catalina.startup.ContextConfig
getDefaultWebXmlFragment
INFO: No global web.xml found
Dec 07, 2015 5:43:47 PM org.apache.catalina.core.ContainerBase startInternal
SEVERE: A child container failed during start
java.util.concurrent.ExecutionException:
org.apache.catalina.LifecycleException: Failed to start component
[StandardEngine[Tomcat].StandardHost[localhost].
StandardContext[/StrateboBackEnd]]
...
Caused by: org.apache.catalina.LifecycleException:
Failed to start component StandardEngine[Tomcat].StandardHost[localhost].
StandardContext[/myApp]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
... 6 more
Caused by: java.lang.NoSuchMethodError:
javax.servlet.ServletContext.getClassLoader()Ljava/lang/ClassLoader;
I have the following setup for the embed tomcat
final Tomcat tomcat = new Tomcat();
tomcat.setPort(9191);
File baseDir = new File(".");
Context context = tomcat.addContext("", baseDir.getAbsolutePath());
tomcat.addWebapp("/Myapp", "C://Users//Add//git//Myapp//build//libs//Myapp-1.1.war");
tomcat.enableNaming();
ContextResource res = new ContextResource();
res.setName("jdbc/myDatabase");
res.setType("javax.sql.DataSource");
res.setAuth("Container");
res.setProperty("username", "username");
res.setProperty("password", "password");
res.setProperty("driverClassName","net.sourceforge.jtds.jdbc.Driver");
res.setProperty("url", "jdbc:jtds:sqlserver://127.0.0.1:1433//myDatabase");
res.setProperty("maxTotal", "10");
res.setProperty("maxIdle", "10");
res.setProperty("maxWaitMillis", "10000");
res.setProperty("removeAbandonedTimeout","300");
res.setProperty("defaultAutoCommit","true");
context.getNamingResources().addResource(res);
tomcat.start();
tomcat.getServer().await();
I have a context.xml that I use for normal deployment but this does not get read in embedded from what I can tell.
Since the jar works on a standalone I know that the jar is not the problem
There is this Info line
INFO: No global web.xml found
Which might point to an issue but could not find a reference
Just in case here is the relevant part of my gradle file for dependencies
compile 'org.apache.tomcat:tomcat-util:8.0.28'
compile 'org.apache.tomcat.embed:tomcat-embed-core:8.0.28'
compile 'org.apache.tomcat.embed:tomcat-embed-jasper:8.0.28'
compile 'org.apache.tomcat.embed:tomcat-embed-logging-juli:8.0.28'
compile 'org.apache.tomcat:tomcat-dbcp:8.0.28'
Any suggestions greatly appreciated?
For an extra point is there any way to get tomcat to give better feedback as to what it is not liking?
After some research it looks like most of the questions on StackOverflow regarding embedded Tomcat 8 remain unanswered there does not seem to be interest or development in this area. As this was for integration testing I setup a tomcat server on my Jenkins machine and Jenkins pushes the war unto it, restarts it then runs the integration tests. This work even though it may not be the most elegant.
I developed a webapp with Vaadin and kept it running on an embedded Jetty Server. Now, in order to deploy it on the productive server, I need it to run on a Tomcat v7 server.
I took the WAR-file (built with Maven) and deployed it to Tomcat.
Now, when I call the Tomcat URL in the browser, the webapp starts and shows the Vaadin loading circle. After a few seconds, the circle disappears and Tomcat stops.
The Catalina Logfile says the following:
Nov 17, 2015 11:37:55 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 5830 ms
Nov 17, 2015 11:37:59 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8080"]
Nov 17, 2015 11:37:59 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
Nov 17, 2015 11:37:59 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Nov 17, 2015 11:38:01 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SCHWERWIEGEND: The web application [/myWebapp] is still processing a request that has yet to finish. This is very likely to create memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
Nov 17, 2015 11:38:01 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SCHWERWIEGEND: The web application [/myWebapp] created a ThreadLocal with key of type [com.vaadin.util.CurrentInstance$1] (value [com.vaadin.util.CurrentInstance$1#d8b2a4cd]) and a value of type [java.util.HashMap] (value [{interface com.vaadin.server.VaadinRequest=com.vaadin.util.CurrentInstance#d59e9a8b, class com.vaadin.server.VaadinSession=com.vaadin.util.CurrentInstance#bb1b7c4b, class com.vaadin.server.VaadinService=com.vaadin.util.CurrentInstance#5b5dec40, interface com.vaadin.server.VaadinResponse=com.vaadin.util.CurrentInstance#ea949e2b, class com.vaadin.ui.UI=com.vaadin.util.CurrentInstance#174bb6a7}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Nov 17, 2015 11:38:01 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8080"]
Which fault is likely to cause this behaviour? Do I need to perform certain steps in order to deploy a Jetty webapp on Tomcat?
I recently started working in a project which uses OpenAM for single sign on top of tomcat 7 and server running on Apache 2.4.12.
Installed Java version details:
java version "1.7.0_65"
OpenJDK Runtime Environment (rhel-2.5.1.2.el6_5-x86_64 u65-b17)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
OpenAM 11.0.0 (2013-November-08 10:40)
Most of the time 95% of CPU utilization is by java. I can't find the reason for this high CPU utilization.
We had an issue today with OpenAM, OpenAM Tomcat processes seems to have hung in two servers out of 8 and not listening to the port 8180. After restart, the service is up and running fine with ports listening.
However, I could see below error in openAM logs, I tried to find the root cause but unfortunately I can't, could anyone please help?
Server 1:
Jul 28, 2015 8:33:36 AM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextDestroyed
INFO: WSSERVLET13: JAX-WS context listener destroyed
Jul 28, 2015 8:33:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/sso] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak.
Jul 28, 2015 8:33:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/sso] appears to have started a thread named [Monitor Provider State Updater] but has failed to stop it. This is very likely to create a memory leak.
Jul 28, 2015 8:33:43 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/sso] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#4ee2edc3]) and a value of type [org.glassfish.grizzly.ThreadCache.ObjectCache] (value [org.glassfish.grizzly.ThreadCache$ObjectCache#1467c5ab]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8180"]
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol destroy
Server 2:
Jul 28, 2015 8:34:50 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/sso] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak.
Jul 28, 2015 8:34:50 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/sso] appears to have started a thread named [Monitor Provider State Updater] but has failed to stop it. This is very likely to create a memory leak.
Jul 28, 2015 8:33:43 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/sso] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#4ee2edc3]) and a value of type [org.glassfish.grizzly.ThreadCache.ObjectCache] (value [org.glassfish.grizzly.ThreadCache$ObjectCache#1467c5ab]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8180"]
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8180"]
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8109"]
Jul 28, 2015 8:33:43 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-8109"]
Jul 28, 2015 8:33:50 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jul 28, 2015 8:33:50 AM org.apache.coyote.AbstractProtocol init
Other WARNING in Server 2:
Jul 28, 2015 8:35:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl newResourceClass
WARNING: A sub-resource method, public javax.ws.rs.core.Response org.forgerock.openam.forgerockrest.authn.AuthenticationRestService.getMethodNotSupported(), with URI template, "/", is treated as a resource method
Jul 28, 2015 8:35:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl newResourceClass
WARNING: A sub-resource method, public javax.ws.rs.core.Response org.forgerock.openam.forgerockrest.authn.AuthenticationRestService.authenticate(javax.ws.rs.core.HttpHeaders,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.String,java.lang.String,java.lang.String,java.lang.String), with URI template, "/", is treated as a resource method
Jul 28, 2015 8:35:23 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/tomcat7/openam/webapps/manager
I've deployed two instances of the same Grails app in Tomcat7. I've used the following to create wars, using environments in config files:
grails -Dgrails.env=instance1 war
grails -Dgrails.env=instance2 war
I use setenv.sh for extra configurations and settings for JAVA_OPTS.
First instance runs correctly. The second gives me the following error:
Error 500: Internal Server Error
URI
/instance2/
Class
java.lang.IllegalStateException
Message
No thread-bound request found: Are you referring to request attributes outside of an actual web request, or processing a request outside of the originally receiving thread? If you are actually operating within a web request and still receive this message, your code is probably running outside of DispatcherServlet/DispatcherPortlet: In this case, use RequestContextListener or RequestContextFilter to expose the current request.
Trace
Line | Method
->> 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 615 | run in java.util.concurrent.ThreadPoolExecutor$Worker
^ 722 | run . . . in java.lang.Thread
Any suggestion?
EDIT:
I've tried to deploy a third instance with a different name. It works correctly. So I've undeployed and redeployed it into Tomcat, but it does not work...
I'm going crazy, because it seems that the app name has an influence...
I've deployed the application using the following:
grails -Dgrails.env=nameofapp war
where nameofapp is the name of the application
I'm using Grails 2.2.1 version
EDIT 2:
I've understood that the problem occurs when an app is deployed for the first time. Infact, when first deploy into Tomcat occurs, I see the following in catalina.out:
java.lang.NullPointerException
at medicalofficemanager.SecUserSecRole.create(SecUserSecRole.groovy:32)
at BootStrap$_closure1.doCall(BootStrap.groovy:61)
at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:308)
at grails.util.Environment.executeForEnvironment(Environment.java:301)
at grails.util.Environment.executeForCurrentEnvironment(Environment.java:277)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Jan 18, 2015 5:49:49 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Error listenerStart
Jan 18, 2015 5:49:49 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/MyAppName] startup failed due to previous errors
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/MyAppName] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [net.sf.ehcache.CacheManager#d5381] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [myapp.SecRole.data] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [org.hibernate.cache.UpdateTimestampsCache.data] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [org.hibernate.cache.StandardQueryCache.data] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [net.sf.ehcache.CacheManager#13e45] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:49:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyAppName] appears to have started a thread named [aclCache.data] but has failed to stop it. This is very likely to create a memory leak.
Jan 18, 2015 5:51:13 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load net.sf.ehcache.store.compound.CompoundStore$KeySet. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1587)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1546)
at net.sf.ehcache.store.compound.CompoundStore.keySet(CompoundStore.java:216)
at net.sf.ehcache.store.compound.factories.DiskStorageFactory$DiskExpiryTask.run(DiskStorageFactory.java:670)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
I think that first exception (about SecUserSecRole) it is not relevant...
If I undeploy and redeploy the same war, without cleaning the database, the application starts correctly.
What about this stacktrace?
There is a problem in this line: grails -Dgrails.env=instance1 war
-Dgrails.env should be set to the environment name eg: dev, test, prod or myCustomEnv, not the name of the application.
This setting tells Grails to look up the settings in the environment section of DataSource.groovy and Config.groovy under the name supplied and apply them. If they do not exist who knows what you will end up with.
As you can see in your stacktrace you are getting a null pointer exception when Grails is trying to execute the environment block. This is because the value you have given it does not exist.
Your problem appears to be in SecUserSecRole.groovy:32. I'd be willing to bet that is the spot where your app first tries to do DB access. Can you post the relevant section of SecUserSecRole.groovy as well as details on your DB configuration?
I'm using Tomcat 7 to deploy my app.
Tomcat goes down randomly and requires me to restart the server time and again to keep it up and running.
Sometimes when i restart ,i check the logs to see waiting for instances to be deallocated.
Other times when i restart ,i get the ERROR 1053.
I tried deleting my temp/work folders.I checked my database connections through hibernate ,i made sure that i have all the required jars in lib under tomcat.
The way i deploy is copy my war file and its respective Applicationfolder into Webapps under Tomcat.Please direct me as to what needs to be done to keep the server stable.
Another important note is that the exception in the logs below,changes from Nullpointer to Illegal state Exception
Any help to identify the root cause will be truly appreciated as this has been an impending and critical issue that has been haunting,inspite of trying different solutions.
My Logs are as follows
INFO: Waiting for 3 instance(s) to be deallocated
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/cep4] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/cep4] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/cep4] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/cep4] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Locale context]) and a value of type [org.springframework.web.servlet.DispatcherServlet$1] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/cep4] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#1655d52d]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [org.springframework.security.core.context.SecurityContextImpl#388100c: Authentication: org.springframework.security.cas.authentication.CasAuthenticationToken#388100c: Principal: com.java.service.MyUserDetails#9154e84; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails#ffff4c9c: RemoteIpAddress: 128.32.111.117; SessionId: CF331A8C27CEA68FAC797ED7E0E654B5; Granted Authorities: com.java.service.MyUserDetails$1#1aca2c25 Assertion: org.jasig.cas.client.validation.AssertionImpl#1a6d280b Credentials (Service/Proxy Ticket): ST-89206-bOamLKnOoLTLDJoIxYqs-cas-p3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/cep4] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Request attributes]) and a value of type [org.springframework.web.context.request.ServletRequestAttributes] (value [SecurityContextHolderAwareRequestWrapper[ org.springframework.security.web.savedrequest.SavedRequestAwareWrapper#5d1e88cf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 27, 2014 2:17:27 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-80"]
Jan 27, 2014 2:17:27 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8443"]
Jan 27, 2014 2:17:27 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load org.hibernate.cfg.ImprovedNamingStrategy. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:258)
at org.springframework.beans.factory.support.AbstractBeanDefinition.resolveBeanClass(AbstractBeanDefinition.java:417)
at org.springframework.beans.factory.support.AbstractBeanFactory.doResolveBeanClass(AbstractBeanFactory.java:1283)
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1254)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:433)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:270)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:120)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Jan 27, 2014 2:17:29 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'maxSpareThreads' to '75' did not find a matching property.
Jan 27, 2014 2:17:29 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'source' to 'org.eclipse.jst.jee.server:cep4' did not find a matching property.
Jan 27, 2014 2:17:29 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-80"]
Jan 27, 2014 2:17:29 PM org.apache.coyote.AbstractProtocol init
Really my colleagues and I faced with a similar issue with nearly the same error logs when using Apache Tomcat 7 as application server. Finally we could solve the problem using helps of contents at Stack Overflow and Apache Tomcat websites. Here I would like to share our experiences in a step by step form of debugging:
Try to find out crashing time due to server time.
Goto log folder of Tomcat and check errors just before the crashing time to the end.
Try to find out the reason of each error and try to solve the problems which caused the errors.
Start to monitor Tomcat behaviour using available tools. We used jConsole, but do not forget to link jConsole to Tomcat.
Try to control important parameters such as changes in Heap Memory and Threads especially in peak time of using your system. Maybe you have to increase specified or default values of Heap Memory or Threads.
P.S.: Do not panic on PS Survivor space! That's full most of time. ;-)
Final Word
Finally in our case by monitoring Tomcat with jConsole, we found out the problem was
insufficient Heap Memory in peak time. The solution was increasing the amount of allocated heap memory of Tomcat.