Develop Reference

How to fix java.lang.NoClassDefFoundError in tomcat

I'm getting below error message often and due to this error I'm suspecting my application creating outofMemory Error in permGen space.
Can someone help how to fix the below exception.
I'm using pure servlet and tomcat 7.
INFO: Illegal access: this web application instance has been stopped already. Could not load com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1745)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1703)
at com.mchange.v2.resourcepool.BasicResourcePool.checkIdleResources(BasicResourcePool.java:1481)
at com.mchange.v2.resourcepool.BasicResourcePool.access$2000(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$CheckIdleResourcesTask.run(BasicResourcePool.java:1964)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Exception in thread "Timer-0" java.lang.NoClassDefFoundError: com/mchange/v2/resourcepool/BasicResourcePool$AsyncTestIdleResourceTask
at com.mchange.v2.resourcepool.BasicResourcePool.checkIdleResources(BasicResourcePool.java:1481)
at com.mchange.v2.resourcepool.BasicResourcePool.access$2000(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$CheckIdleResourcesTask.run(BasicResourcePool.java:1964)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.ClassNotFoundException: com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1854)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1703)
... 5 more

That message just occurs because the web application is still trying to load things after you have shut it down.
As such, it should not have "business impact" (you don't care about errors in an app that you shut down), especially if it happens in the process of shutting down the whole app server.
You may want to figure out why it does not shutdown more cleanly (maybe some background processing that needs to be stopped when the webapp is asked to terminate). If there is stuff still going on in there, that might cause problems for any other web apps still running.
If you did not intend to shut down the webapp, then there is probably another error earlier in the log.

Managed VM Issue? Frequently found error log about VmApiProxyDelegate (using Datastore API, TaskQueue API) on AppEngine Managed VM instance

I'm using the Google AppEngine Managed VM/Java since March 2015. Everything is work well. But after September, October 2015, I noticed I 've seen the error log of "com.google.apphosting.vmruntime.VmApiProxyDelegate" in Managed VM instance log frequently.
2 groups of error log I noticed.
The first one is related to datastore operation on Managed VM instance. It happens on
datastore_v3.Get().
datastore_v3.RunQuery()
datastore_v3.Put()
memcache.Get().
Sample of stacktrace I saw in the log is below ... (sample below is for datastore_v3.Put())
com.google.apphosting.vmruntime.VmApiProxyDelegate runSyncCall: HTTP
ApiProxy I/O error for datastore_v3.Put: Read timed out
com.google.apphosting.api.ApiProxy$RPCFailedException: The remote RPC
to the application server failed for the call datastore_v3.Get(). at
com.google.apphosting.vmruntime.VmApiProxyDelegate.runSyncCall(VmApiProxyDelegate.java:182)
at
com.google.apphosting.vmruntime.VmApiProxyDelegate.makeApiCall(VmApiProxyDelegate.java:141)
at
com.google.apphosting.vmruntime.VmApiProxyDelegate.access$000(VmApiProxyDelegate.java:47)
at
com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:375)
at
com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:351)
at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
So, when this error occurred, at application level (my code), I got RPCFailedException but right now, I've not handled it with retry mechanism ( I only use retry mechanism for ConcurrentModificationException with App Engine Datastore API)
The second group of error log is about TaskQueue API on the Managed VM.
The error message I got is ...
com.google.apphosting.vmruntime.VmApiProxyDelegate runSyncCall: Error
body: RPC Error: /StubbyService.Send to (unknown) : APP_ERROR(2)
When tracing , the detailed stacktrace is ...
com.wat_suttiwat.batchengine.job.PushNotificationTaskExecutor
executeTask: The remote RPC to the application server failed for the
call taskqueue.QueryAndOwnTasks().
com.google.apphosting.api.ApiProxy$RPCFailedException: The remote RPC
to the application server failed for the call
taskqueue.QueryAndOwnTasks(). at
com.google.apphosting.vmruntime.VmApiProxyDelegate.runSyncCall(VmApiProxyDelegate.java:161)
at
com.google.apphosting.vmruntime.VmApiProxyDelegate.makeApiCall(VmApiProxyDelegate.java:141)
at
com.google.apphosting.vmruntime.VmApiProxyDelegate.access$000(VmApiProxyDelegate.java:47)
at
com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:375)
at
com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:351)
at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"RPCFailedException" exception raised at the application level (like the first case). And I noticed the Google AppEngine front-end instance (not the managed-vm instance), it work as usual no error about these.
So my question is
Should I add retry-mechanism for RPCFailedException ? Is it useful to add retry? I don't see any documentation from the Google AppEngine document on this.
Does anyone has the same issues with me? If yes, please help to vote the issue at this issue-tracker (#12393): https://code.google.com/p/googleappengine/issues/detail?can=2&start=0&num=100&q=&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log&groupby=&sort=&id=12393
If you have any workaround, please share.
Thanks so much

Exception on Spring integration, when shut down the tomcat server

I have a Spring Integration application which runs on Tomcat 6. It has a message-driven-channel-adapter and it communicates with remote Active MQ server. So far so good I am able to do my work.
However when I shut down Tomcat I am getting the following exception endlessly looping in the catalina log:
Exception in thread "org.springframework.jms.listener.DefaultMessageListenerContainer#0-178" Exception in thread "org.springframework.jms.listener.DefaultMessageListenerContainer#0-179"
java.lang.NullPointerException
at org.apache.log4j.LogManager.getLogger(LogManager.java:188)
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:289)
at org.apache.commons.logging.impl.Log4JLogger.trace(Log4JLogger.java:166)
at org.springframework.jms.support.JmsUtils.closeMessageConsumer(JmsUtils.java:158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1111)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:971)
at java.lang.Thread.run(Thread.java:722)
java.lang.NullPointerException
at org.apache.log4j.LogManager.getLogger(LogManager.java:188)
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:289)
at org.apache.commons.logging.impl.Log4JLogger.trace(Log4JLogger.java:166)
at org.springframework.jms.support.JmsUtils.closeSession(JmsUtils.java:115)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1112)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:971)
at java.lang.Thread.run(Thread.java:722)
Nov 13, 2012 7:23:37 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load org.apache.log4j.spi.VectorWriter. The eventual following stack trace is ca
used by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1273)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)
at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:154)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:234)
at org.springframework.jms.listener.DefaultMessageListenerContainer.handleListenerSetupFailure(DefaultMessageListenerContainer.java:831)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:981)
at java.lang.Thread.run(Thread.java:722)
Do you guys have any idea bout this? do i need to close or destroy Spring context before I shut down Tomcat?
Appreciate your help
Thanks,
Keth

Well, looks like you are correct:
do i need to close or destroy Spring context before I shut down Tomcat?
From Spring JSM source code it looks like your Tomcat closes its log4j.LogManager earlier, than Spring stops its context.

java.lang.IllegalStateException. Possible error with WebClassLoader

I'm developing a web application with Spring and I have this strange error. I can't figure out where it comes from, I've been google'ing hard and found nothing so far.
In the project I'm using: MongoDB, Spring, WSS4J (X.509 security headers for soap), Apache CXF. I've been trying to discover whether there is something that causes this error, any particular action, behaviour, condition something, but I found nothing so far, sometimes it happens after 3 min of running sometimes after 3h or not at all, any clues and hints appreciated.
INFO: Illegal access: this web application instance has been stopped already. Could not load net.sf.ehcache.store.disk.Segment$1. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1597)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1556)
at net.sf.ehcache.store.disk.Segment$HashIterator.<init>(Segment.java:988)
at net.sf.ehcache.store.disk.Segment.hashIterator(Segment.java:936)
at net.sf.ehcache.store.disk.DiskStore$HashIterator.<init>(DiskStore.java:1038)
at net.sf.ehcache.store.disk.DiskStore$KeyIterator.<init>(DiskStore.java:1111)
at net.sf.ehcache.store.disk.DiskStore$KeyIterator.<init>(DiskStore.java:1111)
at net.sf.ehcache.store.disk.DiskStore$KeySet.iterator(DiskStore.java:949)
at net.sf.ehcache.store.disk.DiskStorageFactory$DiskExpiryTask.run(DiskStorageFactory.java:838)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)`

It means the ehcache thread is trying to interact with the application after the application has been stopped or redeployed. It happens only when ehcache reach limits you have in ehcache.xml and trying to overflow to disk.
Look at tomcat logs for warnings when stop your application:
SEVERE: The web application [/APP] appears to have started a thread
named [Ehcache_Worker-1] but has failed to stop it.
This is very likely to create a memory leak
If find something like this, you should stop ecache properly in your ServletContextListener:
public void contextDestroyed(ServletContextEvent servletContextEvent){
CacheManager.getInstance().shutdown();
}
maybe with 1 sec sleep after that, to be sure ehcache is stopped.
As mentioned in http://ehcache.org/documentation/faq
If the JVM keeps running after you stop using Ehcache, you should call
CacheManager.getInstance().shutdown() so that the threads are stopped
and cache memory is released back to the JVM. Calling shutdown also
insures that your persistent disk stores get written to disk in a
consistent state and will be usable the next time they are used. If
the CacheManager does not get shut down, it should not be a problem.
There is a shutdown hook which calls the shutdown on JVM exit.
Or you can set property overflowToDisk="false" in ehcache.xml or restart tomcat every time you deploy apllication.

web instance already stopped

i get this error
Mar 22, 2011 12:36:01 AM org.apache.catalina.loader.WebappClassLoader findResourceInternal
INFO: Illegal access: this web application instance has been stopped already. Could not load META-INF/services/javax.xml.parsers.DocumentBuilderFactory. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
Mar 22, 2011 12:36:01 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1562)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1521)
at javax.xml.parsers.FactoryFinder.getProviderClass(Unknown Source)
at javax.xml.parsers.FactoryFinder.newInstance(Unknown Source)
at javax.xml.parsers.FactoryFinder.find(Unknown Source)
at javax.xml.parsers.DocumentBuilderFactory.newInstance(Unknown Source)
at com.xmlparser.XmlParser.parseXmlFile(XmlParser.java:33)
at com.xmlparser.XmlParser.<init>(XmlParser.java:25)
at com.jobs.SendRoutineMessagesJob.execute(SendRoutineMessagesJob.java:29)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
could some one please guide me as to what mistake i am committing here. i am not sure if i would need to paste the source code as well.

This because of you are deploying new instance of your application to server but the some thread or connection of previous instance attempts doing some business logic you have written. That's why, the main part of JVM called ClassLoader informs you about "Illegal access: this web application instance has been stopped already" and throws appropriate exception ... You can switch off auto deploy feature of Application Server which preferable on Production Environment and after deploying application restart server manually. Or you can in Class level control Servlet Life Cycle :))) during "pre deploy" close all connection or stop some threads. Or use sophisticated way controlling Threads with ManagedExecutorService. Good luck!!!

seems like i was creating instance of something which was already created..i removed the instance and now it works fine

Are you using Netbeans? I had a similar error except it was could not load oracle.sql.lnxlib. Like many posters have pointed out, this error usually has to do with Tomcat caching multiple instances of the application. A coworker was able to solve the problem for me by undeploying old instances of the application within Netbeans. To do so, go to the Services Tab > Servers and then double click on Apache Tomcat or TomEE and after running the application once since opening netbeans, a Web Applications Folder should appear. If you double click on that, you should see instances of your application appear. You should be able to right click them and then click undeploy (you may have to right click them and hit stop first). Finally, you should stop and then start tomcat before building and running your application.
Hope this helps.
See attached screenshot for clarity

If you use gradle there is an error
Illegal access: this web application instance has been stopped already. Could not load [].
The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
You can change mysql-connect-java to
mysql: mysql-connector-java: 8.0.13
Good luck

I got this error on Tomcat on an Ubuntu server ....finally resolved it by doing a Tomcat shutdown with "systemctl stop tomcat" and then "sudo reboot" and then once the box came back up, starting Tomcat again