Develop Reference

java.lang.IllegalStateException while reloading Tomcat

I'm working on Spring with Hibernate application, It's working fine but while starting or reloading tomcat server, i'm getting java.lang.IllegalStateException Exception.
Can anybody explain, why this exception occurring and how to resolve it??
INFO: Illegal access: this web application instance has been stopped already. Could not load java.net.BindException. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at com.mysql.jdbc.SQLError.createLinkFailureMessageBasedOnHeuristics(SQLError.java:1220)
at com.mysql.jdbc.exceptions.jdbc4.CommunicationsException.<init>(CommunicationsException.java:57)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3270)
at com.mysql.jdbc.MysqlIO.quit(MysqlIO.java:1659)
at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4296)
at com.mysql.jdbc.ConnectionImpl.cleanup(ConnectionImpl.java:1265)
at com.mysql.jdbc.ConnectionImpl.finalize(ConnectionImpl.java:2667)
at java.lang.System$2.invokeFinalize(Unknown Source)
at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
at java.lang.ref.Finalizer.access$100(Unknown Source)

I want to give you couple of choices. You can try it. Any option can fulfill your demand.
Restart your tomcat and apache server because a long time of using,
it keeps older version of your application.
Clean your tomcat temp directory and restart
As stated in error,
The eventual following stack trace is caused by an error thrown for
debugging purposes as well as to attempt to terminate the thread which
caused the illegal access, and has no functional impact.
It may be the real cause that at this moment you are in debugging mode and it doesn't clear the running thread. so remove your break point and run it not debugging
If your code contains any kind of thread which are not properly terminated, then this type of error may occur.
That you have init() method but no destroy() method, then this type of error may occur. For details, you can follow the link - http://www.javaspecialists.eu/archive/Issue056.html
if the webapp has stopped, or is stopping, that means that the .war file or WEB-INF/web.xml timestamp changed, and the webapp might be reloading. Please check timestamp is OK or not.
To turn it off, set reloadable="false" in the context definition of
your app. It might be tomcat's server.xml.
In details solution:
The tomcat's server.xml of the reloadable Context is set to false.
For example:
Context path="/expert" docBase="expert" debug="0" reloadable ="false"/>
The solution is easy, as long as the tomcat's server.xml in reloadable = "true" into false on the line, but to do so would lose the advantage of hot deployment, and for the development is not very convenient, simply change it or not. This error does not matter.
Error principle:
The reason is because the tomcat restart, because the previous tomcat thread has not completely shut down, restart tomcat will report this exception, but this does not affect the normal use, just jump abnormal annoying people. Used hibernate, spring or other large components, when a WEB application system has a lot of class,
if you turned the Tomcat reloadable = true, then whenever the relevant documents change, Tomcat stops web app and frees up memory, and then reload web app. it may be a huge project. So we always think if there is only a certain class of overloaded functions, will greatly meet our debugger.
UPDATE: For code related issue
First, I want to tell you that I have given you some solutions for tomcat basis. Now I want to give you a solution for code basis. Would you please cross check your code with this issue? Please follow the URL.
http://www.coderanch.com/t/660126/Wiki/Illegal-State-Exception
UPDATE: For MySQL related issue
There are 2 issues.
This web application instance has been stopped already. Could not load java.net.BindException.
This web application instance has been
stopped already.Could not load com.mysql.jdbc.
This is because the MySQL JDBC driver on the application under the WEB-INF/lib directory, in the re-release of its loaded twice, so long as it can be copied to %TOMCAT_HOME%/lib can solve the problem.
We can solve these two anomalies MySQL drivers from the WEB-INF/lib folder moved to %TOMCAT_HOME%/lib.
I suspect this might be happening after the web application is restarted, where it's down for a short period of time. Then some finalize() method in the code is probably trying to do some cleanup too late. Whether or not that's in your code or the MySQL driver I can't say. You definitely should only have one version of a jar in a directory at a time. You might want to upgrade it to the latest (5.1.38 right now) in case something has been fixed that might be affecting you.(Number 9 is copied from #WhiteFang34)
Related link for 9: tomcat 6.0.24 Exception: Could not load com.mysql.jdbc.SQLError

The issue was because of the server cache. Follow the below steps to fix this issue
Find the directory called "work" under your tomcat's root directory
Delete it completely
Restart the server
Issue will be fixed.
Note: In case you have default tomcat installation in linux machine, you have to delete the folder /var/cache/tomcat7/Catalina

I think that the root cause of this problem is that something is leaking JDBC Connection objects.
These objects are actually instances of com.mysql.jdbc.ConnectionImpl ... which is a class with a finalize(). Normally, object finalization tidies up the leaked connections. In this case, it looks like the sequence of events is as follows:
the GC runs after the webapp has shutdown
a ConnectionImpl object is found to be unreachable
the object's finalize() method attempts to shutdown the MySQL database connection ... which entails sending a message to the server
when the message send fails for some reason, and the comms-level code tries to create an exception by calling com.mysql.jdbc.SQLError.createCommunicationsException
that tries to create an exception reflectively ...
The last step is failing because it is trying to use the webapp's classloader for the webapp ... which has been shut down. Apparently, this is detected by a check that is intended to help programmers diagnose webapps that don't shutdown cleanly.
The best solution is to track down the source of the Connection leak(s) and fix it. If Connection objects did not leak, then they would be closed while the webapp's classloader was still active.
Alternatively, you could simply turn off the check, as described in #Skywalker's answer.

The problem could be a simple SQL Error. tThe
java.lang.IllegalStateException
means that an object or class was called on when it was not ready to be called on. These errors at the bottom of the stack
at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
at java.lang.ref.Finalizer.access$100(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
look like they come from the Java Garbage Collector
attempting to clean up MySQL Object Pointers
that were temporarily unreferenced during a reboot. It seems to me that your error here is just a few MySQL Object Pointers being halted in the middle of execution when the GC calls their finalize methods and then pauses the application to attempt a clean up the object. (leading to a fail and a throw of this exception)
This error is most likely harmless, and could maybe be fixed by adding a
JNI_DestroyJavaVM();
to your main thread to be executed on shutdown.

How to re-deploy a hibernate-c3p0 project on tomcat 7 without getting strange c3p0 errors

If the project is re-deployed through netbeans on tomcat 7 then I get errors like
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at com.mchange.v2.resourcepool.BasicResourcePool.checkIdleResources(BasicResourcePool.java:1481)
at com.mchange.v2.resourcepool.BasicResourcePool.access$2000(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$CheckIdleResourcesTask.run(BasicResourcePool.java:1964)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Exception in thread "Timer-5" java.lang.NoClassDefFoundError: com/mchange/v2/resourcepool/BasicResourcePool$AsyncTestIdleResourceTask
at com.mchange.v2.resourcepool.BasicResourcePool.checkIdleResources(BasicResourcePool.java:1481)
at com.mchange.v2.resourcepool.BasicResourcePool.access$2000(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$CheckIdleResourcesTask.run(BasicResourcePool.java:1964)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.lang.ClassNotFoundException: com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
... 5 more
Today we got another strange error when we tried to re-deploy the project on tomcat 7
[5:07:02 PM] Nitin - Webscraper/Tester,Java/PHP developer: java.lang.NoClassDefFoundError: com/mchange/v2/lang/VersionUtils
com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:104)
com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:62)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:531)
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:78)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1700)
org.hibernate.loader.Loader.doQuery(Loader.java:801)
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
org.hibernate.loader.Loader.doList(Loader.java:2542)
org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
org.hibernate.loader.Loader.list(Loader.java:2271)
We have been getting such strange errors since quite some time now. When we try to debug, we find the classes already existing.
What I can think of is dangling c3p0 connection-pooling threads, which are either not destroyed properly on re-deploying, or may be having some active-connection executing or something similar.
Are there any best practices on how to re-deploy such a project which uses hibernate & c3p0 ? Is there some code that I have to write on contextDestroyed for proper closing of c3p0 threads ?

I was also facing the same problem and I could see the below warning in my tomcat console
Jul 30, 2014 3:20:16 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
WARNING: The web application [/rmlcrm] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->1hge50p9311d8syo1hfjimz|19ddf1db]-HelperThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:635)
I was reading a lot to find out a solution to this and came across the post
Hibernate :OutOfMemoryError: PermGen space
One of the comment in the post by Nicholas Hemley suggested to add a custom ServletContextListener and explicitly close the C3P0 connections in contextDestroyed() method of the listener which will be executed when the application is un-deployed.
We did not use the code exactly as we did not want to hard-couple with C3P0. But we realized that we are not closing hibernate sessionFactory anywhere in our application.
We added the code to close the hibernate session factory in contextDestroyed() of the ServletContextListener.
Now we do not have the error and neither we get a warning in the tomcat console.
You may also want to read Hibernate : closing the session factory does not close the c3p0 connection pool

a few thoughts:
1) if you have set up your hibernate app's lifecycle to map to your web-app's lifecycle (definitely true if hibernate and c3p0 libs live in your web-app's lib directory, potentially true even if not), you absolutely do need to make sure that c3p0 pools are destroyed prior to application recycle, often meaning a contextDestroyed method. in hibernate-speak, it is the SessionFactory that wraps the Connection pool; make sure that your application's SessionFactory is close()ed when your app shuts down on hot redeploy. there should be a symmetry: either in contextInitialized or lazily upon a first request, your SessionFactory should be initialized. it should be destroyed on application shutdown.
2) the latest (still pre-release) version of c3p0 has some settings designed to reduce the likelihood of contamination between c3p0 Threads and objects derived from expired web-app ClassLoaders, especially if c3p0 is loaded by a non web-app specific ClassLoader (e.g. if c3p0 libs live in $CATALINA_HOME/lib rather than in a webapp lib directory). if you are willing to upgrade to a prerelease [latest now is c3p0-0.9.5-pre5], then try the following new config settings:
c3p0.privilegeSpawnedThreads=true
c3p0.contextClassLoaderSource=library
hope this helps!

INFO: Illegal access: this web application instance has been stopped already. Could not load java.net.InetAddress

I'm experiencing this kind of exception, can someone help me about this problem?
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at org.quartz.utils.UpdateChecker.getClientId(UpdateChecker.java:149)
at org.quartz.utils.UpdateChecker.buildParamsString(UpdateChecker.java:120)
at org.quartz.utils.UpdateChecker.buildUpdateCheckUrl(UpdateChecker.java:114)
at org.quartz.utils.UpdateChecker.doCheck(UpdateChecker.java:55)
at org.quartz.utils.UpdateChecker.checkForUpdate(UpdateChecker.java:47)
at org.quartz.utils.UpdateChecker.run(UpdateChecker.java:39)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
I'm also getting the same exception with Could not load java.net.URLEncoder and Could not load java.net.URLConnection. I'm using Eclipse Indigo SR1 and Tomcat V6.0

You can solve this problem after restarting the server, the server iscaching older version of the app.
see here

You cannot resolve this problem by "restarting the server".
The problem is that when you update your app by uploading a new version to Tomcat, some resource is not cleaned up properly. Could be anything from a runaway thread, to some kind of scheduled service, to a listener. When that process tries to load a class, Tomcat will detect and prevent it.
The solution to the problem is to properly keep track of, and on shutdown close, those threads.

I have also experienced this problem recently.
In my case the problem was generated by an unhandled exception in javax.servlet.ServletContextListener which was logged in one localhost-date.log.
Because of that exception the webapp was not deployed, but an already started java.util.concurrent.Executors was still running which caused another chain of exceptions in catalina-date.log:
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load WhatEver.class.
The solution for my case:
check all the Tomcat logs
fix exceptions in javax.servlet.ServletContextListener

I have faced the similar issue. I have solved by increasing the permsize of Server in the VM arguments -XX:MaxPermSize=1024m

java.lang.IllegalStateException. Possible error with WebClassLoader

I'm developing a web application with Spring and I have this strange error. I can't figure out where it comes from, I've been google'ing hard and found nothing so far.
In the project I'm using: MongoDB, Spring, WSS4J (X.509 security headers for soap), Apache CXF. I've been trying to discover whether there is something that causes this error, any particular action, behaviour, condition something, but I found nothing so far, sometimes it happens after 3 min of running sometimes after 3h or not at all, any clues and hints appreciated.
INFO: Illegal access: this web application instance has been stopped already. Could not load net.sf.ehcache.store.disk.Segment$1. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1597)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1556)
at net.sf.ehcache.store.disk.Segment$HashIterator.<init>(Segment.java:988)
at net.sf.ehcache.store.disk.Segment.hashIterator(Segment.java:936)
at net.sf.ehcache.store.disk.DiskStore$HashIterator.<init>(DiskStore.java:1038)
at net.sf.ehcache.store.disk.DiskStore$KeyIterator.<init>(DiskStore.java:1111)
at net.sf.ehcache.store.disk.DiskStore$KeyIterator.<init>(DiskStore.java:1111)
at net.sf.ehcache.store.disk.DiskStore$KeySet.iterator(DiskStore.java:949)
at net.sf.ehcache.store.disk.DiskStorageFactory$DiskExpiryTask.run(DiskStorageFactory.java:838)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)`

It means the ehcache thread is trying to interact with the application after the application has been stopped or redeployed. It happens only when ehcache reach limits you have in ehcache.xml and trying to overflow to disk.
Look at tomcat logs for warnings when stop your application:
SEVERE: The web application [/APP] appears to have started a thread
named [Ehcache_Worker-1] but has failed to stop it.
This is very likely to create a memory leak
If find something like this, you should stop ecache properly in your ServletContextListener:
public void contextDestroyed(ServletContextEvent servletContextEvent){
CacheManager.getInstance().shutdown();
}
maybe with 1 sec sleep after that, to be sure ehcache is stopped.
As mentioned in http://ehcache.org/documentation/faq
If the JVM keeps running after you stop using Ehcache, you should call
CacheManager.getInstance().shutdown() so that the threads are stopped
and cache memory is released back to the JVM. Calling shutdown also
insures that your persistent disk stores get written to disk in a
consistent state and will be usable the next time they are used. If
the CacheManager does not get shut down, it should not be a problem.
There is a shutdown hook which calls the shutdown on JVM exit.
Or you can set property overflowToDisk="false" in ehcache.xml or restart tomcat every time you deploy apllication.

web instance already stopped

i get this error
Mar 22, 2011 12:36:01 AM org.apache.catalina.loader.WebappClassLoader findResourceInternal
INFO: Illegal access: this web application instance has been stopped already. Could not load META-INF/services/javax.xml.parsers.DocumentBuilderFactory. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
Mar 22, 2011 12:36:01 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1562)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1521)
at javax.xml.parsers.FactoryFinder.getProviderClass(Unknown Source)
at javax.xml.parsers.FactoryFinder.newInstance(Unknown Source)
at javax.xml.parsers.FactoryFinder.find(Unknown Source)
at javax.xml.parsers.DocumentBuilderFactory.newInstance(Unknown Source)
at com.xmlparser.XmlParser.parseXmlFile(XmlParser.java:33)
at com.xmlparser.XmlParser.<init>(XmlParser.java:25)
at com.jobs.SendRoutineMessagesJob.execute(SendRoutineMessagesJob.java:29)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
could some one please guide me as to what mistake i am committing here. i am not sure if i would need to paste the source code as well.

This because of you are deploying new instance of your application to server but the some thread or connection of previous instance attempts doing some business logic you have written. That's why, the main part of JVM called ClassLoader informs you about "Illegal access: this web application instance has been stopped already" and throws appropriate exception ... You can switch off auto deploy feature of Application Server which preferable on Production Environment and after deploying application restart server manually. Or you can in Class level control Servlet Life Cycle :))) during "pre deploy" close all connection or stop some threads. Or use sophisticated way controlling Threads with ManagedExecutorService. Good luck!!!

seems like i was creating instance of something which was already created..i removed the instance and now it works fine

Are you using Netbeans? I had a similar error except it was could not load oracle.sql.lnxlib. Like many posters have pointed out, this error usually has to do with Tomcat caching multiple instances of the application. A coworker was able to solve the problem for me by undeploying old instances of the application within Netbeans. To do so, go to the Services Tab > Servers and then double click on Apache Tomcat or TomEE and after running the application once since opening netbeans, a Web Applications Folder should appear. If you double click on that, you should see instances of your application appear. You should be able to right click them and then click undeploy (you may have to right click them and hit stop first). Finally, you should stop and then start tomcat before building and running your application.
Hope this helps.
See attached screenshot for clarity

If you use gradle there is an error
Illegal access: this web application instance has been stopped already. Could not load [].
The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
You can change mysql-connect-java to
mysql: mysql-connector-java: 8.0.13
Good luck

I got this error on Tomcat on an Ubuntu server ....finally resolved it by doing a Tomcat shutdown with "systemctl stop tomcat" and then "sudo reboot" and then once the box came back up, starting Tomcat again