Related
I'd like to create configuration/bean to automatically start H2DB in my development profile. I'd like to have it running as a tcp server. It's needed to be started before any DataSource configuration. Can someone tell me how to achieve this?
Wha have I done is
#Profile("h2")
#Component
public class H2DbServerConfiguration implements SmartLifecycle {
private static final Logger logger = LoggerFactory.getLogger(H2DbServerConfiguration.class);
private Server server;
#Override
public boolean isAutoStartup() {
return true;
}
#Override
public void stop(Runnable callback) {
stop();
new Thread(callback).start();
}
#Override
public void start() {
logger.debug("############################################");
logger.debug("############################################");
logger.debug("STARTING SERVER");
logger.debug("############################################");
logger.debug("############################################");
try {
server = Server.createTcpServer("-web", "-webAllowOthers", "-webPort", "8082").start();
} catch (SQLException e) {
throw new RuntimeException("Unable to start H2 server", e);
}
}
#Override
public void stop() {
logger.debug("############################################");
logger.debug("############################################");
logger.debug("STOPPING SERVER");
logger.debug("############################################");
logger.debug("############################################");
if (server != null)
if (server.isRunning(true))
server.stop();
}
#Override
public boolean isRunning() {
return server != null ? server.isRunning(true) : false;
}
#Override
public int getPhase() {
return 0;
}
}
but this isn't an option for me because component is created after datasource (I have liquibase setup so it's too late) and Phase is still the same that means FIFO order and I'd like to be FILO.
Mix #Profile and #Component seams to me a bad idea. Profiles are designed to work with Configuration (documentation)
Do you really need profile? In my opinion it makes sense if you have several possible configurations, one based on H2, and if you want be able to switch between these configurations (typically at start time by setting a properties...)
Manage the H2 server with a bean (documentation) seams correct to me (as suggested by Stefen). Maybe you will prefer annotations... If you want a spring profile, then you will need a Configuration object too. It will simply load the H2 server bean (in my opinion it's better to manage the H2 server lifecycle with a bean than with a context/config).
Create your server as a bean :
#Bean(initMethod = "start", destroyMethod = "stop")
Server h2Server() throws Exception {
return Server.createTcpServer("-tcp","-tcpAllowOthers","-tcpPort","9192");
}
Now you can configure spring to create other beans (e.g the datasource)
after the bean h2Server using #DependsOn
#DependsOn("h2Server")
#Bean
DataSource dataSource(){
...
}
Hi, what about using spring boot? It has automatically configured datasource so I don't want to reconfigure it.
You are right, to use the above approach you have to create your own datasource in order to annotate it with #DependsOn .
But it looks like this is not really necessary.
In one of my projects I am creating the h2Server as a bean as described.
I use the datasource created by spring, so without any #DependsOn.
It works perfectly. Just give it a try.
Your solution with SmartLifecycle does not work, because it creates the server on ApplicationContext refresh, which happens after all beans (including the datasource ) were created.
Is it possible to have my app update the config settings at runtime? I can easily expose the settings I want in my UI but is there a way to allow the user to update settings and make them permanent ie save them to the config.yaml file? The only way I can see it to update the file by hand then restart the server which seems a bit limiting.
Yes. It is possible to reload the service classes at runtime.
Dropwizard by itself does not have the way to reload the app, but jersey has.
Jersey uses a container object internally to maintain the running application. Dropwizard uses the ServletContainer class of Jersey to run the application.
How to reload the app without restarting it -
Get a handle to the container used internally by jersey
You can do this by registering a AbstractContainerLifeCycleListener in Dropwizard Environment before starting the app. and implement its onStartup method as below -
In your main method where you start the app -
//getting the container instance
environment.jersey().register(new AbstractContainerLifecycleListener() {
#Override
public void onStartup(Container container) {
//initializing container - which will be used to reload the app
_container = container;
}
});
Add a method to your app to reload the app. It will take in the list of string which are the names of the service classes you want to reload. This method will call the reload method of the container with the new custom DropWizardConfiguration instance.
In your Application class
public static synchronized void reloadApp(List<String> reloadClasses) {
DropwizardResourceConfig dropwizardResourceConfig = new DropwizardResourceConfig();
for (String className : reloadClasses) {
try {
Class<?> serviceClass = Class.forName(className);
dropwizardResourceConfig.registerClasses(serviceClass);
System.out.printf(" + loaded class %s.\n", className);
} catch (ClassNotFoundException ex) {
System.out.printf(" ! class %s not found.\n", className);
}
}
_container.reload(dropwizardResourceConfig);
}
For more details see the example documentation of jersey - jersey example for reload
Consider going through the code and documentation of following files in Dropwizard/Jersey for a better understanding -
Container.java
ContainerLifeCycleListener.java
ServletContainer.java
AbstractContainerLifeCycleListener.java
DropWizardResourceConfig.java
ResourceConfig.java
No.
Yaml file is parsed at startup and given to the application as Configuration object once and for all. I believe you can change the file after that but it wouldn't affect your application until you restart it.
Possible follow up question: Can one restart the service programmatically?
AFAIK, no. I've researched and read the code somewhat for that but couldn't find a way to do that yet. If there is, I'd love to hear that :).
I made a task that reloads the main yaml file (it would be useful if something in the file changes). However, it is not reloading the environment. After researching this, Dropwizard uses a lot of final variables and it's quite hard to reload these on the go, without restarting the app.
class ReloadYAMLTask extends Task {
private String yamlFileName;
ReloadYAMLTask(String yamlFileName) {
super("reloadYaml");
this.yamlFileName = yamlFileName;
}
#Override
public void execute(ImmutableMultimap<String, String> parameters, PrintWriter output) throws Exception {
if (yamlFileName != null) {
ConfigurationFactoryFactory configurationFactoryFactory = new DefaultConfigurationFactoryFactory<ReportingServiceConfiguration>();
ValidatorFactory validatorFactory = Validation.buildDefaultValidatorFactory();
Validator validator = validatorFactory.getValidator();
ObjectMapper objectMapper = Jackson.newObjectMapper();
final ConfigurationFactory<ServiceConfiguration> configurationFactory = configurationFactoryFactory.create(ServiceConfiguration.class, validator, objectMapper, "dw");
File confFile = new File(yamlFileName);
configurationFactory.build(new File(confFile.toURI()));
}
}
}
You can change the configuration in the YAML and read it while your application is running. This will not however restart the server or change any server configurations. You will be able to read any changed custom configurations and use them. For example, you can change the logging level at runtime or reload other custom settings.
My solution -
Define a custom server command. You should use this command to start your application instead of the "server" command.
ArgsServerCommand.java
public class ArgsServerCommand<WC extends WebConfiguration> extends EnvironmentCommand<WC> {
private static final Logger LOGGER = LoggerFactory.getLogger(ArgsServerCommand.class);
private final Class<WC> configurationClass;
private Namespace _namespace;
public static String COMMAND_NAME = "args-server";
public ArgsServerCommand(Application<WC> application) {
super(application, "args-server", "Runs the Dropwizard application as an HTTP server specific to my settings");
this.configurationClass = application.getConfigurationClass();
}
/*
* Since we don't subclass ServerCommand, we need a concrete reference to the configuration
* class.
*/
#Override
protected Class<WC> getConfigurationClass() {
return configurationClass;
}
public Namespace getNamespace() {
return _namespace;
}
#Override
protected void run(Environment environment, Namespace namespace, WC configuration) throws Exception {
_namespace = namespace;
final Server server = configuration.getServerFactory().build(environment);
try {
server.addLifeCycleListener(new LifeCycleListener());
cleanupAsynchronously();
server.start();
} catch (Exception e) {
LOGGER.error("Unable to start server, shutting down", e);
server.stop();
cleanup();
throw e;
}
}
private class LifeCycleListener extends AbstractLifeCycle.AbstractLifeCycleListener {
#Override
public void lifeCycleStopped(LifeCycle event) {
cleanup();
}
}
}
Method to reload in your Application -
_ymlFilePath = null; //class variable
public static boolean reloadConfiguration() throws IOException, ConfigurationException {
boolean reloaded = false;
if (_ymlFilePath == null) {
List<Command> commands = _configurationBootstrap.getCommands();
for (Command command : commands) {
String commandName = command.getName();
if (commandName.equals(ArgsServerCommand.COMMAND_NAME)) {
Namespace namespace = ((ArgsServerCommand) command).getNamespace();
if (namespace != null) {
_ymlFilePath = namespace.getString("file");
}
}
}
}
ConfigurationFactoryFactory configurationFactoryFactory = _configurationBootstrap.getConfigurationFactoryFactory();
ValidatorFactory validatorFactory = _configurationBootstrap.getValidatorFactory();
Validator validator = validatorFactory.getValidator();
ObjectMapper objectMapper = _configurationBootstrap.getObjectMapper();
ConfigurationSourceProvider provider = _configurationBootstrap.getConfigurationSourceProvider();
final ConfigurationFactory<CustomWebConfiguration> configurationFactory = configurationFactoryFactory.create(CustomWebConfiguration.class, validator, objectMapper, "dw");
if (_ymlFilePath != null) {
// Refresh logging level.
CustomWebConfiguration webConfiguration = configurationFactory.build(provider, _ymlFilePath);
LoggingFactory loggingFactory = webConfiguration.getLoggingFactory();
loggingFactory.configure(_configurationBootstrap.getMetricRegistry(), _configurationBootstrap.getApplication().getName());
// Get my defined custom settings
CustomSettings customSettings = webConfiguration.getCustomSettings();
reloaded = true;
}
return reloaded;
}
Although this feature isn't supported out of the box by dropwizard, you're able to accomplish this fairly easy with the tools they give you.
Before I get started, note that this isn't a complete solution for the question asked as it doesn't persist the updated config values to the config.yml. However, this would be easy enough to implement yourself simply by writing to the config file from the application. If anyone would like to write this implementation feel free to open a PR on the example project I've linked below.
Code
Start off with a minimal config:
config.yml
myConfigValue: "hello"
And it's corresponding configuration file:
ExampleConfiguration.java
public class ExampleConfiguration extends Configuration {
private String myConfigValue;
public String getMyConfigValue() {
return myConfigValue;
}
public void setMyConfigValue(String value) {
myConfigValue = value;
}
}
Then create a task which updates the config:
UpdateConfigTask.java
public class UpdateConfigTask extends Task {
ExampleConfiguration config;
public UpdateConfigTask(ExampleConfiguration config) {
super("updateconfig");
this.config = config;
}
#Override
public void execute(Map<String, List<String>> parameters, PrintWriter output) {
config.setMyConfigValue("goodbye");
}
}
Also for demonstration purposes, create a resource which allows you to get the config value:
ConfigResource.java
#Path("/config")
public class ConfigResource {
private final ExampleConfiguration config;
public ConfigResource(ExampleConfiguration config) {
this.config = config;
}
#GET
public Response handleGet() {
return Response.ok().entity(config.getMyConfigValue()).build();
}
}
Finally wire everything up in your application:
ExampleApplication.java (exerpt)
environment.jersey().register(new ConfigResource(configuration));
environment.admin().addTask(new UpdateConfigTask(configuration));
Usage
Start up the application then run:
$ curl 'http://localhost:8080/config'
hello
$ curl -X POST 'http://localhost:8081/tasks/updateconfig'
$ curl 'http://localhost:8080/config'
goodbye
How it works
This works simply by passing the same reference to the constructor of ConfigResource.java and UpdateConfigTask.java. If you aren't familiar with the concept see here:
Is Java "pass-by-reference" or "pass-by-value"?
The linked classes above are to a project I've created which demonstrates this as a complete solution. Here's a link to the project:
scottg489/dropwizard-runtime-config-example
Footnote: I haven't verified this works with the built in configuration. However, the dropwizard Configuration class which you need to extend for your own configuration does have various "setters" for internal configuration, but it may not be safe to update those outside of run().
Disclaimer: The project I've linked here was created by me.
When I shut down Tomcat, I observe a correct shutdown and cleanup of the Spring WebApplicationContext. However, when I redeploy my Spring-based WAR (by copying the new WAR to webapps), normal shutdown does not occur. This is a problem for me due to all the ensuing resource leaks:
org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [hz.hazelcast-swipe-instance.scheduled] but has failed to stop it. This is very likely to create a memory leak.
org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [hz.hazelcast-swipe-instance.operation.thread-0] but has failed to stop it. This is very likely to create a memory leak.
... and many more. I am using XML-less configuration, this is my WebApplicationInitializer:
public class WebApplicationInitializer extends AbstractAnnotationConfigDispatcherServletInitializer
{
#Override protected Class<?>[] getRootConfigClasses() {
return new Class[] { WebSecurityConfig.class, WebMvcConfig.class };
}
#Override protected Class<?>[] getServletConfigClasses() { return null; }
#Override protected String[] getServletMappings() { return new String[] { "/" }; }
#Override public void onStartup(ServletContext ctx) throws ServletException {
ctx.setInitParameter("spring.profiles.active", "production");
super.onStartup(ctx);
}
}
There is no configuration specific to controlling the behavior upon servlet context reload, and I assume this should have worked out of the box.
Is there a way to make the WebApplicationContext close properly before continuing the servlet context reloading procedure?
I am on Spring 4.0.5, Tomcat 7.0.54, Hazelcast 3.2.1, Hibernate 4.3.4.Final.
Update
I have added a Spring application listener for the ContextClosedEvent and printed the stack trace of its invocation:
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:333) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:335) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:880) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:841) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.destroy(FrameworkServlet.java:819) [spring-webmvc-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1486) [catalina.jar:7.0.54]
at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1847) [catalina.jar:7.0.54]
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232) [catalina.jar:7.0.54]
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5647) [catalina.jar:7.0.54]
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232) [catalina.jar:7.0.54]
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575) [catalina.jar:7.0.54]
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564) [catalina.jar:7.0.54]
This indicates that the Spring shutdown occurs in its Servlet#destroy method. This is the relevant snippet from AbstractApplicationContext#close():
if (logger.isInfoEnabled()) {
logger.info("Closing " + this);
}
LiveBeansView.unregisterApplicationContext(this);
try {
// Publish shutdown event.
publishEvent(new ContextClosedEvent(this));
}
catch (Throwable ex) {
logger.warn("Exception thrown from ApplicationListener handling ContextClosedEvent", ex);
}
// Stop all Lifecycle beans, to avoid delays during individual destruction.
try {
getLifecycleProcessor().onClose();
}
catch (Throwable ex) {
logger.warn("Exception thrown from LifecycleProcessor on context close", ex);
}
// Destroy all cached singletons in the context's BeanFactory.
destroyBeans();
// Close the state of this context itself.
closeBeanFactory();
// Let subclasses do some final clean-up if they wish...
onClose();
synchronized (this.activeMonitor) {
this.active = false;
}
I see the log entry from the start of this snippet, and I get my ContextClosedEvent. I also see an entry DefaultLifecycleProcessor - Stopping beans in phase 2147483647, which probably comes from the getLifecycleProcessor.onClose() line. It seems that some error occurs downstream from that. Some exception may be swallowed.
Update 2
As requested, this is how I configure Hazelcast:
#Bean(destroyMethod="shutdown") public HazelcastInstance hazelcast() {
final Config c = hzConfig();
final JoinConfig join = c.getNetworkConfig().getJoin();
join.getMulticastConfig().setEnabled(false);
join.getTcpIpConfig().setEnabled(true);
return getOrCreateHazelcastInstance(c);
}
hzConfig() is a method where instance name, group name and password, map names, and map indices are configured, so I don't think it is of interest here.
And this is my Hibernate SessionFactory config:
#Bean
public LocalSessionFactoryBean sessionFactory() {
final LocalSessionFactoryBean b = new LocalSessionFactoryBean();
b.setDataSource(dataSource);
b.setHibernateProperties(props(
"hibernate.connection.release_mode", "on_close",
"hibernate.id.new_generator_mappings", "true",
"hibernate.hbm2ddl.auto", "update",
"hibernate.order_inserts", "true",
"hibernate.order_updates", "true",
"hibernate.max_fetch_depth", "0",
"hibernate.jdbc.fetch_size", "200",
"hibernate.jdbc.batch_size", "50",
"hibernate.jdbc.batch_versioned_data", "true",
"hibernate.jdbc.use_streams_for_binary", "true",
"hibernate.use_sql_comments", "true"
));
return b;
}
At some point, you mentioned that there was a NoClassDefFoundError for Logback. You got this fixed by removing this dependency, but then the problem moved to a another class - one of Spring's own classes.
This can mean that either one of the libraries you have does something buggy with class loaders or maybe Tomcat needs instructions not to keep locks on some resources. See here more about Tomcat resources being locked and the <Context> setting to try: in your Tomcat's conf/context.xml place a antiResourceLocking="true" to the element.
Have you tried upping unloadDelay (defaults to 2000ms) for Tomcat contexts? See http://tomcat.apache.org/tomcat-7.0-doc/config/context.html
UPDATE: I see that you are having issues with logback as well, it might be worth the shot to try and register this listener as well:
class LogbackShutdownListener implements ServletContextListener {
#Override
public void contextDestroyed(ServletContextEvent event) {
LoggerContext loggerContext = (LoggerContext)LoggerFactory.getILoggerFactory();
System.out.println("Shutting down Logback context '" + loggerContext.getName() + "' for " + contextRootFor(event));
loggerContext.stop();
}
#Override
public void contextInitialized(ServletContextEvent event) {
System.out.println("Logback context shutdown listener registered for " + contextRootFor(event));
}
private String contextRootFor(ServletContextEvent event) {
return event.getServletContext().getContextPath();
}
}
Be sure to declare this listener before the spring context loader listener so that it is invoked after the context listener upon shutdown.
UPDATE 2: Also it might be worth the try to register another bean to handle closing of the Hazelcast stuff manually (be sure to also remove destroyMethod from the hazelcast bean):
#Component
class HazelcastDestructor {
#Autowired
private HazelcastInstance instance;
#PreDestroy
public void shutdown() {
try {
instance.shutdown();
} catch (Exception e) {
System.out.println("Hazelcast failed to shutdown(): " + e);
throw e;
}
}
}
UPDATE 3: Just out of curiosity, have you tried parallel deployment: http://www.javacodegeeks.com/2011/06/zero-downtime-deployment-and-rollback.html. It might behave differently than reloading the very same context. At the very least you should be able to undeploy the old version lazily and see if that makes a difference.
There is a similar issue on the dangling threads while container restarting here.
Of all the answers, one particular answer of interest was by Howard - which shows the way these threads are cleared.
There is some good discussion and reasoning as to how this can terminate the threads here.
Now implement ServletContextListener and take care of these threads in the contextDestroyed() method as:
public class YourListener implements ServletContextListener{
....
#Override
public void contextDestroyed(ServletContextEvent event) {
//Call the immolate method here
}
}
Register this listener in WebApplicationInitilizer as:
ctx.addListener(new YourListener());
So when server is restarted - contextDestroyed method is called and this takes care of all these threads.
From Web App development point of view, ServletContainer can only notify the before started and before end process of app.
It is using ServletContextListener.
Config ServletContextListener in web.xml
<listener>
<listener-class>com.var.YourListener</listener-class>
</listener>
YourListener.java
public class YourListener implements ServletContextListener {
public void contextInitialized(ServletContextEvent sce) {
//initialization process
}
public void contextDestroyed(ServletContextEvent sce) {
//destory process
}
}
Update -XML Less
Programmatically
#Override
public void onStartup(ServletContext ctx) throws ServletException {
ctx.addListener(new YourContextListener());
ctx.setInitParameter("spring.profiles.active", "production");
super.onStartup(ctx);
}
Annotation
#WebListener / #WebServletContextListener
public class YourContextListener implements ServletContextListener {
public void contextInitialized(ServletContextEvent servletContextEvent) {
}
public void contextDestroyed(ServletContextEvent servletContextEvent) {
}
}
Update- ShoutDown Hook In Spring
I never use it before my app development, we can register shoutdownhook event into AbstractApplicationContext of Spring.
I am not sure it will be ok or not for you.
AbstractApplicationContext context = ...
context.registerShutdownHook();
Reference 1 Reference 2
I have been trying to get a Windows service running from my JAR file. WinRun4j seems to be able to do the job, but I can't get it to work. I am especially finding it quite difficult to debug. I tried several methods for logging (writing to a .txt file, WinRun4j's EventLog class) but I can't seem to generate any output.
The service installs fine (eventually..) and I can start it. It should start a Jetty server that generates an XML file that can be reached over HTTP. The app works for a stand-alone version, just not for the service. The service is started, but as soon as I call the URL it stops without generating an error.
This is my Service class:
package com.some.package;
import org.boris.winrun4j.AbstractService;
import org.boris.winrun4j.ServiceException;
/**
* A basic service.
*/
public class StockService extends AbstractService {
private StockServer srv;
public int serviceMain(String[] args) throws ServiceException {
while (!shutdown) {
try {
Thread.sleep(5000);
} catch(InterruptedException e) {
}
if(srv == null) {
try {
srv = new StockServer();
srv.start();
} catch (Exception e) {
}
}
}
return 0;
}
}
I found out that the service didn't want to start if I started the Jetty server from the serviceMain class. I had to start a new thread. So StockServer extends Thread:
public class StockServer extends Thread {
private Server server;
public void run() {
if (server == null) {
try {
server = new Server(8080);
ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS);
context.setContextPath("/example");
StockServlet stockServlet = new StockServlet();
context.addServlet(new ServletHolder(stockServlet), "/stock/*");
server.setHandler(context);
server.setStopAtShutdown(true);
server.start();
server.join();
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
}
Since it runs perfectly fine as Java application I just don't know how to get this thing debugged. I hope one of you can point me in the right direction :).
I ended up using the Java Service Wrapper (JSW). This seemed a lot more complex but ended up to be quite easy. It also provides logging by default so I could easily fix the errors. The JSW had problems finding the correct JDK, since JSW is 32bit and I installed JDK1.7 64 bit (and 1.6 32bit). Installing JDK1.7 32bit fixed it. That might have been the problem with WinRun4j as well, but that is something I will never know :).
I'm experiencing a memory leak due to orphaned threads in Tomcat. Particularly, it seems that Guice and the JDBC driver are not closing threads.
Aug 8, 2012 4:09:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak.
Aug 8, 2012 4:09:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
I know this is similar to other questions (such as this one), but in my case, the answer of "don't worry about it" won't be sufficient, as it is causing problems for me. I have CI server which regularly updates this application, and after 6-10 reloads, the CI server will hang because Tomcat is out of memory.
I need to be able to clear up these orphaned threads so I can run my CI server more reliably. Any help would be appreciated!
I just dealt with this problem myself. Contrary to some other answers, I do not recommend issuing the t.stop() command. This method has been deprecated, and for good reason. Reference Oracle's reasons for doing this.
However there is a solution for removing this error without needing to resort to t.stop()...
You can use most of the code #Oso provided, just replace the following section
Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]);
for(Thread t:threadArray) {
if(t.getName().contains("Abandoned connection cleanup thread")) {
synchronized(t) {
t.stop(); //don't complain, it works
}
}
}
Replace it using the following method provided by the MySQL driver:
try {
AbandonedConnectionCleanupThread.shutdown();
} catch (InterruptedException e) {
logger.warn("SEVERE problem cleaning up: " + e.getMessage());
e.printStackTrace();
}
This should properly shutdown the thread, and the error should go away.
I've had the same issue, and as Jeff says, the "don't worry about it approach" was not the way to go.
I did a ServletContextListener that stops the hung thread when the context is being closed, and then registered such ContextListener on the web.xml file.
I already know that stopping a thread is not an elegant way to deal with them, but otherwise the server keeps on crashing after two or three deploys (it is not always possible to restart the app server).
The class I created is:
public class ContextFinalizer implements ServletContextListener {
private static final Logger LOGGER = LoggerFactory.getLogger(ContextFinalizer.class);
#Override
public void contextInitialized(ServletContextEvent sce) {
}
#Override
public void contextDestroyed(ServletContextEvent sce) {
Enumeration<Driver> drivers = DriverManager.getDrivers();
Driver d = null;
while(drivers.hasMoreElements()) {
try {
d = drivers.nextElement();
DriverManager.deregisterDriver(d);
LOGGER.warn(String.format("Driver %s deregistered", d));
} catch (SQLException ex) {
LOGGER.warn(String.format("Error deregistering driver %s", d), ex);
}
}
Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]);
for(Thread t:threadArray) {
if(t.getName().contains("Abandoned connection cleanup thread")) {
synchronized(t) {
t.stop(); //don't complain, it works
}
}
}
}
}
After creating the class, then register it on the web.xml file:
<web-app...
<listener>
<listener-class>path.to.ContextFinalizer</listener-class>
</listener>
</web-app>
The least invasive workaround is to force initialisation of the MySQL JDBC driver from code outside of the webapp's classloader.
In tomcat/conf/server.xml, modify (inside the Server element):
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
to
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
classesToInitialize="com.mysql.jdbc.NonRegisteringDriver" />
With mysql-connector-java-8.0.x use com.mysql.cj.jdbc.NonRegisteringDriver instead
This assumes you put the MySQL JDBC driver into tomcat's lib directory and not inside your webapp.war's WEB-INF/lib directory, as the whole point is to load the driver before and independently of your webapp.
References:
http://bugs.mysql.com/bug.php?id=68556#c400606
http://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html#JRE_Memory_Leak_Prevention_Listener_-_org.apache.catalina.core.JreMemoryLeakPreventionListener
http://markmail.org/message/dmvlkps7lbgpngil
com.mysql.jdbc.NonRegisteringDriver source v5.1
com.mysql.cj.jdbc.NonRegisteringDriver source v8.0
Changes in connector/J v8.0
Effective from MySQL connector 5.1.23 onwards, a method is provided to shut the abandoned connection cleanup thread down, AbandonedConnectionCleanupThread.shutdown.
However, we don't want direct dependencies in our code on the otherwise opaque JDBC driver code, so my solution is to use reflection to find the class and method and invoke it if found. The following complete code snippet is all that's needed, executed in the context of the class loader that loaded the JDBC driver:
try {
Class<?> cls=Class.forName("com.mysql.jdbc.AbandonedConnectionCleanupThread");
Method mth=(cls==null ? null : cls.getMethod("shutdown"));
if(mth!=null) { mth.invoke(null); }
}
catch (Throwable thr) {
thr.printStackTrace();
}
This cleanly ends the thread if the JDBC driver is a sufficiently recent version of the MySQL connector and otherwise does nothing.
Note it has to be executed in the context of the class loader because the thread is a static reference; if the driver class is not being or has not already been unloaded when this code is run then the thread will not be running for subsequent JDBC interactions.
I took the best parts of the answers above and combined them into an easily extensible class. This combines Oso's original suggestion with Bill's driver improvement and Software Monkey's reflection improvement. (I liked the simplicity of Stephan L's answer too, but sometimes modifying the Tomcat environment itself is not a good option, especially if you have to deal with autoscaling or migration to another web container.)
Instead of directly referring to the class name, thread name, and stop method, I also encapsulated these into an private inner ThreadInfo class. Using a list of these ThreadInfo objects, you can include additional troublesome threads to be shutdown with the same code. This is a bit more complex of a solution than most people likely need, but should work more generally when you need that.
import java.lang.reflect.Method;
import java.sql.Driver;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Arrays;
import java.util.Enumeration;
import java.util.List;
import java.util.Set;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Context finalization to close threads (MySQL memory leak prevention).
* This solution combines the best techniques described in the linked Stack
* Overflow answer.
* #see Tomcat Guice/JDBC Memory Leak
*/
public class ContextFinalizer
implements ServletContextListener {
private static final Logger LOGGER =
LoggerFactory.getLogger(ContextFinalizer.class);
/**
* Information for cleaning up a thread.
*/
private class ThreadInfo {
/**
* Name of the thread's initiating class.
*/
private final String name;
/**
* Cue identifying the thread.
*/
private final String cue;
/**
* Name of the method to stop the thread.
*/
private final String stop;
/**
* Basic constructor.
* #param n Name of the thread's initiating class.
* #param c Cue identifying the thread.
* #param s Name of the method to stop the thread.
*/
ThreadInfo(final String n, final String c, final String s) {
this.name = n;
this.cue = c;
this.stop = s;
}
/**
* #return the name
*/
public String getName() {
return this.name;
}
/**
* #return the cue
*/
public String getCue() {
return this.cue;
}
/**
* #return the stop
*/
public String getStop() {
return this.stop;
}
}
/**
* List of information on threads required to stop. This list may be
* expanded as necessary.
*/
private List<ThreadInfo> threads = Arrays.asList(
// Special cleanup for MySQL JDBC Connector.
new ThreadInfo(
"com.mysql.jdbc.AbandonedConnectionCleanupThread", //$NON-NLS-1$
"Abandoned connection cleanup thread", //$NON-NLS-1$
"shutdown" //$NON-NLS-1$
)
);
#Override
public void contextInitialized(final ServletContextEvent sce) {
// No-op.
}
#Override
public final void contextDestroyed(final ServletContextEvent sce) {
// Deregister all drivers.
Enumeration<Driver> drivers = DriverManager.getDrivers();
while (drivers.hasMoreElements()) {
Driver d = drivers.nextElement();
try {
DriverManager.deregisterDriver(d);
LOGGER.info(
String.format(
"Driver %s deregistered", //$NON-NLS-1$
d
)
);
} catch (SQLException e) {
LOGGER.warn(
String.format(
"Failed to deregister driver %s", //$NON-NLS-1$
d
),
e
);
}
}
// Handle remaining threads.
Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]);
for (Thread t:threadArray) {
for (ThreadInfo i:this.threads) {
if (t.getName().contains(i.getCue())) {
synchronized (t) {
try {
Class<?> cls = Class.forName(i.getName());
if (cls != null) {
Method mth = cls.getMethod(i.getStop());
if (mth != null) {
mth.invoke(null);
LOGGER.info(
String.format(
"Connection cleanup thread %s shutdown successfully.", //$NON-NLS-1$
i.getName()
)
);
}
}
} catch (Throwable thr) {
LOGGER.warn(
String.format(
"Failed to shutdown connection cleanup thread %s: ", //$NON-NLS-1$
i.getName(),
thr.getMessage()
)
);
thr.printStackTrace();
}
}
}
}
}
}
}
I went a step further from Oso,improved the code above in two points:
Added the Finalizer thread to the need-to-kill check:
for(Thread t:threadArray) {
if(t.getName().contains("Abandoned connection cleanup thread")
|| t.getName().matches("com\\.google.*Finalizer")
) {
synchronized(t) {
logger.warn("Forcibly stopping thread to avoid memory leak: " + t.getName());
t.stop(); //don't complain, it works
}
}
}
Sleep for a little while to give threads time to stop. Without that, tomcat kept complaining.
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
logger.debug(e.getMessage(), e);
}
Bill's solution looks good, however I found another solution directly in MySQL bug reports:
[5 Jun 2013 17:12] Christopher Schultz
Here is a much better workaround until something else changes.
Enable Tomcat's JreMemoryLeakPreventionListener (enabled by default on Tomcat 7), and add this attribute to the element:
classesToInitialize="com.mysql.jdbc.NonRegisteringDriver"
If "classesToInitialize" is already set on your , just add NonRegisteringDriver to the existing value separated by a comma.
and the answer:
[8 Jun 2013 21:33] Marko Asplund
I did some testing with the JreMemoryLeakPreventionListener / classesToInitialize workaround (Tomcat 7.0.39 + MySQL Connector/J 5.1.25).
Before applying the workaround thread dumps listed multiple AbandonedConnectionCleanupThread instances after redeploying the webapp several times. After applying the workaround there's only one AbandonedConnectionCleanupThread instance.
I had to modify my app, though, and move MySQL driver from the webapp to Tomcat lib.
Otherwise, the classloader is unable to load com.mysql.jdbc.NonRegisteringDriver at Tomcat startup.
I hope it helps for all who still fighting with this issue...
It seems this was fixed in 5.1.41. You could upgrade Connector/J to 5.1.41 or newer.
https://dev.mysql.com/doc/relnotes/connector-j/5.1/en/news-5-1-41.html
The implementation of AbandonedConnectionCleanupThread has now been improved, so that there are now four ways for developers to deal with the situation:
When the default Tomcat configuration is used and the Connector/J jar is put into a local library directory, the new built-in application detector in Connector/J now detects the stopping of the web application within 5 seconds and kills AbandonedConnectionCleanupThread. Any unnecessary warnings about the thread being unstoppable are also avoided. If the Connector/J jar is put into a global library directory, the thread is left running until the JVM is unloaded.
When Tomcat's context is configured with the attribute clearReferencesStopThreads="true", Tomcat is going to stop all spawned threads when the application stops unless Connector/J is being shared with other web applications, in which case Connector/J is now protected against an inappropriate stop by Tomcat; the warning about the non-stoppable thread is still issued into Tomcat's error log.
When a ServletContextListener is implemented within each web application that calls AbandonedConnectionCleanupThread.checkedShutdown() on context destruction, Connector/J now, again, skips this operation if the driver is potentially shared with other applications. No warning about the thread being unstoppable is issued to Tomcat's error log in this case.
When AbandonedConnectionCleanupThread.uncheckedShutdown() is called, the AbandonedConnectionCleanupThread is closed even if Connector/J is shared with other applications. However, it may not be possible to restart the thread afterwards.
If you look at source code, they called setDeamon(true) on thread, so it won't block shutdown.
Thread t = new Thread(r, "Abandoned connection cleanup thread");
t.setDaemon(true);
See To prevent a memory leak, the JDBC Driver has been forcibly unregistered. Bill's answer deregisters all Driver instances as well as instances that may belong to other web applications. I have extended Bill's answer with a check that the Driver instance belongs to the right ClassLoader.
Here is the resulting code (in a separate method, because my contextDestroyed has other things to do):
// See https://stackoverflow.com/questions/25699985/the-web-application-appears-to-have-started-a-thread-named-abandoned-connect
// and
// https://stackoverflow.com/questions/3320400/to-prevent-a-memory-leak-the-jdbc-driver-has-been-forcibly-unregistered/23912257#23912257
private void avoidGarbageCollectionWarning()
{
ClassLoader cl = Thread.currentThread().getContextClassLoader();
Enumeration<Driver> drivers = DriverManager.getDrivers();
Driver d = null;
while (drivers.hasMoreElements()) {
try {
d = drivers.nextElement();
if(d.getClass().getClassLoader() == cl) {
DriverManager.deregisterDriver(d);
logger.info(String.format("Driver %s deregistered", d));
}
else {
logger.info(String.format("Driver %s not deregistered because it might be in use elsewhere", d.toString()));
}
}
catch (SQLException ex) {
logger.warning(String.format("Error deregistering driver %s, exception: %s", d.toString(), ex.toString()));
}
}
try {
AbandonedConnectionCleanupThread.shutdown();
}
catch (InterruptedException e) {
logger.warning("SEVERE problem cleaning up: " + e.getMessage());
e.printStackTrace();
}
}
I wonder whether the call AbandonedConnectionCleanupThread.shutdown() is safe. Can it interfere with other web applications? I hope not, because the AbandonedConnectionCleanupThread.run() method is not static but the AbandonedConnectionCleanupThread.shutdown() method is.