Develop Reference

Tomcat hangs when shutting down (daemon) threads (ScheduledExecutorService)

In my webapp i have classes that used to extend thread class with a run() routine. I had problems with them when shutting down tomcat so replaced the thread implementations with a ScheduledExecutorService.
However it still does not work: When shutting down tomcat in my eclipse console there are logging statements like this:
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Appname] appears to have started a thread named [Timer-1] but has failed to stop it. This is very likely to create a memory leak.
After a timeout i always get a message, that tomcat is not responding and i have to terminate it manually.
I am starting those in an init() and destroy() method of my servlet like this:
public void init() throws ServletException
{
worker = new worker();
worker.schedule();
}
public void destroy()
{
worker.shutdownTimer();
worker = null;
}
The methods look like this:
private ScheduledExecutorService mScheduler = null;
public void schedule()
{
scheduler = getScheduleService();
scheduler.scheduleAtFixedRate(new Runnable()
{
public void run()
{
//...things to do in the routine
}
},
1, //start after 1 second
10, //wait 10 seconds
TimeUnit.SECONDS);
}
private ScheduledExecutorService getScheduleService()
{
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1, new ThreadFactory()
{
public Thread newThread(Runnable r)
{
Thread thread = new Thread(r);
// allow the JVM to kill the scheduled task
thread.setDaemon(true);
return thread;
}
});
return scheduler;
}
public void shutdownTimer()
{
if (scheduler != null)
scheduler.shutdownNow();
}
In my debugger i can see that the executorservice has started following thread:
Daemon Thread [Timer-0] Running
Thread [pool-3-thread-1] Running
Why does the ExecutorService start a normal threadpool?
And when does the jvm still have problem closing those threads when i'm explicitly telling it to shut them down in my servlet?

Trouble stopping a tomcat server that is running a web application with a while (true) loop

I'm developing a web application to be deployed onto Tomcat. When Tomcat is started, I use a servlet (in web.xml) to call a Java class:
<web-app>
<display-name>Consumer</display-name>
<servlet>
<servlet-name>start</servlet-name>
<servlet-class>com.test.sample.Consumer</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
</web-app>
My Consumer.java subscribes to a queue on an AMQP server. I achieve this by using a while (true) loop, which works fine in a standalone Java program. Itt also works in the context of the web application, but I can never stop my Tomcat server (within my NetBeans IDE), and I believe that the while loop is the culprit. Here is some code:
public class Consumer {
public Consumer()
consume();
}
private void consume()
...
while (true) {
// Await incoming messages from queue
// Process message
}
}
}
Is there a better way to handle this? Or to signal a stop to break out of the loop?
Thanks!
Updated to use ServletContextListener:
public final class ApplicationListener implements ServletContextListener {
private ScheduledExecutorService scheduler;
public ApplicationListener() {
}
#Override
public void contextDestroyed(ServletContextEvent event) {
System.out.println("***** Stopping Consumer *****");
scheduler.shutdownNow();
}
#Override
public void contextInitialized(ServletContextEvent event) {
System.out.println("***** Starting Consumer *****");
scheduler = Executors.newSingleThreadScheduledExecutor();
scheduler.scheduleAtFixedRate(new ScheduledConsumer(), 0, 15000, TimeUnit.MILLISECONDS);
}
public class ScheduledConsumer implements Runnable {
#Override
public void run() {
Consumer k = new Consumer();
k.consumeOnce();
}
}
}

I have some suggestions, but they require that you modify your architecture a bit in order to more nicely play with your container environment.
Servlet containers support "listeners" that can get notification of various events. Specifically, one of them is the ServletContextListener which gets notified when the context (aka. webapp) is being brought into service (via the contextInitialized method) and when it is being brought out of service (via the contextDestroyed method).
My recommendation would be to do the following:
Change your Consumer class's constructor so that it does not automatically call consume(); instead, add a public method like consumeOnce and don't use a loop at that level at all
Write a ServletContextListener that has a Consumer and a Thread reference as members as well as a volatile boolean stop flag; in contextInitialized it should create a new Consumer object, then launch a new (daemon) thread that:
Calls Consumer.consumeOnce
Calls Thread.sleep for an appropriate amount of time
Loops over the previous 2 steps until the stop flag is true
Have your ServletContextListener's contextDestroyed method set the stop flag to true and call Thread.interrupt on the running thread.
I'm sure I'm missing some exact details, but that's the general idea. When Tomcat shuts down, your code will be notified of the shutdown and you can cleanly terminate your own looping-thread. You may need to provide a way for the Consumer to abort an attempt to consume whatever it consumes (e.g. stop waiting to pull an object from an empty queue) if it doesn't abort when it gets a Thread.interrupt signal. (For instance if you use an Object.wait() in order to wait for a monitor notification, then you'll want to change that so it uses a wait with a timeout so that you won't block forever).

You have to place the code with the loop in a different thread and start the thread from your consumer.
private void consume() {
Thread x = new Thread(new Runnable() {
#Override
public void run() {
while(true) {
....
}
});
x.start();
}

Java Shutdown hook not run

I am new to Java/threads and I inherited something like the following code. It is a command line program that main() only starts 5-6 different kind of threads and exits with ^C. I want to add a shutdown hook to close all threads properly and adapted it the following way.
I added a Shutdown hook and a stopThread() method in all threads (like the one in MyWorker class)
The problem is that when I press ^C I don't see the end message from the Thread's run method. Is this done in the background or is there something wrong with my method. Also, Is there a better pattern I should follow?
Thanks
public class Main {
public static MyWorker worker1 = new MyWorker();
// .. various other threads here
public static void startThreads() {
worker1.start();
// .. start other threads
}
public static void stopThreads() {
worker1.stopThread();
// .. stop other threads
}
public static void main(String[] args)
throws Exception {
startThreads();
// TODO this needs more work (later)
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
try {
stopThreads();
} catch (Exception exp) {
}
}
});
} }
public class MyWorker extends Thread {
private volatile boolean stop = false;
public void stopThread() {
stop = true;
}
public void run() {
while (!stop) {
// Do stuff here
}
// Print exit message with logger
}
}

Shutdown Hooks may not be executed in some cases!
First thing to keep in mind is that it is not guaranteed that shutdown hooks will always run. If the JVM crashes due to some internal error, then it might crash down without having a chance to execute a single instruction.
Also, if the O/S gives a SIGKILL (http://en.wikipedia.org/wiki/SIGKILL) signal (kill -9 in Unix/Linux) or TerminateProcess (Windows), then the application is required to terminate immediately without doing even waiting for any cleanup activities. In addition to the above, it is also possible to terminate the JVM without allowing the shutdown hooks to run by calling Runime.halt() method.
Shutdown hooks are called when the application terminates normally (when all threads finish, or when System.exit(0) is called). Also, when the JVM is shutting down due to external causes such as user requesting a termination (Ctrl+C), a SIGTERM being issued by O/S (normal kill command, without -9), or when the operating system is shutting down.

When you call System.exit() or terminate via a signal, it stop all the existing threads and starts all the shutdown hooks. i.e. all your threads could be dead by the time you hook starts.
Instead of trying to stop threads cleanly, you should ensure resources are closed cleanly.

I guess you can shift your code to ExecutorService
private final ExecutorService pool;
pool = Executors.newFixedThreadPool(poolSize);
pool.execute(Instance of Runnable);
pool.shutdown();
ExecutorService.shutdown
Initiates an orderly shutdown in which previously submitted tasks are executed, but no new tasks will be accepted. Invocation has no additional effect if already shut down.

Try making your threads as daemon threads.
Add a constructor
public MyWorker(boolean isDaemon) {
this.setDaemon(true);
}
or set to daemon before calling start.
worker1.setDaemon(true);
worker1.start();
When you press Ctrl C and exit, the threads will be stopped.

What is happening here is that you invoke the stopThread() method, but you don't wait the the threads are actually finished before terminating.
If you invoke a join() on all threads before stoping the JVM, you will probably see your 'stop logs'.
public static void stopThreads() {
worker1.stopThread();
// .. stop other threads
for(Thread t: workers) {
t.join();
}
}

Java webapp memory leak when using ScheduledExecutorService

My Tomcat 7 is reporting that there may be a memory leak in my webapp
SEVERE: The web application [/mywebapp] appears to have started a
thread named [pool-1-thread-1] but has failed to stop it. This is
very likely to create a memory leak.
I have a long running task in my webapp that gets initialized when the webapp is started.
public class MyContextListener implements ServletContextListener{
Scheduler scheduler = null;
public MyContextListener(){
scheduler = new Scheduler();
}
#Override
public void contextDestroyed(ServletContextEvent arg0) {
scheduler.stop();
}
#Override
public void contextInitialized(ServletContextEvent arg0) {
scheduler.start();
}
}
.. and my Scheduler.java
public class Scheduler {
private final ScheduledExecutorService fScheduler;
public Scheduler() {
fScheduler = Executors.newScheduledThreadPool(1);
}
public void start(){
fScheduler.scheduleWithFixedDelay(new Runnable() {
#Override
public void run() {
//Perform some task
}
}, 1, 240, TimeUnit.MINUTES);
}
public void stop(){
fScheduler.shutdownNow();
}
}
Even though I calling scheduler.stop(); when shutting down the server, its still reporting there could be a memory leak.
This app is deployed on jelastic.com and I find that once it is started, it runs well for around two days and then the tasks don't seem to be running. There is no exceptions or errors in the logs too.
Am I doing anything wrong here ? Is there really a potential memory leak ?

Calling fScheduler.shutdownNow(); is not enough:
There are no guarantees beyond best-effort attempts to stop processing actively executing tasks.
From JavaDoc.
Instead you must explicitly wait for the tasks that are currently running:
fScheduler.shutdownNow();
fScheduler.awaitTermination(10, TimeUnit.SECONDS);

I believe you should not call the shutdown from the Listener but from the Servlet directly.
contextDestroyed() of the listener is too late for the executor service. As stated in the javadoc All servlets and filters will have been destroyed before any ServletContextListeners are notified of context destruction.
whereas overriding the servlet destroy() should be OK as according to the javadoc This method gives the servlet an opportunity to clean up any resources that are being held (for example, memory, file handles, threads...
#Override
public void destroy( ) {
fScheduler.shutdownNow();
fScheduler.awaitTermination(10, TimeUnit.SECONDS);
super.destroy( );
}

Do I have to manually stop threads in Java?

When my application is ready to exit, either by closing a window or invoking the System.exit() method. Do I have to manually stop the threads I may have created or will Java take care of that for me?

In cases you use System.exit(). All the threads will stop whether or not they are daemon.
Otherwise, the JVM will automatically stop all threads that are daemon threads set by Thread.setDaemon(true). In other words, the jvm will only exit when only threads remaining are all daemon threads or no threads at all.
Consider the example below, it will continue to run even after the main method returns.
but if you set it to daemon, it will terminate when the main method (the main thread) terminates.
public class Test {
public static void main(String[] arg) throws Throwable {
Thread t = new Thread() {
public void run() {
while(true) {
try {
Thread.sleep(300);
System.out.println("Woken up after 300ms");
}catch(Exception e) {}
}
}
};
// t.setDaemon(true); // will make this thread daemon
t.start();
System.exit(0); // this will stop all threads whether are not they are daemon
System.out.println("main method returning...");
}
}

If you want stop threads before exit gracefully, Shutdown Hooks may be a choice.
looks like:
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
//Stop threads }
});
See: hook-design