We use Hazelcast 2.6.2 in a legacy Java clustered application. When the application is stopped the JVM does not terminate any more. It seems that it is caused by Hazelcast threads not being flagged daemon. I did not find a way way through the Hazelcast API to flag them daemon.
Are there recommended solutions to prevent Hazelcast from preventing the JVM to terminate?
Regards
Looking at the Hazelcast Javadocs, I see that there is a shutdownAll(); method. To quote the javadocs:
Shuts down all running Hazelcast Instances on this JVM, including the default one if it is running. It doesn't shutdown all members of the cluster but just the ones running on this JVM.
If you aren't shutting it down I suspect there are non-daemon threads there that aren't being terminated that will keep the JVM from closing.
In Tomcat, I added a lifecycle listener via server.xml.
<Listener className="com.mycompany.StartupHandler" DEBUG="false"/>
My StartupHandler.class went in a jar in $TOMCAT_HOME/lib/, and it contains this snippet to trigger Hazelcast shutdown when Tomcat shutdown is detected:
public void lifecycleEvent(LifecycleEvent lifecycleEvent) {
String eventType = lifecycleEvent.getType();
if (eventType.equals(Lifecycle.BEFORE_STOP_EVENT)) {
com.hazelcast.core.Hazelcast.shutdownAll();
}
}
Other web servers should have similar shutdown hooks where you can invoke Hazelcast.shutdownAll() automatically.
See https://github.com/hazelcast/hazelcast/issues/718 for my original Hazelcast thread on the same topic. Thanks for the hint here to call Hazelcast.shutdownAll(). That idea combined with my Tomcat shutdown hook should be sufficient to resolve this problem for me, and hopefully you find it useful too.
Here is a Tomcat independent way to detect JVM shutdown. It uses a non-daemon thread to poll a daemon thread to detect when the JVM is shutting down (the JVM shuts down daemon threads automatically). Poll the static IS_SHUTDOWN flag in your code, or inline the call to Hazelcast.shutdownAll() here. Be careful of a race condition, because if you try to shutdown Hazelcast before it starts then it throws an exception in your log. The exception does not break anything, but it looks ugly.
/**
* Non-Daemon thread monitors a sacrificial, low-priority, daemon thread
* to detect when JVM is shutting down, so shutdown hooks can be invoked.
* #author justin.cranford
*/
public class ThreadUtil {
public static boolean IS_SHUTDOWN = false; // threads can poll this flag, or inline below where flag is set
private static final Thread DAEMON_THREAD = new Thread() {
public void run() {
while (true) {
try {
Thread.sleep(Long.MAX_VALUE); // sleep forever
} catch(Exception e) {}
}
}
};
private static final Thread NONDAEMON_THREAD = new Thread() {
public void run() {
while (true) {
if (!DAEMON_THREAD.isAlive()) { // poll forever, until daemon thread dies
ThreadUtil.IS_SHUTDOWN = true;
return;
}
try {
Thread.sleep(1000); // poll every 1000msec = 1sec
} catch(Exception e) {}
}
}
};
static {
DAEMON_THREAD.setName("ShutdownMonitorNonDaemonThread");
DAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
DAEMON_THREAD.setDaemon(true);
DAEMON_THREAD.start();
try {
Thread.sleep(3000); // wait 3000msec = 3sec before monitoring
} catch(Exception e) {}
NONDAEMON_THREAD.setName("ShutdownMonitorDaemonThread");
NONDAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
NONDAEMON_THREAD.setDaemon(false);
NONDAEMON_THREAD.start();
}
}
As everyone said, hazelcastInstance.shutdownAll() is the solution.
But, I would to get a new feature in Hazelcast - provide a demon client as well. There are many use cases where cache instance needs to be shutdown as soon as application ends.
Related
I have a web app which runs an interface for controlling an irrigation system. The system starts up to 10 or so threads, using a ScheduledThreadPoolExecutor.
I have found that when I close down the server, many of these threads continue to run. From the Tomcat catalina.out log:
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [GH2] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.
The classes that start these threads are dotted around my app. In order to try to have some control over stopping them, I have a single ScheduledThreadPoolExecutor created by a ServletContextListener and passed into each of the classes that might start a task.
I have tried setting properties in the ScheduledThreadPoolExecutor that I assumed would close down all threads when contextDestroyed() is called in the listener:
public void contextInitialized(ServletContextEvent event) {
executor = (ScheduledThreadPoolExecutor) Executors.newScheduledThreadPool(10);
executor.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);
executor.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
executor.setRemoveOnCancelPolicy(true);
event.getServletContext().setAttribute("executor", executor);
Here is a typical bit of code that schedules a thread:
public WeatherScheduler(ScheduledThreadPoolExecutor executor) {
this.executor = executor;
checkEveryHour();
}
public void checkEveryHour() {
final Runnable weatherChecker = new Runnable() {
public void run() {
if (!Thread.interrupted()) {
if (Date_TimeUtils.isBetween7amAnd8pm()) {
weather = new Weather();
weather.getNewForecast();
IrrigationLogger.getInstance().logEvent("weather forecast: " + weather.toString());
} else
LOGGER.log(Level.INFO, "weatherChecker not called as isBetween7amAnd8pm() == false ");
}
}
};
// delay of a minute to allow for getting internet connection, repeat every hour
executor.scheduleAtFixedRate(weatherChecker, 1, 60, TimeUnit.MINUTES);
}
When the server is closed down, the contextDestroyed() method is called in the listener, and I have
executor.shutdownNow();
included in that method.
But still I get the message in the log file that threads have been started that can't be stopped.
What am I missing?
Thanks.
The way any ThreadExecutor tries to stop a currently running thread is by setting interrupt flag by invoking Thread class method interrupt() on that thread. It is responsibility of the running thread to detect that it was "asked" to finish, so the running thread should clean up and terminate once the signal received. So first in your runner code surround it with try-catch where you will catch InterruptedException and also in your runner code you suppose to invoke method Thread.currentThread().isInterrupted() and if it returns true clean up and finish the thread (same as in your catch statement)
Does ScheduledExecutorService take care of handling terminated thread and generates a new one?
In the example below if any one of my thread terminates due to Error, what happens to thread pool size?
While debugging, I could notice one of the threads created by this service got silently terminated without printing any log statement. On checking Thread dump, I could still see 32 threads were still there and none of them were blocked.
public class CacheManager
{
private static class CacheRefresher extends Thread
{
Cache cache;
public CacheRefresher(Cache cache)
{
this(cache);
}
#Override
public final void run()
{
try {
LOG.info("cache is getting refreshed for " + cache.type);
cache.refreshCache();
} catch (Exception e) {
String subject = "Cache refresh failed in BMW";
LOG.log(Level.WARN, subject + ". Exception thrown:", e);
}
}
}
public void refreshCaches(List<cache> caches)
{
ThreadFactory ourThreadFactory =
new NamedThreadFactory("CacheHandler", true);
ScheduledExecutorService scheduleService =
Executors.newScheduledThreadPool(32, ourThreadFactory);
initialDelay = 60;
for (Cache cache : caches) {
service.scheduleWithFixedDelay(new CacheRefresher(cache), initialDelay, 20, TimeUnit.SECONDS);
initialDelay += 2;
cacheContainers.add(cache);
}
}
}
Uncaught exceptions in scheduled tasks will not cause scheduler's threads to terminate. However, it will prevent the failing task from being re-scheduled. See the respective documentation for ScheduledThreadPoolExecutor.html#scheduleWithFixedDelay:
The sequence of task executions continues indefinitely until one of the following exceptional completions occur:
[...]
An execution of the task throws an exception. In this case calling get on the returned future will throw ExecutionException, holding the exception as its cause.
While the JavaDocs for Executors.newFixedThreadPool explicitly mention this:
If any thread terminates due to a failure during execution prior to shutdown, a new one will take its place if needed to execute subsequent tasks. The threads in the pool will exist until it is explicitly shutdown.
there is no such strong guarantee about the Executors.newScheduledThreadPool.
It is possible that it behaves the same in this regard, but that is an implementation detail you should not need to care about. The Executor service will provide/create enough threads to perform the given tasks.
As found in Javadoc newScheduledThreadPool(int) there always will be the specified amount of threads. Even if a thread will shutdown, there will be started another one. But in the first place, threads within the ScheduledExecutorService should be reused, even when a exception occurs within the Runnable.run().
And sure the threads are not blocked but waiting for new action to do...
Suppose a Java thread performs some I/O operation like reading a file with traditional blocking Java I/O.
The question is: What is the state of the thread while waiting?
I don't know if it is RUNNING (doing some active wait) or WAITING (maybe there is some kind of monitor that wakes up the thread when file data is ready).
How can I find it out?
Thanks.
A thread that is blocked in an I/O syscall should be in RUNNABLE state, according to my reading of the javadocs.
public static final Thread.State RUNNABLE
Thread state for a runnable thread. A thread in the runnable state is executing in the Java virtual machine but it may be waiting for other resources from the operating system such as processor.
The other possible thread states are far more tightly specified, and the respective specifications make them clearly inapplicable.
I have confirmed this by testing (with OpenJDK Java 8 on Linux).
public class Test {
public static void main(String[] args) throws Exception {
Thread t = new Thread(new Runnable() {
public void run() {
try {
while (true) {
System.in.read(); // Block for input
}
} catch (Exception e) {
e.printStackTrace();
}
}
});
t.start();
while (true) {
System.out.println("State: " + t.getState());
Thread.sleep(1000);
}
}
}
Now, hypothetically, if you designed your application so that one thread handed off I/O requests to second thread, then the first thread would be in WAITING (or possibly BLOCKED) state waiting for the I/O thread to deliver. However, normal Java I/O doesn't behave like that. The thread requesting I/O is the one that makes the syscall.
How can I find it out?
Write tests; see above.
Deep-dive the OpenJDK source code.
You can find the status of thread using Thread.getState(). It will return
java.lang.Thread.State
The following piece of code tries to accompolish this.
The code loops forever and checks if there are any pending requests to be processed. If there is any, it creates a new thread to process the request and submits it to the executor. Once all the threads are done,it sleeps for 60 seconds and again checks for pending requests.
public static void main(String a[]){
//variables init code omitted
ExecutorService service = Executors.newFixedThreadPool(15);
ExecutorCompletionService<Long> comp = new ExecutorCompletionService<Long>(service);
while(true){
List<AppRequest> pending = service.findPendingRequests();
int noPending = pending.size();
if (noPending > 0) {
for (AppRequest req : pending) {
Callable<Long> worker = new RequestThread(something, req);
comp.submit(worker);
}
}
for (int i = 0; i < noPending; i++) {
try {
Future<Long> f = comp.take();
long name;
try {
name = f.get();
LOGGER.debug(name + " got completed");
} catch (ExecutionException e) {
LOGGER.error(e.toString());
}
} catch (InterruptedException e) {
LOGGER.error(e.toString());
}
}
TimeUnit.SECONDS.sleep(60);
}
}
My question is most of the processing done by these threads deal with database. And this program will run on a windows machine. What happens to these threads when someone tries to shutdown or logoff the machine.? How to gracefully shutdown the running threads and also the executor.?
A typical orderly shutdown of an ExecutorService might look something like this:
final ExecutorService executor;
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
executor.shutdown();
if (!executor.awaitTermination(SHUTDOWN_TIME)) { //optional *
Logger.log("Executor did not terminate in the specified time."); //optional *
List<Runnable> droppedTasks = executor.shutdownNow(); //optional **
Logger.log("Executor was abruptly shut down. " + droppedTasks.size() + " tasks will not be executed."); //optional **
}
}
});
*You can log that the executor still had tasks to process after waiting the time you were willing to wait.
**You can attempt to force the executor's worker Threads to abandon their current tasks and ensure they don't start any of the remaining ones.
Note that the solution above will work when a user issues an interrupt to your java process or when your ExecutorService only contains daemon threads. If, instead, the ExecutorService contains non-daemon threads that haven't completed, the JVM won't try to shutdown, and therefore the shutdown hooks won't be invoked.
If attempting to shutdown a process as part of a discrete application lifecycle (not a service) then shutdown code should not be placed inside a shutdown hook but at the appropriate location where the program is designed to terminate.
The book "Java Concurrency in Practice" states:
7.4. JVM Shutdown
The JVM can shut down in either an
orderly or abrupt manner. An orderly
shutdown is initiated when the last
"normal" (nondaemon) thread
terminates, someone calls System.exit,
or by other platform-specific means
(such as sending a SIGINT or hitting
Ctrl-C). [...]
7.4.1. Shutdown Hooks
In an orderly shutdown, the JVM first
starts all registered shutdown hooks.
Shutdown hooks are unstarted threads
that are registered with
Runtime.addShutdownHook. The JVM makes
no guarantees on the order in which
shutdown hooks are started. If any
application threads (daemon or
nondaemon) are still running at
shutdown time, they continue to run
concurrently with the shutdown
process. When all shutdown hooks have
completed, the JVM may choose to run
finalizers if runFinalizersOnExit is
true, and then halts. The JVM makes no
attempt to stop or interrupt any
application threads that are still
running at shutdown time; they are
abruptly terminated when the JVM
eventually halts. If the shutdown
hooks or finalizers don't complete,
then the orderly shutdown process
"hangs" and the JVM must be shut down
abruptly. [...]
The important bits are,
"The JVM makes no attempt to stop or interrupt any application threads that are still running at shutdown time; they are abruptly terminated when the JVM eventually halts." so I suppose the connection to the DB will abruptly terminate, if no shutdown hooks are there to do a graceful clean up (if you are using frameworks, they usually do provide such shutdown hooks). In my experience, session to the DB can remain until it is timed out by the DB, etc. when the app. is terminated without such hooks.
Since adding a shutdown hook to explicitly call shutdown() didn't work for me, I found an easy solution in Google's Guava:
com.google.common.util.concurrent.MoreExecutors.getExitingExecutorService.
You can either call shutdown() on the ExecutorService:
Initiates an orderly shutdown in which
previously submitted tasks are
executed, but no new tasks will be
accepted.
or you can call shutdownNow():
Attempts to stop all actively
executing tasks, halts the processing
of waiting tasks, and returns a list
of the tasks that were awaiting
execution.
There are no guarantees beyond
best-effort attempts to stop
processing actively executing tasks.
For example, typical implementations
will cancel via Thread.interrupt(), so
any task that fails to respond to
interrupts may never terminate.
Which one you call depends how badly you want it to stop....
I had similar issue, i use to get error like
o.a.c.loader.WebappClassLoaderBase :: The web application [ROOT] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Bellow code fixed it
private ThreadPoolExecutor executorPool;
#PostConstruct
public void init() {
log.debug("Initializing ThreadPoolExecutor");
executorPool = new ThreadPoolExecutor(1, 3, 1, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(1));
}
#PreDestroy
public void destroy() {
log.debug("Shuting down ThreadPoolExecutor");
executorPool.shutdown();
}
I just read the Trail on RMI from sun at http://java.sun.com/docs/books/tutorial/rmi/implementing.html
When I run the example, the JVM does not terminate although main has finished. Is RMI spawning a Thread somewhere internally?
What is the behaviour of multiple Threads spawned in main, after main exits?
Is it a clean way to let the Threads exit whenever they want or should you do a join on each Thread you spawn? I did not find any documentation on this question.
Thank you very much for your help!!
public class ComputeEngine implements Compute {
public ComputeEngine() {
super();
}
public <T> T executeTask(Task<T> t) {
return t.execute();
}
public static void main(String[] args) {
if (System.getSecurityManager() == null) {
System.setSecurityManager(new SecurityManager());
}
try {
String name = "Compute";
Compute engine = new ComputeEngine();
Compute stub = (Compute) UnicastRemoteObject.exportObject(engine, 0);
Registry registry = LocateRegistry.getRegistry();
registry.rebind(name, stub);
System.out.println("ComputeEngine bound");
} catch (Exception e) {
System.err.println("ComputeEngine exception:");
e.printStackTrace();
}
}
}
A thread is created for listening the socket and reply to requests to your object. One way to stop the JVM is to unbind the server:
Registry.unbind()
and unexport the objects:
UnicastRemoteObject.unexportObject()).
You can use the jstack utility program, included in the JDK to see which threads are running in a java program, and even what line those threads are on.
So if you're program is still running you just run jstack on the pid and it will tell you which threads are still running and what they're doing.
Concerning the "behaviour of multiple threads" part of your question: yes, the JVM will keep on running until all threads finished. The only exception are threads marked as daemon (see Thread.setDaemon()), the JVM will not wait for them.
Yes, when you are exposing objects through RMI it needs a thread to accept incoming requests for these objects. This thread could be a daemon thread which wouldn't stop the JVM from exiting but it isn't for several reasons and as long as there are still active exported objects it hinders the JVM from exiting normally. So you can use unexportObject for all objects or just use System.exit() to end the JVM although this would leave the clients uninformed about the shutdown.