Develop Reference

RMI server shuts down on its own

I'm writing a Java client-server application that uses RMI for communication. My problem is that for some reason, the RMI server just shuts down with no exception or error, on its own. I'm using Netbeans and I ran a profile to look at the threads.
You can see in the attached image the point in time where the application supposedly finished executing as the end of the GC Daemon and the RMI Reaper threads. However, even after the application ended, the RMI TCP Accept-1099 thread is still running. The part that confuses me even more is that after the Information message popped up (you can see it in the screenshot) telling me that the server has stopped, the threads continue to be updated in the diagram so I tried to connect with the client again. Although it failed, I can see a new RMI thread being created (connection 18).
I have no idea how to debug this issue, and I can't figure out how it is possible for the application to exit when the RMI accept thread is still running.
Update: Here is the server's main method:
/**
* Main entry point.
*
* #param args the application arguments - unused.
*/
public static void main(String[] args) {
try {
System.setProperty("java.rmi.dgc.leaseValue", "30000");
sServerProperties = new ServerProperties();
System.setProperty("java.rmi.server.hostname", sServerProperties.
getRmiServer());
createRmiRegistry();
ConfigCore configCore = new ConfigCore();
ServerCore server = new ServerCore(configCore);
LoginHandler loginHandler = new LoginHandler(server);
sRegistry.
bind(Login.class.getSimpleName(), loginHandler.getRemote());
Logger.log(Level.INFO, "Server ready!");
} catch (RemoteException ex) {
Logger.log(Level.SEVERE, "Unable to start RMI registry", ex);
} catch (SQLException ex) {
Logger.log(Level.SEVERE, "Unable to connect to the MySQL server",
ex);
System.err.println(ex.getMessage());
} catch (IOException ex) {
Logger.log(Level.SEVERE, "Unable to load or write properties file",
ex);
System.err.println(ex.getMessage());
} catch (AlreadyBoundException ex) {
Logger.log(Level.SEVERE, "RMI port already bounded", ex);
} catch (NoSuchAlgorithmException ex) {
Logger.log(Level.SEVERE, "Unable to digest password", ex);
}
}
/**
* Creates the RMI registry.
*
* #throws RemoteException if the RMI registry could not be created.
*/
private static void createRmiRegistry() throws RemoteException {
if (sRegistry == null) {
Logger.log(Level.INFO, "Creating RMI Registry...");
sRegistry = LocateRegistry.createRegistry(sServerProperties.
getRmiPort());
}
}

You're seeing an interaction of the VM exiting when its last non-daemon thread exits, combined with HotSpot garbage collection behavior, RMI's exporting behavior, and running the JVM under the NetBeans profiler.
The main thread exits after the main() method returns. However, you've exported an RMI server object, so RMI keeps the JVM alive by running the "RMI Reaper" thread (non-daemon) as long as there are live, exported objects. RMI determines whether an exported object is "alive" by keeping only a weak reference to it in it object table.
Unfortunately, from looking at the main() method, it appears that your RMI server object is only referenced via local variables. Thus, it will get garbage collected sooner or later, but for the weak reference to it in RMI's object table. When the object becomes weakly reachable, the RMI Reaper unexports it and exits. Since the RMI Reaper is the last non-daemon thread, the JVM exits.
Note that the RMI registry is treated specially. Exporting a registry will not keep a JVM alive.
Note also that putting an infinite loop at the end of the main() method will not necessarily prevent the RMI server object from being unexported and GC'd. The reason is that objects are subject to GC when they become unreachable, and a reference being present in a local variable of an active method is not sufficient to make it reachable. See my answer to another question on that topic. Of course, putting an infinite loop into main() will prevent the JVM from exiting, since it keeps the main thread alive, and the main thread is not a daemon thread.
To prevent your RMI server from being unexported, it's usually sufficient to store a reference to it in a static field.
Now, why does the JVM stick around when run under the profiler? That's just an artifact of the way the profiler works. The profiler detects that the last non-daemon thread has exited (other than additional threads running in the JVM on behalf of the profiler), so that's when it pops up the dialog that says "The profiled application has finished execution." It keeps the JVM alive, though, so you can continue to get data from it; this has the side effect keeping all the daemon threads alive. You've exported a registry, so that keeps listening on port 1099. When the JVM is in this state you might still be able to register RMI objects if you tried. Your RMI server object has long since been unexported and GC'd, so requests to it won't work. But that's why the RMI connection was still accepted when the JVM was in this state.
Bottom line is, make sure your RMI server objects don't get GC'd. A good way to do this is to make sure they're reachable from a static field.

This is a common problem with RMI. You need to save the reference to your implementation as a class field so it doesn't get GC. And you need to keep that class (with the main()) alive as well.
I use this as a never-ending keep alive:
for(;;) LockSupport.park();
You can do anything you like so the implementation remains alive and the class with the main() stays alive.

Hazelcast prevents the JVM from terminating

We use Hazelcast 2.6.2 in a legacy Java clustered application. When the application is stopped the JVM does not terminate any more. It seems that it is caused by Hazelcast threads not being flagged daemon. I did not find a way way through the Hazelcast API to flag them daemon.
Are there recommended solutions to prevent Hazelcast from preventing the JVM to terminate?
Regards

Looking at the Hazelcast Javadocs, I see that there is a shutdownAll(); method. To quote the javadocs:
Shuts down all running Hazelcast Instances on this JVM, including the default one if it is running. It doesn't shutdown all members of the cluster but just the ones running on this JVM.
If you aren't shutting it down I suspect there are non-daemon threads there that aren't being terminated that will keep the JVM from closing.

In Tomcat, I added a lifecycle listener via server.xml.
<Listener className="com.mycompany.StartupHandler" DEBUG="false"/>
My StartupHandler.class went in a jar in $TOMCAT_HOME/lib/, and it contains this snippet to trigger Hazelcast shutdown when Tomcat shutdown is detected:
public void lifecycleEvent(LifecycleEvent lifecycleEvent) {
String eventType = lifecycleEvent.getType();
if (eventType.equals(Lifecycle.BEFORE_STOP_EVENT)) {
com.hazelcast.core.Hazelcast.shutdownAll();
}
}
Other web servers should have similar shutdown hooks where you can invoke Hazelcast.shutdownAll() automatically.
See https://github.com/hazelcast/hazelcast/issues/718 for my original Hazelcast thread on the same topic. Thanks for the hint here to call Hazelcast.shutdownAll(). That idea combined with my Tomcat shutdown hook should be sufficient to resolve this problem for me, and hopefully you find it useful too.

Here is a Tomcat independent way to detect JVM shutdown. It uses a non-daemon thread to poll a daemon thread to detect when the JVM is shutting down (the JVM shuts down daemon threads automatically). Poll the static IS_SHUTDOWN flag in your code, or inline the call to Hazelcast.shutdownAll() here. Be careful of a race condition, because if you try to shutdown Hazelcast before it starts then it throws an exception in your log. The exception does not break anything, but it looks ugly.
/**
* Non-Daemon thread monitors a sacrificial, low-priority, daemon thread
* to detect when JVM is shutting down, so shutdown hooks can be invoked.
* #author justin.cranford
*/
public class ThreadUtil {
public static boolean IS_SHUTDOWN = false; // threads can poll this flag, or inline below where flag is set
private static final Thread DAEMON_THREAD = new Thread() {
public void run() {
while (true) {
try {
Thread.sleep(Long.MAX_VALUE); // sleep forever
} catch(Exception e) {}
}
}
};
private static final Thread NONDAEMON_THREAD = new Thread() {
public void run() {
while (true) {
if (!DAEMON_THREAD.isAlive()) { // poll forever, until daemon thread dies
ThreadUtil.IS_SHUTDOWN = true;
return;
}
try {
Thread.sleep(1000); // poll every 1000msec = 1sec
} catch(Exception e) {}
}
}
};
static {
DAEMON_THREAD.setName("ShutdownMonitorNonDaemonThread");
DAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
DAEMON_THREAD.setDaemon(true);
DAEMON_THREAD.start();
try {
Thread.sleep(3000); // wait 3000msec = 3sec before monitoring
} catch(Exception e) {}
NONDAEMON_THREAD.setName("ShutdownMonitorDaemonThread");
NONDAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
NONDAEMON_THREAD.setDaemon(false);
NONDAEMON_THREAD.start();
}
}

As everyone said, hazelcastInstance.shutdownAll() is the solution.
But, I would to get a new feature in Hazelcast - provide a demon client as well. There are many use cases where cache instance needs to be shutdown as soon as application ends.

How to gracefully handle the SIGKILL signal in Java

How do you handle clean up when the program receives a kill signal?
For instance, there is an application I connect to that wants any third party app (my app) to send a finish command when logging out. What is the best say to send that finish command when my app has been destroyed with a kill -9?
edit 1: kill -9 cannot be captured. Thank you guys for correcting me.
edit 2: I guess this case would be when the one calls just kill which is the same as ctrl-c

It is impossible for any program, in any language, to handle a SIGKILL. This is so it is always possible to terminate a program, even if the program is buggy or malicious. But SIGKILL is not the only means for terminating a program. The other is to use a SIGTERM. Programs can handle that signal. The program should handle the signal by doing a controlled, but rapid, shutdown. When a computer shuts down, the final stage of the shutdown process sends every remaining process a SIGTERM, gives those processes a few seconds grace, then sends them a SIGKILL.
The way to handle this for anything other than kill -9 would be to register a shutdown hook. If you can use (SIGTERM) kill -15 the shutdown hook will work. (SIGINT) kill -2 DOES cause the program to gracefully exit and run the shutdown hooks.
Registers a new virtual-machine shutdown hook.
The Java virtual machine shuts down in response to two kinds of events:
The program exits normally, when the last non-daemon thread exits or when the exit (equivalently, System.exit) method is invoked, or
The virtual machine is terminated in response to a user interrupt, such as typing ^C, or a system-wide event, such as user logoff or system shutdown.
I tried the following test program on OSX 10.6.3 and on kill -9 it did NOT run the shutdown hook, as expected. On a kill -15 it DOES run the shutdown hook every time.
public class TestShutdownHook
{
public static void main(String[] args) throws InterruptedException
{
Runtime.getRuntime().addShutdownHook(new Thread()
{
#Override
public void run()
{
System.out.println("Shutdown hook ran!");
}
});
while (true)
{
Thread.sleep(1000);
}
}
}
There isn't any way to really gracefully handle a kill -9 in any program.
In rare circumstances the virtual
machine may abort, that is, stop
running without shutting down cleanly.
This occurs when the virtual machine
is terminated externally, for example
with the SIGKILL signal on Unix or the
TerminateProcess call on Microsoft
Windows.
The only real option to handle a kill -9 is to have another watcher program watch for your main program to go away or use a wrapper script. You could do with this with a shell script that polled the ps command looking for your program in the list and act accordingly when it disappeared.
#!/usr/bin/env bash
java TestShutdownHook
wait
# notify your other app that you quit
echo "TestShutdownHook quit"

I would expect that the JVM gracefully interrupts (thread.interrupt()) all the running threads created by the application, at least for signals SIGINT (kill -2) and SIGTERM (kill -15).
This way, the signal will be forwarded to them, allowing a gracefully thread cancellation and resource finalization in the standard ways.
But this is not the case (at least in my JVM implementation: Java(TM) SE Runtime Environment (build 1.8.0_25-b17), Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode).
As other users commented, the usage of shutdown hooks seems mandatory.
So, how do I would handle it?
Well first, I do not care about it in all programs, only in those where I want to keep track of user cancellations and unexpected ends. For example, imagine that your java program is a process managed by other. You may want to differentiate whether it has been terminated gracefully (SIGTERM from the manager process) or a shutdown has occurred (in order to relaunch automatically the job on startup).
As a basis, I always make my long-running threads periodically aware of interrupted status and throw an InterruptedException if they interrupted. This enables execution finalization in way controlled by the developer (also producing the same outcome as standard blocking operations). Then, at the top level of the thread stack, InterruptedException is captured and appropriate clean-up performed. These threads are coded to known how to respond to an interruption request. High cohesion design.
So, in these cases, I add a shutdown hook, that does what I think the JVM should do by default: interrupt all the non-daemon threads created by my application that are still running:
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
System.out.println("Interrupting threads");
Set<Thread> runningThreads = Thread.getAllStackTraces().keySet();
for (Thread th : runningThreads) {
if (th != Thread.currentThread()
&& !th.isDaemon()
&& th.getClass().getName().startsWith("org.brutusin")) {
System.out.println("Interrupting '" + th.getClass() + "' termination");
th.interrupt();
}
}
for (Thread th : runningThreads) {
try {
if (th != Thread.currentThread()
&& !th.isDaemon()
&& th.isInterrupted()) {
System.out.println("Waiting '" + th.getName() + "' termination");
th.join();
}
} catch (InterruptedException ex) {
System.out.println("Shutdown interrupted");
}
}
System.out.println("Shutdown finished");
}
});
Complete test application at github: https://github.com/idelvall/kill-test

There are ways to handle your own signals in certain JVMs -- see this article about the HotSpot JVM for example.
By using the Sun internal sun.misc.Signal.handle(Signal, SignalHandler) method call you are also able to register a signal handler, but probably not for signals like INT or TERM as they are used by the JVM.
To be able to handle any signal you would have to jump out of the JVM and into Operating System territory.
What I generally do to (for instance) detect abnormal termination is to launch my JVM inside a Perl script, but have the script wait for the JVM using the waitpid system call.
I am then informed whenever the JVM exits, and why it exited, and can take the necessary action.

You can use Runtime.getRuntime().addShutdownHook(...), but you cannot be guaranteed that it will be called in any case.

Reference https://aws.amazon.com/blogs/containers/graceful-shutdowns-with-ecs/
import sun.misc.Signal;
import sun.misc.SignalHandler;
public class ExampleSignalHandler {
public static void main(String... args) throws InterruptedException {
final long start = System.nanoTime();
Signal.handle(new Signal("TERM"), new SignalHandler() {
public void handle(Signal sig) {
System.out.format("\nProgram execution took %f seconds\n", (System.nanoTime() - start) / 1e9f);
System.exit(0);
}
});
int counter = 0;
while(true) {
System.out.println(counter++);
Thread.sleep(500);
}
}
}

There is one way to react to a kill -9: that is to have a separate process that monitors the process being killed and cleans up after it if necessary. This would probably involve IPC and would be quite a bit of work, and you can still override it by killing both processes at the same time. I assume it will not be worth the trouble in most cases.
Whoever kills a process with -9 should theoretically know what he/she is doing and that it may leave things in an inconsistent state.

About multi threading

How to kill the thread?
.....
How to restart them again in multi threading?

Since your post is tagged "Java," I have a good idea of what you are saying. Let's say you start a thread by doing:
Thread foo = new Thread(someRunnable);
foo.start();
Now that destroy and friends are deprecated, you need a way to kill the thread. Luckily for you, there has always been the concept of "interrupts." Simply change your runnable so that, on interrupt, it exits. Then call the thread's interrupt method.
foo.interrupt();
If you wrote your Runnable to handle this correctly, it will stop whatever it is doing and terminate.

Thread.stop() kills a thread, but you definitely don't want to do this (see the API documentation for an explanation why). Thread.interrupt() sends an asynchronous notification to a thread, so that it can shut itself gracefully.
For a comprehensive text on Java multithreading, I recommend B. Goetz, Java Concurrency in Practice, Addison-Wesley Professional.

The preferred way for a Thread to die is for the execution of the run method to go to completion:
Thread t = new Thread(new Runnable() {
public void run() {
// Do something...
// Thread will end gracefully here.
}
}
Once a thread gracefully dies in the example above, the Thread cannot be restarted. (Trying to call Thread.start on a thread that has already been started will cause an IllegalThreadStateException.)
In that case, one can make another instance of the thread and call start on that.
Probably a good place to get more information on threading would be Lesson: Concurrency from The Java Tutorials.

i wrap my worker threads up in their own class and use a terminated property to kill the thread proc loop.
sorry i dont have a java version to hand right now but you should get the idea from this
http://pastie.org/880516
using System.Threading;
namespace LoaderDemo
{
class ParserThread
{
private bool m_Terminated;
private AutoResetEvent m_Signal;
private string m_FilePath;
...
public ParserThread(AutoResetEvent signal, string filePath)
{
m_Signal = signal;
m_FilePath = filePath;
Thread thrd = new Thread(this.ThreadProc);
thrd.Start();
}
public bool Terminated {
set { m_Terminated = value; }
}
private Guid Parse(ref string s)
{
//parse the string s and return a populated Guid object
Guid g = new Guid();
// do stuff...
return g;
}
private void ThreadProc()
{
TextReader tr = null;
string line = null;
int lines = 0;
try
{
tr = new StreamReader(m_FilePath);
while ((line = tr.ReadLine()) != null)
{
if (m_Terminated) break;
Guid g = Parse(ref line);
m_GuidList.Add(g);
lines++;
}
m_Signal.Set(); //signal done
}
finally
{
tr.Close();
}
}
}
}

The best way to kill a thread is to set up a flag for the thread to watch. Program the thread to exit when it sees the flag is set to true. There's no way to restart a killed thread.

If you want to start, stop, restart threads at will, maybe using the Java 5 concurrency package would be a good idea. You can have an Executor that will do a bit of work, and when you need that bit of work to be done again, you can just re-schedule it to be done in the executor.

Regarding your first query on killing thread:
You can find more details about topic in below SE questions:
How to properly stop the Thread in Java?
How can I kill a thread? without using stop();
How to start/stop/restart a thread in Java?
Regarding your second query of re-starting thread, it's not possible in java.
You can find below details in documentation page
public void start()
Causes this thread to begin execution; the Java Virtual Machine calls the run method of this thread.
The result is that two threads are running concurrently: the current thread (which returns from the call to the start method) and the other thread (which executes its run method).
It is never legal to start a thread more than once. In particular, a thread may not be restarted once it has completed execution.
Instead of plain Threads, you can use advanced concurrent API for thread life cycle management. Have a look at this post for ExecutorService details :
How to properly use Java Executor?

How to stop java process gracefully?

How do I stop a Java process gracefully in Linux and Windows?
When does Runtime.getRuntime().addShutdownHook get called, and when does it not?
What about finalizers, do they help here?
Can I send some sort of signal to a Java process from a shell?
I am looking for preferably portable solutions.

Shutdown hooks execute in all cases where the VM is not forcibly killed. So, if you were to issue a "standard" kill (SIGTERM from a kill command) then they will execute. Similarly, they will execute after calling System.exit(int).
However a hard kill (kill -9 or kill -SIGKILL) then they won't execute. Similarly (and obviously) they won't execute if you pull the power from the computer, drop it into a vat of boiling lava, or beat the CPU into pieces with a sledgehammer. You probably already knew that, though.
Finalizers really should run as well, but it's best not to rely on that for shutdown cleanup, but rather rely on your shutdown hooks to stop things cleanly. And, as always, be careful with deadlocks (I've seen far too many shutdown hooks hang the entire process)!

Ok, after all the possibilities I have chosen to work with "Java Monitoring and Management"
Overview is here
That allows you to control one application from another one in relatively easy way. You can call the controlling application from a script to stop controlled application gracefully before killing it.
Here is the simplified code:
Controlled application:
run it with the folowing VM parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
//ThreadMonitorMBean.java
public interface ThreadMonitorMBean
{
String getName();
void start();
void stop();
boolean isRunning();
}
// ThreadMonitor.java
public class ThreadMonitor implements ThreadMonitorMBean
{
private Thread m_thrd = null;
public ThreadMonitor(Thread thrd)
{
m_thrd = thrd;
}
#Override
public String getName()
{
return "JMX Controlled App";
}
#Override
public void start()
{
// TODO: start application here
System.out.println("remote start called");
}
#Override
public void stop()
{
// TODO: stop application here
System.out.println("remote stop called");
m_thrd.interrupt();
}
public boolean isRunning()
{
return Thread.currentThread().isAlive();
}
public static void main(String[] args)
{
try
{
System.out.println("JMX started");
ThreadMonitorMBean monitor = new ThreadMonitor(Thread.currentThread());
MBeanServer server = ManagementFactory.getPlatformMBeanServer();
ObjectName name = new ObjectName("com.example:type=ThreadMonitor");
server.registerMBean(monitor, name);
while(!Thread.interrupted())
{
// loop until interrupted
System.out.println(".");
try
{
Thread.sleep(1000);
}
catch(InterruptedException ex)
{
Thread.currentThread().interrupt();
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
finally
{
// TODO: some final clean up could be here also
System.out.println("JMX stopped");
}
}
}
Controlling application:
run it with the stop or start as the command line argument
public class ThreadMonitorConsole
{
public static void main(String[] args)
{
try
{
// connecting to JMX
System.out.println("Connect to JMX service.");
JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://:9999/jmxrmi");
JMXConnector jmxc = JMXConnectorFactory.connect(url, null);
MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();
// Construct proxy for the the MBean object
ObjectName mbeanName = new ObjectName("com.example:type=ThreadMonitor");
ThreadMonitorMBean mbeanProxy = JMX.newMBeanProxy(mbsc, mbeanName, ThreadMonitorMBean.class, true);
System.out.println("Connected to: "+mbeanProxy.getName()+", the app is "+(mbeanProxy.isRunning() ? "" : "not ")+"running");
// parse command line arguments
if(args[0].equalsIgnoreCase("start"))
{
System.out.println("Invoke \"start\" method");
mbeanProxy.start();
}
else if(args[0].equalsIgnoreCase("stop"))
{
System.out.println("Invoke \"stop\" method");
mbeanProxy.stop();
}
// clean up and exit
jmxc.close();
System.out.println("Done.");
}
catch(Exception e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
That's it. :-)

An another way: your application can open a server socet and wait for an information arrived to it. For example a string with a "magic" word :) and then react to make shutdown: System.exit(). You can send such information to the socke using an external application like telnet.

Here is a bit tricky, but portable solution:
In your application implement a shutdown hook
When you want to shut down your JVM gracefully, install a Java Agent that calls System.exit() using the Attach API.
I implemented the Java Agent. It is available on Github: https://github.com/everit-org/javaagent-shutdown
Detailed description about the solution is available here: https://everitorg.wordpress.com/2016/06/15/shutting-down-a-jvm-process/

Similar Question Here
Finalizers in Java are bad. They add a lot of overhead to garbage collection. Avoid them whenever possible.
The shutdownHook will only get called when the VM is shutting down. I think it very well may do what you want.

Thanks for you answers. Shutdown hooks seams like something that would work in my case.
But I also bumped into the thing called Monitoring and Management beans:
http://java.sun.com/j2se/1.5.0/docs/guide/management/overview.html
That gives some nice possibilities, for remote monitoring, and manipulation of the java process. (Was introduced in Java 5)

Signalling in Linux can be done with "kill" (man kill for the available signals), you'd need the process ID to do that. (ps ax | grep java) or something like that, or store the process id when the process gets created (this is used in most linux startup files, see /etc/init.d)
Portable signalling can be done by integrating a SocketServer in your java application. It's not that difficult and gives you the freedom to send any command you want.
If you meant finally clauses in stead of finalizers; they do not get extecuted when System.exit() is called.
Finalizers should work, but shouldn't really do anything more significant but print a debug statement. They're dangerous.