Develop Reference

Proper shutdown of JVM when launching from C++

I'm launching JVM from C++ code via JNI. I have a problem that when just quitting my C++ process it seems some shutdown hooks from JVM are not run, and therefore some temp resources are still being around, that in my particular case prevents launching JVM next time I open a C++ process.
I tried jvm->DestroyJavaVM(), but after all my process windows were closed, I still could see the process running. What's the best wait to ensure that the JVM is shut down properly when launched via JNI?
Thanks!

First of all, jvm->DestroyJavaVM() won't return till all non-daemon jvm threads have stopped, it does nothing but waiting for them to stop, so you should stop them in java.
Secondly, System.exit will cause the whole process to be shut down.
So what you really need is check your java code that which thread is not stopped yet, for example the background message loop thread of the ui framework such as gwt or swing.

The easiest way is to call System.exit via JNI.

How to gracefully shutdown a Java application that is terminated as a result of closing the command line from which it was executed?

There is a an answered question on Best Way to Gracefully Shutdown a Java Command Line Program. A shutdown hook does the job in case when a program was terminated by Ctrl+C.
My question is how to gracefully exit if the command line itself is closed during the execution of a Java program? I tested with shutdown hook but it didn't work in this case. I cannot find out what happens to the virtual machine in this scenario. Is the process and all its threads killed instantly?
What kind of signal does closing command line produce?
So, how this particular problem can be solved?
EDIT: The problem concerns Windows environment.

Logically, SIGHUP (terminal hangup) should be raised.
Actually, I've just checked my guess with a simple shell script. Yes, when a user closes a terminal emulator in which an application was started (and from which it wasn't detached), then the application receives SIGHUP. So set up a SIGHUP handler and react accordingly. A usual behaviour is to terminate an application, but your intents may be different.
Also if your Java application performs any STDIN/STDOUT operations, it should be closed or at least re-cofigured when HUP is received, because an attempt to read/write from non existing terminal would lead to SIGPIPE and/or program block.
For Windows take a look at Catch Windows terminal closing on running process

Edit for windows environment:
I don't have much experience on windows environment but if you want your application to keep running, it's generally deployed as Windows service (it's similar to daemon on Linux). You would typically start/stop/restart service through a utility that lists all services (I think you get to it via control panel -> Administrative Tools -> Services. I would guess that issuing a "stop" via this tool would signal a graceful shutdown. And if you kill the service via the task manager, then it won't be a graceful shutdown.
Is this a Linux based or Windows based environment? In Linux, if you ran the program in background (and exit the shell with 'exit' command), it'll continue running. You can put your application in the background by adding an & at the end. Also, a lot of applications/services run in the background. If you execute a Tomcat startup script with the startup.sh command, it'll continue running in the background even when you quit the terminal you launched it from. On windows too, the concept should be similar.
In terms of closing application, you use kill command on Linux systems. kill command on a process sends a SIGTERM signal to it. Applications can implement code to intercept SIGTERM and shutdown gracefully on detecting a SIGTERM. If the application doesn't handle SIGTERM gracefully, then it won't respond to a SIGTERM / kill. In that case, you need to explicitly give it a SIGKILL (kill -9) to kill it forcefully. In that case, graceful shutdown is not possible.

In Java, there is a special Runtime method for that : addShutdownHook.
This allows you to initialize a thread that the JVM will try to run just before stopping. It is the place to put any cleanup you want to execute even in case of Ctrl-C of closing of parent window. Extract from javadoc : A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt.
The shutdown hook is call even if the program ends normally. In that case, it is cleaner to remove the registered hook before exiting with removeShutdownHook (still a method from Runtime)
EDIT :
In the case of Windows environment, there are no real signals, but special callbacks when system is shutting down. AFAIK, the system hook is correctly called in that case, but I admit I never really tested that. In Windows, processes can be asked to terminate with 2 ways :
PostQuitMessage function posts a WM_QUIT message in process event loop - normally the process should exit, but it can do its cleanup (equivallent of Unix SIG_TERM)
TerminateProcess immediately stops the process and all its threads (equivallent of Unix SIG_KILL)
Console processes can use a ConsoleControlHandler that can intercept Ctrl-C, Ctrl-Break or Ctrl-Close events. First two are generated through keyboard, last is generated when the user closes the console. But normally, the Oracle JVM should use the system hook mechanisme when getting the Ctrl-Close event that is processed the same as a SIGTERM.

How can I debug a hanging java thread?

I have the following problem:
I deploy a web application in Tomcat (Linux) and after shutdown of Tomcat, if I do ps -ef I still can see the java process running.
I believe this happens due to some hanging thread but I don't know how can I track this thread down.
How can I debug this issue?

You can generate 4-5 thread dumps as described below and then analyze them using tools like Samurai.
What you want to check is when a stuck thread or long running transaction happens, all the thread dumps will show a certain thread id is at the same line in your java stack trace. In simpler terms, the transaction is spanning across multiple thread dumps and hence needs more investigation.
Now when you run these through Samurai, it will highlight these in Red colour so you can quickly click on it and get to the lines showing issues.
See an example of this here. Look at the Samurai output image in that link. The Green cells are fine. Red and Grey cells need looking at.
Generating a Thread Dump:
(Linux)
If the JVM is running in a console then simply press Ctrl-\.
If the JVM is running in the background then send it the QUIT signal:
kill -QUIT process_id
There process_id is the process number of the running Java process. The thread dump will be sent to wherever standard out is redirected too.
You can generally get the process numbers of of all running Java processes with the command:
ps axf | grep java

You say your java process still exists, right?
Processes exist as long as they have attached threads, right?
If so, I would go for the following approach:
- run the process with the MBean server attached and managed internally by the JVM
Then connect to the process after you send the QUIT signal and get the thread dump (there should be a JMX for that. See which threads look suspicious to you.
I think you can also use JVisualVM to take thread dumps...

killing a bash process does not kill the process the bash is currently running

The scenario is as follows: I have a java daemon, which is supposed to not terminate. However, in case of an unexpected error, the crashed JVM should be restarted by a script. So I wrote a command which starts a background bash which has a loop starting the JVM (so when the JVM terminates, it will be restarted again).
/bin/bash -c "while true; do java ...; done" &
In order to be able to stop the daemon, I thought of killing this bash background process (by saving it's process id in a file). This works insofar as the background bash doesn't restart the JVM, but still doesn't kill the currently running process - so the bash seems to end it's current command before it checks for a kill command. I would like to have the currently running JVM to be killed, too.
Since I don't want to manage 2 PIDs (one for the background bash and one for the currently running JVM), is there a way of "force kill" which by design stops the current command? (I couldn't find such thing in man kill)?

There are a number of process-management tools built for exactly this purpose: runit, daemontools, upstart... even an entry in the SysV inittab table.
All of these will automate restarting immediately on shutdown, track desired status as opposed to current status (and attempt to signal startup or shutdown as-desired), manage signal delivery, etc.
You can trap signals in bash and trigger events on them, but that only handles the subset which can be trapped (you can't trap a KILL, for instance). The better thing is to use a tool built-to-purpose.
The ProcessManagement page of the wooledge.org wiki (used by irc.freenode.org's #bash channel) has some other concrete suggestions on doing this yourself in bash... though it too suggests runit, daemontools, and their kin as the best-practices approach.

Why not use cron to start your app, and manage only 1 pid, the one belonging to your app? That way you'll always be killing the correct process.
Emphasising a bit, you could create a bash script to manage your app: start|stop|status. On start it will save the java pid to a file. Then you can schedule a cron job to verify the status of the app, and if the pid does not exist, relaunch it.

Isn't this the default behaviour of bash? I thought for example zsh does the opposite and doesn't send a SIGHUP to all child process? Maybe you can try this answer and write a little script and start it with disown?
see this question: Tie the life of a process to the shell that started it
I didn't test it but I need zsh in my webserver because I start it manually and exit my shell with double CTRL-D.

Spawn a process in Java that survives a JVM shutdown

I need to spawn a process in Java (under Linux exclusively) that will continue to run after the JVM has exited. How can I do this?
Basically the Java app should spawn an updater which stops the Java app, updates files and then starts it again.
I'm interested in a hack & slash method to just get it working as well as a better design proposal if you have one :)

If you're spawning the process using java.lang.Process it should "just work" - I don't believe the spawned process will die when the JVM exits. You might find that the Ant libraries make it easier for you to control the spawning though.

It does actually "just work", unless you're trying to be clever.
My wrapped java.lang.Process was trying to capture the script's output, so when the JVM died, the script didn't have anywhere to send output so it just dies. If I don't try to capture the output, or the script doesn't generate any or redirects everything to a file or /dev/null, everything works as it should.

I was having trouble with this and the launched process was getting killed when the JVM shutdown.
Redirecting stdout and stderr to a file fixed the issue. I guess the process was tied to the launched java app as by default it was expecting to pass its output to it.
Here's the code that worked for me (minus exception handling):
ProcessBuilder pb = new ProcessBuilder(cmd);
pb.redirectOutput(logFile);
pb.redirectError(logFile);
Process p = pb.start();

I thought the whole point of Java was that it's fully contained within the JVM. It's kinda hard to run bytecode when there's no runtime.
If you're looking to have a totally separate process you might look into trying to start a second java.exe instance. Although for your application, it might be easier to simply make a synchronized block that stops (but doesn't kill) your app, does the updating, and then re-initializes your app's data.

It won't always "just work". When JVM spawns the child and then shuts down, the child process will also shutdown in some cases. That is expected behaviour of the process. Under WIN32 systems, it just works.
E.g. If WebLogic server was started up by a Java process, and then that process exits, it also sends the shutdown signal to the WebLogic via shutdown hook in JVM, which causes WebLogic to also shutdown.
If it "just works" for you then there is no problem, however if you find yourself in a position that child process also shutsdown with JVM it is worth having a look at the "nohup" command. The process won't respond to SIGTERM signal, but will respond to SIGKILL signal, as well as normal operations.
Update: The way described above is a bit of an overkill. Another way of doing this would be to use "&" on the end of command. This will spawn a new process that is not a child of current java process.
P.S. Sorry for so many updates, I have been learning and trying it from scratch.

>>don't believe the spawned process will die when the JVM exits.
Child process is always dying on my box(SuSE) whenever I kill java. I think, the child process will die if it's dealing with I/O of the parent process(i.e., java)

If you're looking at making an updater on Linux, you're probably barking up the wrong tree. I believe all major linux distros have a package manager built in. You should use the package manager to do your updating. Nothing frustrates me more than programs that try to self-update... (I'm looking at you, Eclipse)