Situation: I have a keep-alive shell script that restarts an application whenever it shuts down. However I do not want it to do this if the application was closed via a SIGTERM or SIGINT (kill, Ctrl+C, etc.) i.e. a shutdown hook. However I have no way of setting the exit code, hence communicating to the keep-alive script, when exiting from a shutdown hook as calling exit is illegal.
From Javadocs for exit:
If this method is invoked after the virtual machine has begun its shutdown sequence then if shutdown hooks are being run this method will block indefinitely. If shutdown hooks have already been run and on-exit finalization has been enabled then this method halts the virtual machine with the given status code if the status is nonzero; otherwise, it blocks indefinitely.
Is this possible?
If the process has been killed by a signal, the $? variable will be set to 128 + signal:
bash$ sleep 3;echo $?
0
bash$ sleep 3;echo $?
^C
130
Here, 130 is 128 + SIGINT.
Grab the PID of the process in a variable and use the wait builtin: if the process has been terminated by a signal, the return code of wait will be 128 + the signal number.
#
# Note: output from shell trimmed
#
# Launch cat in the background, capture the PID
$ cat & PIDTOCHECK=$!
$ echo $PIDTOCHECK
27764
#
# Call wait a first time: the program is halted waiting for input (SIGTTIN)
#
$ wait $PIDTOCHECK ; echo $?
149
#
# Now kill cat, and call wait again
#
$ kill %1
$ wait $PIDTOCHECK ; echo $?
143
Here's what I do:
Runtime.getRuntime().halt(0);
Note that this will exit the program immediately, so you need to do it after the last shutdown hook has finished.
Related
I have been looking for a long time to see if anyone has had an answer to my issue, but it doesn't seem to exist. I recently found I rarely used M1 Mac Mini. Since I had barely used it, I decided to turn it into a functioning server that runs 24/7. The only issue is that sometimes while I'm sleeping and my friends are playing, the server crashes, and there's no way of starting it back up unless I'm awake. So I'm looking for help on how to make a .command file that either A.) It Pings the server every minute, and if it senses it's down, it terminates the current terminal and restarts the start command. B.) Once the server crashes and the terminal ends, it restarts. I prefer to go with option A, but I'll take any help that I get! Thank you so much in advance, everyone!
I tried a script online, and one of them goes like this.
while true
do
cd Desktop
cd server
/Library/Internet_Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java -Xmx7G -Xms7G -jar forgeserver.jar
echo "If you want to completely stop the server process now, press Ctrl+C before
the time is up!"
echo "Rebooting in:"
for i in 5 4 3 2 1
echo "$i..."
sleep 1
done
echo "Rebooting now!"
done
However I am met with this error
Last login: Thu Feb 9 02:37:12 on ttys001
/Users/myname/Desktop/start.command ; exit;
davidking#Davids-Mac-mini ~ % /Users/myname/Desktop/start.command ; exit;
/Users/davidking/Desktop/start.command: line 11: syntax error near unexpected token `echo'
'Users/davidking/Desktop/start.command: line 11: `echo "$i..."
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]
You can try something like that:
while true; do
ping -c 1 -W 1 your_server_IP
if [ $? -eq 0 ]; then
echo "Server is up"
else
echo "Server is down, restarting..."
# Add your server restart command here
/path/to/restart/command
fi
sleep 60
done
The ping -c 1 -W 1 your_server_IP command sends one Echo Request packet to your_server_IP, and waits for a response for 1 second. If a response is received, it means the server is up and running. If no response is received, it means the server is down.
You have to save this file as a .command file and add it to your PATH.
Let me know if it works or you're getting errors! I'll be happy to help :)
I was trying to start the jobserver.It fails and i dont see any logs for ERROR
/usr/share/dse/spark/spark-jobserver
./server_start.sh: line 41: kill: (21556) - No such process
You need to start it with the dse command
dse spark-jobserver start [any_spark_submit_options] //Start the job server
dse spark-jobserver stop //Stop the job server
https://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/spark/sparkJobserverOverview.html
If that doesn't fix it it probably means a false start or improper shutdown has left a spark-jobserver.pid file in the spark-jobserver resource dir. Remove this so that the following code can pass.
if [ -f "$PIDFILE" ] && kill -0 $(cat "$PIDFILE"); then
echo 'Job server is already running'
exit 1
fi
This code is checking to see if that PID file exists and if it does it tries to get info from the process. If the process doesn't exist then that means the pid file is there in error and must be removed (or you lack permission).
When shutting down my program I need to let my akka actors finish what they're doing before shutting down, so I have a shutdown hook:
sys.addShutdownHook(
{
log.info("\n shutting down startWorkScheduler\n")
assignWorkScheduler.cancel()
log.info("\ntelling manager to shutdown gracefully\n")
manager ! Manager.ShutdownGracefully
//spam shutdown messages until we exit
val resendShutDown = system.scheduler.schedule(5 seconds, 2 seconds){
manager ! Manager.ShutdownGracefully
}
system.awaitTermination()
}
and elsewhere I wait for a "all ok, you can shutdown" message:
case Terminated(httpDlrSqlRouter) => {
println("received terminated message from http router")
if(shuttingDown) {
println("httpDlrRouter done, shutting down......")
log.info("httpDlrRouter done, shutting down......")
context.system.shutdown()
java.lang.System.exit(0)
}else{
log.warning("httpDlrRouter terminated, don't know why!")
}
}
When I run the code using sbt run it works as expected: kill or CTRL+c finishes the work, prints "httpDlrRouter done, shutting down......" and exits.
When I build a .jar using sbt assembly and run it using java -jar filename.jar it does the same except actually exiting, it just hangs until i kill it with kill -9
do I need any alternatives to System.exit or some configuration to make it work?
Apparently changing
java.lang.System.exit(0)
to
Runtime.getRuntime().halt(0)
solved the problem
I am writing to get some help regarding starting my java programs via Monit. I have written a start script program.sh. The monit code and the scipt code is given with this posting.
The issue is that I am not able the start and stop the program using the the script file executed via monit. I can monitor the process if I start it using the terminal but I can't start/stop it with monit. The log from monit says "Failed to start"
However, I can start and stop programs like ssh easliy from monit. The monit runs under sudo and I am running the scripts from an account with administrative privileges.
It will very helpful if someone helps me figure this out
Thanks
monitrc file
#++++++++++#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Monit settings
set daemon 10 with start delay 2 # check services at 2-minute intervals
set logfile syslog facility log_daemon
set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# Mail Server
set mailserver smtp.gmail.com port 587
username "monit.abc123#gmail.com" password "password"
using tlsv1 with timeout 30 seconds
set eventqueue
basedir /var/lib/monit/events # set the base directory where events will be stored
slots 100 # optionally limit the queue size
set alert abc123#gmail.com # receive all alerts
set alert abc123#gmail.com only on { timeout } # receive just service-
# # timeout alert
#set alert foo#bar { nonexist, timeout, resource, icmp, connection }
#set alert security#bar on { checksum, permission, uid, gid }
# setup the email for the SMS thing over here.......................
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
set httpd port 2813 and
# use address localhost # only accept connection from localhost
allow localhost # allow localhost to connect to the server and
allow 0.0.0.0/0.0.0.0
allow admin:monit # require user 'admin' with password 'monit'
# allow #monit # allow users of group 'monit' to connect (rw)
# allow #users readonly # allow users of group 'users' to connect readonly
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
set mail-format {
from: monit#$HOST
subject: Monit Alert -- $EVENT $SERVICE
message: $EVENT Service $SERVICE
Date: $DATE
Action: $ACTION
Host: $HOST
Description: $DESCRIPTION
Your faithful employee,
Monit
}
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#***********************************************************************************************
#Computer Resources check
check system myhost.mydomain.tld
if loadavg (1min) > 4 for 5 cycles then alert
if loadavg (5min) > 2 for 5 cycles then alert
if memory usage > 75% for 3 cycles then alert
if swap usage > 25% for 5 cycles then alert
if cpu usage (user) > 70% for 5 cycles then alert
if cpu usage (system) > 70% for 5 cycles then alert
if cpu usage (wait) > 20% for 5 cycles then alert
#***********************************************************************************************
################################################################################################
#Monitoring SSH Service
check process ssh123 with pidfile /var/run/sshd.pid
start program = "/etc/init.d/ssh start"
stop program = "/etc/init.d/ssh stop"
if cpu > 50% for 5 cycles then alert
if totalmem > 200 MB for 5 cycles then alert
if children > 2 then alert
#if loadavg(5min) greater than 10 for 8 cycles then stop
#if 5 restarts within 5 cycles then timeout
################################################################################################
#Monitoring Prorgam in Java
check process javaprg with pidfile /home/user/Desktop/Binaries/javaprg.pid
start program = "/home/user/Desktop/Binaries/javaprg.sh start"
stop program = "/home/user/Desktop/Binaries/javaprg.sh stop"
if cpu > 50% for 5 cycles then alert
if totalmem > 1500 MB for 5 cycles then alert
if children > 2 then alert
#if loadavg(5min) greater than 10 for 8 cycles then stop
#if 5 restarts within 5 cycles then timeout
Start/Stop script
#!/bin/bash
case $1 in
start)
echo $$ > javaprg.pid;
exec /usr/bin/java -jar javaprg.jar
;;
stop)
kill $(cat javaprg.pid);
rm javaprg.pid
;;
*)
echo "usage: javaprg {start|stop}" ;;
esac
exit 0
You should set absolute path in your start stop script.
You can try to launch it using a rooted shell sudo -s.
And you should concider using the /etc/monit/conf.d folder to put your conf files.
I had the same problem when I was trying to configure a shell script under Monit.
what solved the problem was using the /bin/sh prior to the program itself.
try using:
start program = "/home/user/Desktop/Binaries/javaprg.sh start"
stop program = "/home/user/Desktop/Binaries/javaprg.sh stop"
I had the same problem using your script, and it was because the start script did not specify where to save the PID. It was saving javaprg.pid to / rather then the home folder. Change start script to 'echo $$ > /home/usr/binaries/javaprg.pid' and it will work.
I am trying to diagnose a problem where a Java web application I'm using (Jenkins) becomes unresponsive. If I run jstack without the -F flag it doesn't give me anything, but if I put the flag in to force a thread dump not only do I get a result, but the application starts responding and goes on as if nothing had happened until it eventually stops responding again.
What does jstack -F flag do that would affect a running JVM and cause an unresponsive application to start responding again?
You can see the source to jstack here. The -F argument changes how jstack connects to the jvm. With -F (or -m) JStack connects to the jvm using the java debugger interface. If a pid is specified, JStack connects with the SA PID Attaching Connector which says,
The process to be debugged need not have been started in debug
mode(ie, with -agentlib:jdwp or -Xrunjdwp). It is permissable for the
process to be hung.
I don't know why it would cause an unresponsive application to start responding again, but the link above also says,
The process is suspended when this connector attaches and resumed when
this connector detaches.
This may have an effect.
jstack -F -l pid is similarly to (assume working dir is JAVA_HOME)
bin/java -Dsun.jvm.hotspot.debugger.useWindbgDebugger -Dsun.jvm.hotspot.debugger.useProcDebugger -cp lib/sa-jdi.jar;lib/tools.jar sun.tools.jstack.JStack -F -l pid
and in the sun.tools.jstack.JStack code
if (arg.equals("-F")) {
useSA = true;
}
.....
// now execute using the SA JStack tool or the built-in thread dumper
if (useSA) {
// parameters (<pid> or <exe> <core>
...
runJStackTool(mixed, locks, params);
} else {
// pass -l to thread dump operation to get extra lock info
String pid = args[optionCount];
...
runThreadDump(pid, params);
}
and since -F is passed in, runJStackTool is called to load sun.jvm.hotspot.tools.JStack, it have same effect of invoking directly
bin\java -Dsun.jvm.hotspot.debugger.useWindbgDebugger -Dsun.jvm.hotspot.debugger.useProcDebugger -cp lib/sa-jdi.jar;lib/tools.jar sun.jvm.hotspot.tools.JStack pid
and sun.jvm.hotspot.tools.JStack will call sun.jvm.hotspot.bugspot.BugSpotAgent attach -> go ->setupVM method
Maybe below code is the magic
jvmdi = new ServiceabilityAgentJVMDIModule(debugger, saLibNames);
if (jvmdi.canAttach()) {
jvmdi.attach();
jvmdi.setCommandTimeout(6000);
debugPrintln("Attached to Serviceability Agent's JVMDI module.");
// Jog VM to suspended point with JVMDI module
resume();
suspendJava();
suspend();
debugPrintln("Suspended all Java threads.");
}
it will suspend all Java threads in the target process. if your application is hang for thread starvation, the suspend method call may relax them.