I have a problem with tomcat. Every once in a while Tomcat get's killed by a oom...
So i tested it with stressapptest to force a oom and then trying to restart tomcat.
I wrote a simple bash script, that will restart the tomcat service.
This is the Script:
#!/bin/bash
if [ "$(systemctl is-active tomcat9)" == "failed" ] || [ "$(systemctl is-active tomcat9)" == "inactive" ]; then
echo "Restarting tomcat!"
systemctl restart tomcat9.service
exit
else
exit
fi
And inside the setenv.sh i wrote this option.
-XX:OnOutOfMemoryError='/root/restart.sh'
This is how the setenv.sh script looks like:
#!/bin/sh
JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom -server -Xms1536m -Xmx1536m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:OnOutOfMemoryError='pkill java;/root/restart.sh' -XX:+DisableExplicitGC -DecadiaConsoleLogLevel=off"
But I don't know, why the Option "XX:OnOutOfMemoryError" is not working...
Can someone help me with this problem?
Thanks in advance!
You are running Tomcat via systemd. That has two consequences:
Likely the startup command for Tomcat is coming from the systemd configuration instead of the script you edited. So you need to look at other files to make an effective change.
All you need to tell the JVM is to die on OutOfMemoryErrors. Then tell systemd to automatically restart the service. Your script is not required at all.
Related
For the past 2 weeks I've been busy trying to figure out how to setup my minecraft server onto my freenas server.
I was able to get it up and running stably when going into the jail manually typing in my startup command:
cd /root/Minecraft_Server
java -Xmx4096M -Xms4096M -jar forge-1.12.2-14.23.4.2757-universal.jar
And then just close the shell.
I tried looking to automate this command and put it into and sh file in crontab and everything, that didn't work so i decided to upgrade to 11.2 to see if that has any solutions.
Now the main problem already is that if I try to run my command manually in the shell, and i leave the webui, it will just close the server down to unlike in the 11.1 freenas.
Does anyone have any more ideas here?
In the same location as the server I have a minecraft.sh script with this command.
If I manually run the script it works, but if I use crontab it won't start it either.
The corntab command that i've used is:
#reboot /root/Minecraft_Server/minecraft.sh
I also tried putting in the command directly but this also was useless.
I even tried the exec.poststart but when i direct it to /root/minecraft_Server/minecraft.sh it won't start either, it won't even run the jail anymore
use “screen java ...”
on relog to shell do screen -x to get on the server shell
You can configure your Java command as a service that starts whenever the jail starts. That way, the Java server doesn’t depend on the shell or webui.
Basically, create a usr/local/etc/rc.d/minecraftd file that includes the following script:
#!/bin/sh
#
# PROVIDE: minecraftd
# REQUIRE: LOGIN DAEMON NETWORKING mountcritlocal
# KEYWORD: shutdown
#
# Use the following variables to configure the minecraft server. For example, to
# configure the ON/OFF knob variable:
# sysrc minecraftd_enable="YES"
#
# minecraftd_enable="YES"
# minecraftd_user_dir="/root/minecraft"
# minecraftd_jar_path="/root/minecraft/server.jar"
# minecraftd_java_opts="-Xms512M -Xmx1024M"
. /etc/rc.subr
name=minecraftd
rcvar=`set_rcvar`
pidfile=/var/run/minecraftd.pid
load_rc_config $name
start_cmd="${name}_start"
: ${minecraftd_enable="NO"}
: ${minecraftd_user_dir="/root/minecraft"}
: ${minecraftd_jar_path="/root/minecraft/server.jar"}
: ${minecraftd_java_opts="-Xms512M -Xmx1024M"}
minecraftd_start() {
if [ -e $pidfile ]; then
echo "$name already running."
else
echo "Starting $name..."
/usr/sbin/daemon -f -p $pidfile \
/usr/local/bin/java -Duser.dir=$minecraftd_user_dir \
$minecraftd_java_opts \
-jar $minecraftd_jar_path nogui
echo "$name started."
fi
}
run_rc_command $1
Then configure the service to start on boot:
sysrc minecraftd_enable="YES"
And restart your jail.
For more information, check Installing a Minecraft server on FreeNAS.
Disclaimer: My team published that article.
Hope this helps somebody.
I have written a Java server application that runs on a standard virtual hosted Linux solution. The application runs all the time listening for socket connections and creating new handlers for them. It is a server side implementation to a client-server application.
The way I start it is by including it in the start up rc.local script of the server. However once started I do not know how to access it to stop it and if I want to install an update, so I have to restart the server in order to restart the application.
On a windows PC, for this type of application I might create a windows service and then I can stop and start it as I want. Is there anything like that on a Linux box so that if I start this application I can stop it and restart it without doing a complete restart of the server.
My application is called WebServer.exe. It is started on server startup by including it in my rc.local as such:
java -jar /var/www/vhosts/myweb.com/phpserv/WebServer.jar &
I am a bit of a noob at Linux so any example would be appreciated with any posts. However I do have SSH, and full FTP access to the box to install any updates as well as access to a Plesk panel.
I wrote another simple wrapper here:
#!/bin/sh
SERVICE_NAME=MyService
PATH_TO_JAR=/usr/local/MyProject/MyJar.jar
PID_PATH_NAME=/tmp/MyService-pid
case $1 in
start)
echo "Starting $SERVICE_NAME ..."
if [ ! -f $PID_PATH_NAME ]; then
nohup java -jar $PATH_TO_JAR /tmp 2>> /dev/null >> /dev/null &
echo $! > $PID_PATH_NAME
echo "$SERVICE_NAME started ..."
else
echo "$SERVICE_NAME is already running ..."
fi
;;
stop)
if [ -f $PID_PATH_NAME ]; then
PID=$(cat $PID_PATH_NAME);
echo "$SERVICE_NAME stoping ..."
kill $PID;
echo "$SERVICE_NAME stopped ..."
rm $PID_PATH_NAME
else
echo "$SERVICE_NAME is not running ..."
fi
;;
restart)
if [ -f $PID_PATH_NAME ]; then
PID=$(cat $PID_PATH_NAME);
echo "$SERVICE_NAME stopping ...";
kill $PID;
echo "$SERVICE_NAME stopped ...";
rm $PID_PATH_NAME
echo "$SERVICE_NAME starting ..."
nohup java -jar $PATH_TO_JAR /tmp 2>> /dev/null >> /dev/null &
echo $! > $PID_PATH_NAME
echo "$SERVICE_NAME started ..."
else
echo "$SERVICE_NAME is not running ..."
fi
;;
esac
You can follow a full tutorial for init.d here and for systemd (ubuntu 16+) here
If you need the output log replace the 2
nohup java -jar $PATH_TO_JAR /tmp 2>> /dev/null >> /dev/null &
lines for
nohup java -jar $PATH_TO_JAR >> myService.out 2>&1&
A simple solution is to create a script start.sh that runs Java through nohup and then stores the PID to a file:
nohup java -jar myapplication.jar > log.txt 2> errors.txt < /dev/null &
PID=$!
echo $PID > pid.txt
Then your stop script stop.sh would read the PID from the file and kill the application:
PID=$(cat pid.txt)
kill $PID
Of course I've left out some details, like checking whether the process exists and removing pid.txt if you're done.
Linux service init script are stored into /etc/init.d. You can copy and customize /etc/init.d/skeleton file, and then call
service [yourservice] start|stop|restart
see http://www.ralfebert.de/blog/java/debian_daemon/. Its for Debian (so, Ubuntu as well) but fit more distribution.
Maybe not the best dev-ops solution, but good for the general use of a server for a lan party or similar.
Use screen to run your server in and then detach before logging out, this will keep the process running, you can then re-attach at any point.
Workflow:
Start a screen: screen
Start your server: java -jar minecraft-server.jar
Detach by pressing: Ctl-a, d
Re-attach: screen -r
More info here: https://www.gnu.org/software/screen/manual/screen.html
Another alternative, which is also quite popular is the Java Service Wrapper. This is also quite popular around the OSS community.
Referring to Spring Boot application as a Service as well, I would go for the systemd version, since it's the easiest, least verbose, and best integrated into modern distros (and even the not-so-modern ones like CentOS 7.x).
The easiest way is to use supervisord. Please see full details here: http://supervisord.org/
More info:
https://askubuntu.com/questions/779830/running-an-executable-jar-file-when-the-system-starts/852485#852485
https://www.digitalocean.com/community/tutorials/how-to-install-and-manage-supervisor-on-ubuntu-and-debian-vps
Here is a sample shell script (make sure you replace the MATH name with the name of the your application):
#!/bin/bash
### BEGIN INIT INFO
# Provides: MATH
# Required-Start: $java
# Required-Stop: $java
# Short-Description: Start and stop MATH service.
# Description: -
# Date-Creation: -
# Date-Last-Modification: -
# Author: -
### END INIT INFO
# Variables
PGREP=/usr/bin/pgrep
JAVA=/usr/bin/java
ZERO=0
# Start the MATH
start() {
echo "Starting MATH..."
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
echo "The service is already running"
else
#Run the jar file MATH service
$JAVA -jar /opt/MATH/MATH.jar > /dev/null 2>&1 &
#sleep time before the service verification
sleep 10
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
echo "Service was successfully started"
else
echo "Failed to start service"
fi
fi
echo
}
# Stop the MATH
stop() {
echo "Stopping MATH..."
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
#Kill the pid of java with the service name
kill -9 $($PGREP -f MATH)
#Sleep time before the service verification
sleep 10
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
echo "Failed to stop service"
else
echo "Service was successfully stopped"
fi
else
echo "The service is already stopped"
fi
echo
}
# Verify the status of MATH
status() {
echo "Checking status of MATH..."
#Verify if the service is running
$PGREP -f MATH > /dev/null
VERIFIER=$?
if [ $ZERO = $VERIFIER ]
then
echo "Service is running"
else
echo "Service is stopped"
fi
echo
}
# Main logic
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status
;;
restart|reload)
stop
start
;;
*)
echo $"Usage: $0 {start|stop|status|restart|reload}"
exit 1
esac
exit 0
From Spring Boot application as a Service, I can recommend the Python-based supervisord application. See that stack overflow question for more information. It's really straightforward to set up.
Other answers do a good job giving custom scripts and setups depending on your platform. In addition to those, here are the mature, special purpose programs that I know of:
JSW from TanukiSoftware
YAJSW is an open source clone from the above. It is written in Java, and it is a nanny process that manages the child process (your code) according to configurations. Works on windows / linux.
JSVC is a native application. Its also a nanny process, but it invokes your child application through the JNI, rather than as a subprocess.
You can use Thrift server or JMX to communicate with your Java service.
From Spring Boot Reference Guide
Installation as an init.d service (System V)
Simply symlink the jar to init.d to support the standard start, stop, restart and status commands.
Assuming that you have a Spring Boot application installed in /var/myapp, to install a Spring Boot application as an init.d service simply create a symlink:
$ sudo ln -s /var/myapp/myapp.jar /etc/init.d/myapp
Once installed, you can start and stop the service in the usual way. For example, on a Debian based system:
$ service myapp start
If your application fails to start, check the log file written to /var/log/<appname>.log for errors.
Continue reading to know how to secure a deployed service.
After doing as written I've discovered that my service fails to start with this error message in logs: start-stop-daemon: unrecognized option --no-close. And I've managed to fix it by creating a config file /var/myapp/myapp.conf with the following content
USE_START_STOP_DAEMON=false
It is possible to run the war as a Linux service, and you may want to force in your pom.xml file before packaging, as some distros may not recognize in auto mode. To do it, add the following property inside of spring-boot-maven-plugin plugin.
<embeddedLaunchScriptProperties>
<mode>service</mode>
</embeddedLaunchScriptProperties>
Next, setup your init.d with:
ln -s myapp.war /etc/init.d/myapp
and you will be able to run
service myapp start|stop|restart
There are many other options that you can find in Spring Boot documentation, including Windows service.
Im having Netty java application and I want to run it as a service with systemd. Unfortunately application stops no matter of what Type I'm using. At the end I've wrapped java start in screen. Here are the config files:
service
[Unit]
Description=Netty service
After=network.target
[Service]
User=user
Type=forking
WorkingDirectory=/home/user/app
ExecStart=/home/user/app/start.sh
TimeoutStopSec=10
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
start
#!/bin/sh
/usr/bin/screen -L -dmS netty_app java -cp app.jar classPath
from that point you can use systemctl [start|stop|status] service.
To run Java code as daemon (service) you can write JNI based stub.
http://jnicookbook.owsiak.org/recipe-no-022/
for a sample code that is based on JNI. In this case you daemonize the code that was started as Java and main loop is executed in C. But it is also possible to put main, daemon's, service loop inside Java.
https://github.com/mkowsiak/jnicookbook/tree/master/recipes/recipeNo029
Have fun with JNI!
However once started I don't know how to access it to stop it
You can write a simple stop script that greps for your java process, extracts the PID and calls kill on it. It's not fancy, but it's straight forward.
Something like that may be of help as a start:
#!/bin/bash
PID = ps ax | grep "name of your app" | cut -d ' ' -f 1
kill $PID
I connect to the linux server via putty SSH. I tried to run it as a background process like this:
$ node server.js &
However, after 2.5 hrs the terminal becomes inactive and the process dies. Is there anyway I can keep the process alive even with the terminal disconnected?
Edit 1
Actually, I tried nohup, but as soon as I close the Putty SSH terminal or unplug my internet, the server process stops right away.
Is there anything I have to do in Putty?
Edit 2 (on Feb, 2012)
There is a node.js module, forever. It will run node.js server as daemon service.
nohup node server.js > /dev/null 2>&1 &
nohup means: Do not terminate this process even when the stty is cut
off.
> /dev/null means: stdout goes to /dev/null (which is a dummy
device that does not record any output).
2>&1 means: stderr also goes to the stdout (which is already redirected to /dev/null). You may replace &1 with a file path to keep a log of errors, e.g.: 2>/tmp/myLog
& at the end means: run this command as a background task.
Simple solution (if you are not interested in coming back to the process, just want it to keep running):
nohup node server.js &
There's also the jobs command to see an indexed list of those backgrounded processes. And you can kill a backgrounded process by running kill %1 or kill %2 with the number being the index of the process.
Powerful solution (allows you to reconnect to the process if it is interactive):
screen
You can then detach by pressing Ctrl+a+d and then attach back by running screen -r
Also consider the newer alternative to screen, tmux.
You really should try to use screen. It is a bit more complicated than just doing nohup long_running &, but understanding screen once you never come back again.
Start your screen session at first:
user#host:~$ screen
Run anything you want:
wget http://mirror.yandex.ru/centos/4.6/isos/i386/CentOS-4.6-i386-binDVD.iso
Press ctrl+A and then d. Done. Your session keeps going on in background.
You can list all sessions by screen -ls, and attach to some by screen -r 20673.pts-0.srv command, where 0673.pts-0.srv is an entry list.
This is an old question, but is high ranked on Google. I almost can't believe on the highest voted answers, because running a node.js process inside a screen session, with the & or even with the nohup flag -- all of them -- are just workarounds.
Specially the screen/tmux solution, which should really be considered an amateur solution. Screen and Tmux are not meant to keep processes running, but for multiplexing terminal sessions. It's fine, when you are running a script on your server and want to disconnect. But for a node.js server your don't want your process to be attached to a terminal session. This is too fragile. To keep things running you need to daemonize the process!
There are plenty of good tools to do that.
PM2: http://pm2.keymetrics.io/
# basic usage
$ npm install pm2 -g
$ pm2 start server.js
# you can even define how many processes you want in cluster mode:
$ pm2 start server.js -i 4
# you can start various processes, with complex startup settings
# using an ecosystem.json file (with env variables, custom args, etc):
$ pm2 start ecosystem.json
One big advantage I see in favor of PM2 is that it can generate the system startup script to make the process persist between restarts:
$ pm2 startup [platform]
Where platform can be ubuntu|centos|redhat|gentoo|systemd|darwin|amazon.
forever.js: https://github.com/foreverjs/forever
# basic usage
$ npm install forever -g
$ forever start app.js
# you can run from a json configuration as well, for
# more complex environments or multi-apps
$ forever start development.json
Init scripts:
I'm not go into detail about how to write a init script, because I'm not an expert in this subject and it'd be too long for this answer, but basically they are simple shell scripts, triggered by OS events. You can read more about this here
Docker:
Just run your server in a Docker container with -d option and, voilá, you have a daemonized node.js server!
Here is a sample Dockerfile (from node.js official guide):
FROM node:argon
# Create app directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
# Install app dependencies
COPY package.json /usr/src/app/
RUN npm install
# Bundle app source
COPY . /usr/src/app
EXPOSE 8080
CMD [ "npm", "start" ]
Then build your image and run your container:
$ docker build -t <your username>/node-web-app .
$ docker run -p 49160:8080 -d <your username>/node-web-app
Always use the proper tool for the job. It'll save you a lot of headaches and over hours!
another solution disown the job
$ nohup node server.js &
[1] 1711
$ disown -h %1
nohup will allow the program to continue even after the terminal dies. I have actually had situations where nohup prevents the SSH session from terminating correctly, so you should redirect input as well:
$ nohup node server.js </dev/null &
Depending on how nohup is configured, you may also need to redirect standard output and standard error to files.
Nohup and screen offer great light solutions to running Node.js in the background. Node.js process manager (PM2) is a handy tool for deployment. Install it with npm globally on your system:
npm install pm2 -g
to run a Node.js app as a daemon:
pm2 start app.js
You can optionally link it to Keymetrics.io a monitoring SAAS made by Unitech.
$ disown node server.js &
It will remove command from active task list and send the command to background
I have this function in my shell rc file, based on #Yoichi's answer:
nohup-template () {
[[ "$1" = "" ]] && echo "Example usage:\nnohup-template urxvtd" && return 0
nohup "$1" > /dev/null 2>&1 &
}
You can use it this way:
nohup-template "command you would execute here"
Have you read about the nohup command?
To run command as a system service on debian with sysv init:
Copy skeleton script and adapt it for your needs, probably all you have to do is to set some variables. Your script will inherit fine defaults from /lib/init/init-d-script, if something does not fits your needs - override it in your script. If something goes wrong you can see details in source /lib/init/init-d-script. Mandatory vars are DAEMON and NAME. Script will use start-stop-daemon to run your command, in START_ARGS you can define additional parameters of start-stop-daemon to use.
cp /etc/init.d/skeleton /etc/init.d/myservice
chmod +x /etc/init.d/myservice
nano /etc/init.d/myservice
/etc/init.d/myservice start
/etc/init.d/myservice stop
That is how I run some python stuff for my wikimedia wiki:
...
DESC="mediawiki articles converter"
DAEMON='/home/mss/pp/bin/nslave'
DAEMON_ARGS='--cachedir /home/mss/cache/'
NAME='nslave'
PIDFILE='/var/run/nslave.pid'
START_ARGS='--background --make-pidfile --remove-pidfile --chuid mss --chdir /home/mss/pp/bin'
export PATH="/home/mss/pp/bin:$PATH"
do_stop_cmd() {
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 \
$STOP_ARGS \
${PIDFILE:+--pidfile ${PIDFILE}} --name $NAME
RETVAL="$?"
[ "$RETVAL" = 2 ] && return 2
rm -f $PIDFILE
return $RETVAL
}
Besides setting vars I had to override do_stop_cmd because of python substitutes the executable, so service did not stop properly.
Apart from cool solutions above I'd mention also about supervisord and monit tools which allow to start process, monitor its presence and start it if it died. With 'monit' you can also run some active checks like check if process responds for http request
For Ubuntu i use this:
(exec PROG_SH &> /dev/null &)
regards
Try this for a simple solution
cmd & exit
I have a java app on my (Ubuntu) server. If I do this, then it starts correctly:
/usr/bin/java -cp /home/jenkins/veta/lily.jar com.sugarapp.lily.Main
but I don't know how to get its PID. I don't know how to stop it with an init.d script.
I have a different app, written in Clojure, and for it I was able to write an init.d script that works great. So I tried to refashion that init.d script for my Java app, and this is what I got:
WORK_DIR="/home/jenkins/veta"
NAME="lily"
JAR="lily.jar"
USER="jenkins"
DAEMON="/usr/bin/java"
DAEMON_ARGS=" -cp /home/jenkins/veta/lily.jar com.sugarapp.lily.Main"
start () {
echo "Starting lily..."
if [ ! -f $WORK_DIR/lily.pid ]; then
/sbin/start-stop-daemon --start --verbose --background --chdir $WORK_DIR --exec $DAEMON --pidfile $WORK_DIR/lily.pid --chuid "$USER" --make-pidfile -- $DAEMON_ARGS
else
echo "lily is already running..."
fi
}
stop () {
echo "Stopping lily..."
/sbin/start-stop-daemon --stop --exec $DAEMON --pidfile $WORK_DIR/lily.pid
rm $WORK_DIR/lily.pid
}
But this doesn't work. Although the PID in $WORK_DIR/lily.pid changes every time I run the script, no process with that PID ever seems to run. If I try:
ps aux | grep java
I don't see this app, nor if I try using the PID.
So is there a way I can use the first command, but somehow capture the PID, so I can store it for later?
I just want a reliable way to stop and start this app. That can be by PID or some other factor. I'm open to suggestions.
UPDATE:
Maybe my question is unclear? Something like jps or ps will give me too many answers. If I do something like "ps aux | grep java" I'll see that there are 5 different java apps running on the server. The start-stop-daemon won't know which PID belongs to this particular app, nor can I figure out what I should feed into my init.d script.
If your system has jdk installed there is an utility called jps which resides in jdk/bin. It will display the list of running java process. Make use of it.
If jdk is not installed in your machine then you have to grep the java process from ps -eaf command.
If you want the pid from the command line, this might work:
myCommand & echo $!
Which I copied from the accepted response to a very similar topic in ServerFault: https://serverfault.com/a/205504
I Have set the heap size of tomcat 7 by adding boot script:
export CATALINA_OPTS="-Xms1024m -Xmx248m"
I change /etc/init.d/tomcat7 :
if [ -z "$JAVA_OPTS" ]; then
JAVA_OPTS="-Djava.awt.headless=true -Xmx2048M -Xms1024M"
fi
I Reboot the computer and restart the Tomcat:
service tomcat7 restart
And verify the $CATALINA_OPTS Works:
> echo $CATALINA_OPTS
-Xms1024m -Xmx2048m
But when I go to the tomcat manager, I note that the heap has not changed.
Free memory: 38.02 MB Total memory: 123.75 MB Max memory: 123.75 MB
Please, i need help about this.
Check the setenv.sh in tomcat/bin, according to manual this should be the right place to put those params.
Another option, it depend on your OS tomcat package, may be that config param are overrided in /etc/conf.d/tomcat/ or /etc/tomcat. Just check your init script and your catalina.sh to find where your settings are overrided.
Btw if you run a ps -ef | grep tomcat you should see the full command line with arguments: this may give you an idea of how init script build the command, and so you can investigate where params are set.
Have you tried creating a setenv.sh script in the $CATALINA_HOME/bin directory with your options in it?
I find that setting JAVA_OPTS="-Xmx2048m -Xms1024m" in there works pretty well.