I am trying to measure my Tomcat server shutdown interval and write it to the log. I'm trying to use the following Python code:
log_times.append(datetime.now().strftime(TIME_PATTERN)) # logs start time
subprocess.check_call(['service', 'tomcat7', 'stop'])
pid = subprocess.Popen(['pgrep', '-utomcat7'], stdout=subprocess.PIPE).communicate()[0]
while (pid != ''):
log.info('pgrep found tomcat process: %s', pid)
pid = subprocess.Popen(['pgrep', '-utomcat7'], stdout=subprocess.PIPE).communicate()[0]
log_times.append(datetime.now().strftime(TIME_PATTERN)) # logs tomcat shutdown time
Is there a better \ accurate way to do this ?
You can try use a bash script. For example stop_tomcat.sh:
START=$(date +%s.%N)
service tomcat7 stop
END=$(date +%s.%N)
DIFF=$(echo "$END - $START" | bc)
echo $DIFF
Change permissions :
chmod 755
call it from python and take it stdout :
time = subprocess.Popen(['./stop_tomcat.sh'], stdout=subprocess.PIPE).communicate()[0]
More details about measuring time in bash https://unix.stackexchange.com/a/12069.
Related
I was trying to start the jobserver.It fails and i dont see any logs for ERROR
/usr/share/dse/spark/spark-jobserver
./server_start.sh: line 41: kill: (21556) - No such process
You need to start it with the dse command
dse spark-jobserver start [any_spark_submit_options] //Start the job server
dse spark-jobserver stop //Stop the job server
https://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/spark/sparkJobserverOverview.html
If that doesn't fix it it probably means a false start or improper shutdown has left a spark-jobserver.pid file in the spark-jobserver resource dir. Remove this so that the following code can pass.
if [ -f "$PIDFILE" ] && kill -0 $(cat "$PIDFILE"); then
echo 'Job server is already running'
exit 1
fi
This code is checking to see if that PID file exists and if it does it tries to get info from the process. If the process doesn't exist then that means the pid file is there in error and must be removed (or you lack permission).
I have the following Jenkins post-build shell script:
ssh user#my_server <<EOF
service my_service stop
service my_service start
tail -f /opt/services/my_service/logs/current
exit
EOF
This script restarts my_service on a remote host (my_server).
My problem is: command service my_service start just makes a request to RUNIT to run a my_service, i.e service my_service start returns immediately after execution.
But service my_service start runs a SpringBoot java web application that writes all log info into .../logs/current log file. To catch this log info I've added command tail -f /opt/services/my_service/logs/current but in this case Jenkins build is never ends)) such as tail -f command never stops.
Is there a way to execute my post-build script (which only start my web app on a remote server) and grabbing the .../logs/current log file during 2 minutes or until this log has the line "Web app MyApplication has been Started".
I wanna see the content of .../logs/current log file right in Jenkins's Console output and kill tail -f after 2 minutes
tail -f will not end until it gets interrupted, so your script will never finish running.
what you can do is use grep -q on your log, which will exit with 0 exit status when it finds it's pattern:
grep -q 'Web app MyApplication has been Started' <(tail -f /opt/services/my_service/logs/current)
I am trying to write an init script for a java program. The program has no ability to background itself, so I'm using & to do it. It writes its logfiles to stdout.
The problem I have is regarding the RETVAL in the init script. Here's the function I'm using:
JAR_FILE=/opt/application/server/server.jar
JAVA=/usr/bin/java
start() {
echo -n $"Starting ${NAME}: "
daemon --pidfile=${PID_FILE} --user $USER \
$JAVA $JAVA_ARGS >> /var/log/application/server.log &
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch $LOCKFILE
return $RETVAL
}
When I run the init script, the app starts up or fails nicely, the problem is that the retval is being written to the logfile, rather than to stdout:
[root#rpmbuild rpmbuild]# /etc/init.d/application-server start
Starting application-server:
[root#host]# tail /var/log/application/server.log
JAVA APPLICATION ERRORS ARE HERE
[FAILED]
Is there anyway I can make it log to stdout and also ensure that any retval (whether it good or bad) gets echoed to stdout?
There are options on how to direct the output when using the daemon command
-l, --errlog=spec - Send daemon's error output to syslog or file
-b, --dbglog=spec - Send daemon's debug output to syslog or file
-o, --output=spec - Send client's output to syslog or file
-O, --stdout=spec - Send client's stdout to syslog or file
-E, --stderr=spec - Send client's stderr to syslog or file
I would change the output using the -o option and not redirect. You should not have to background the task either
daemon -o /var/log/application/server.log -name SuperJavaProgram \
--pidfile=${PID_FILE} --user $USER $JAVA $JAVA_ARGS
Then you could test if it's running with daemon --running -n SuperJavaProgram
Maybe, you could print to stderr in java, to separate standard logs and errors. Your >> will redirect only stdout.
BTW, it's surprising to test $? whereas your program as not executed yet, because background-ed.
I am trying to diagnose a problem where a Java web application I'm using (Jenkins) becomes unresponsive. If I run jstack without the -F flag it doesn't give me anything, but if I put the flag in to force a thread dump not only do I get a result, but the application starts responding and goes on as if nothing had happened until it eventually stops responding again.
What does jstack -F flag do that would affect a running JVM and cause an unresponsive application to start responding again?
You can see the source to jstack here. The -F argument changes how jstack connects to the jvm. With -F (or -m) JStack connects to the jvm using the java debugger interface. If a pid is specified, JStack connects with the SA PID Attaching Connector which says,
The process to be debugged need not have been started in debug
mode(ie, with -agentlib:jdwp or -Xrunjdwp). It is permissable for the
process to be hung.
I don't know why it would cause an unresponsive application to start responding again, but the link above also says,
The process is suspended when this connector attaches and resumed when
this connector detaches.
This may have an effect.
jstack -F -l pid is similarly to (assume working dir is JAVA_HOME)
bin/java -Dsun.jvm.hotspot.debugger.useWindbgDebugger -Dsun.jvm.hotspot.debugger.useProcDebugger -cp lib/sa-jdi.jar;lib/tools.jar sun.tools.jstack.JStack -F -l pid
and in the sun.tools.jstack.JStack code
if (arg.equals("-F")) {
useSA = true;
}
.....
// now execute using the SA JStack tool or the built-in thread dumper
if (useSA) {
// parameters (<pid> or <exe> <core>
...
runJStackTool(mixed, locks, params);
} else {
// pass -l to thread dump operation to get extra lock info
String pid = args[optionCount];
...
runThreadDump(pid, params);
}
and since -F is passed in, runJStackTool is called to load sun.jvm.hotspot.tools.JStack, it have same effect of invoking directly
bin\java -Dsun.jvm.hotspot.debugger.useWindbgDebugger -Dsun.jvm.hotspot.debugger.useProcDebugger -cp lib/sa-jdi.jar;lib/tools.jar sun.jvm.hotspot.tools.JStack pid
and sun.jvm.hotspot.tools.JStack will call sun.jvm.hotspot.bugspot.BugSpotAgent attach -> go ->setupVM method
Maybe below code is the magic
jvmdi = new ServiceabilityAgentJVMDIModule(debugger, saLibNames);
if (jvmdi.canAttach()) {
jvmdi.attach();
jvmdi.setCommandTimeout(6000);
debugPrintln("Attached to Serviceability Agent's JVMDI module.");
// Jog VM to suspended point with JVMDI module
resume();
suspendJava();
suspend();
debugPrintln("Suspended all Java threads.");
}
it will suspend all Java threads in the target process. if your application is hang for thread starvation, the suspend method call may relax them.
I have set up an automated deployment script (in shell script) for my web application.
It uses java, tomcat, maven and a postgres database.
The deployment script does this:
builds the deployable application from source repository
stops tomcat
applies database migration patches
deploys the war files in tomcat
starts tomcat (by invoking $TOMCAT_HOME/bin/startup.sh)
exits with a success message
It's all working and it's pretty neat - but it needs a little improvement.
You see, even though it exits with a success message, sometimes the deploy was not successful because the web application did not start correctly.
I would like to refactor steps 5 and 6 so that after bring up the tomcat server, the deployment script would "tail -f" in the catalina.out file, looking either for a "server started successfully" message or an exception stack trace.
The tail -f output up to that point should be part of the output of the deployment script, and step 6 would "exit 0" or "exit 1" accordingly.
I know that should be possible, if not in shell script, maybe with python.
The problem is I'm a java specialist - and by specialist I mean I suck at everything else :-)
Help please? :-)
Maybe something like this?
tmp=$(mktemp -t catalina.XXXXXXX) || exit 136
trap 'rm "$tmp"' 0
trap 'exit 255' 2 15
tail -n 200 catalina.out >"$tmp"
if grep -q error "$tmp"; then
cat "$tmp"
exit 1
fi
exit 0
On the other hand, if startup.sh were competently coded, you could just
if startup.sh; then
tail -f catalina.out
else
exit $?
fi
which can be shortened to
startup.sh || exit $?
tail -f catalina.out
As an alternative, you might want to take a look at the Apache Tomcat Manager application. It supports, amongst other things:
Deploying applications remotely, and from local paths
Listing currently deployed applications
Reloading existing applications
Starting an existing application
Stopping an existing application
Undeploying an existing application
The manager provides a web interface that can be called via curl, and which returns simple, parseable messages to indicate the status of the invoked command. Management functions can also be invoked via JMX, or Ant scripts. All in all, a very handy tool.
I ended up implementing a solution using Python's subprocess.Popen, as suggested by #snies.
Here's what it looks like:
waitForIt.py
#! /usr/bin/env python
import subprocess
import sys
def main(argv):
filename = argv[1]
match=argv[2]
p = subprocess.Popen(['tail', '-n', '0', '-f', filename], stdout=subprocess.PIPE)
while True :
line = p.stdout.readline()
print line ,
if match in line :
break
p.terminate()
if __name__ == "__main__":
main(sys.argv)
tailUntil.sh
#!/bin/bash
set -e
filename=$1
match=$2
thisdir=$(dirname $0)
python $thisdir/waitForIt.py "$filename" "$match"
and then
startTomcat.sh
${TOMCAT_HOME}/bin/startup.sh
logDeploy.sh "Agora vamos dar um tail no catalina.out..."
util_tailUntil.sh "$TOMCAT_HOME/logs/catalina.out" 'INFO: Server startup in '
It doesn't do what I originally intended (it still exits with return code 0 even when there is a stacktrace - but that could be changed with a little bit more of Python magic),
but all of tomcat's initialization log is part of the automated deploy out (and easily viewable on Jenkins' deploy job) -
so that's good enough.