I am trying to diagnose a problem where a Java web application I'm using (Jenkins) becomes unresponsive. If I run jstack without the -F flag it doesn't give me anything, but if I put the flag in to force a thread dump not only do I get a result, but the application starts responding and goes on as if nothing had happened until it eventually stops responding again.
What does jstack -F flag do that would affect a running JVM and cause an unresponsive application to start responding again?
You can see the source to jstack here. The -F argument changes how jstack connects to the jvm. With -F (or -m) JStack connects to the jvm using the java debugger interface. If a pid is specified, JStack connects with the SA PID Attaching Connector which says,
The process to be debugged need not have been started in debug
mode(ie, with -agentlib:jdwp or -Xrunjdwp). It is permissable for the
process to be hung.
I don't know why it would cause an unresponsive application to start responding again, but the link above also says,
The process is suspended when this connector attaches and resumed when
this connector detaches.
This may have an effect.
jstack -F -l pid is similarly to (assume working dir is JAVA_HOME)
bin/java -Dsun.jvm.hotspot.debugger.useWindbgDebugger -Dsun.jvm.hotspot.debugger.useProcDebugger -cp lib/sa-jdi.jar;lib/tools.jar sun.tools.jstack.JStack -F -l pid
and in the sun.tools.jstack.JStack code
if (arg.equals("-F")) {
useSA = true;
}
.....
// now execute using the SA JStack tool or the built-in thread dumper
if (useSA) {
// parameters (<pid> or <exe> <core>
...
runJStackTool(mixed, locks, params);
} else {
// pass -l to thread dump operation to get extra lock info
String pid = args[optionCount];
...
runThreadDump(pid, params);
}
and since -F is passed in, runJStackTool is called to load sun.jvm.hotspot.tools.JStack, it have same effect of invoking directly
bin\java -Dsun.jvm.hotspot.debugger.useWindbgDebugger -Dsun.jvm.hotspot.debugger.useProcDebugger -cp lib/sa-jdi.jar;lib/tools.jar sun.jvm.hotspot.tools.JStack pid
and sun.jvm.hotspot.tools.JStack will call sun.jvm.hotspot.bugspot.BugSpotAgent attach -> go ->setupVM method
Maybe below code is the magic
jvmdi = new ServiceabilityAgentJVMDIModule(debugger, saLibNames);
if (jvmdi.canAttach()) {
jvmdi.attach();
jvmdi.setCommandTimeout(6000);
debugPrintln("Attached to Serviceability Agent's JVMDI module.");
// Jog VM to suspended point with JVMDI module
resume();
suspendJava();
suspend();
debugPrintln("Suspended all Java threads.");
}
it will suspend all Java threads in the target process. if your application is hang for thread starvation, the suspend method call may relax them.
Related
My java program hang there and i cannot debug it. Here is what i tried
jstack $pid, hang and never response
jstack -F $pid with /proc/sys/kernel/yama/ptrace_scope == 0. Get errors below
Error: -F option used
Cannot connect to core dump or remote debug server. Use jhsdb jstack instead
Then i tried jhsdb jstack --pid $pid. Still hangs forever and i got
Attaching to process ID 123212, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11+28
Deadlock Detection:
After i press CTRL+C. I got:
Attaching to process ID 123212, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11+28
Deadlock Detection:
java.lang.RuntimeException: VM.initialize() was not yet called
at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VM.getVM(VM.java:460)
at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:299)
at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102)
at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:73)
at jdk.hotspot.agent/sun.jvm.hotspot.oops.MetadataField.getValue(MetadataField.java:43)
at jdk.hotspot.agent/sun.jvm.hotspot.oops.MetadataField.getValue(MetadataField.java:40)
at jdk.hotspot.agent/sun.jvm.hotspot.oops.Klass.getNextLinkKlass(Klass.java:131)
at jdk.hotspot.agent/sun.jvm.hotspot.classfile.ClassLoaderData.find(ClassLoaderData.java:96)
at jdk.hotspot.agent/sun.jvm.hotspot.classfile.ClassLoaderDataGraph.find(ClassLoaderDataGraph.java:61)
at jdk.hotspot.agent/sun.jvm.hotspot.memory.SystemDictionary.getAbstractOwnableSynchronizerKlass(SystemDictionary.java:116)
at jdk.hotspot.agent/sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:84)
at jdk.hotspot.agent/sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:62)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JStack.run(JStack.java:67)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JStack.runWithArgs(JStack.java:90)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJSTACK(SALauncher.java:259)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:450)
Can't print deadlocks:VM.initialize() was not yet called
Exception in thread "main" java.lang.RuntimeException: VM.initialize() was not yet called
at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VM.getVM(VM.java:460)
We are trying to install the JZOS Batch Launcher. The function consists of three pieces: a load module that must be put into a z/OS PDSE, a sample start proc that can be tailored and put into an appropriate PROCLIB, and sample JCL that can be tailored and put into an appropriate SAMPLIB.
On submitting the job we are getting return code=0101. Below are the trace details which we are getting in the job:
Output from DD:STDENV config shell script:
waiting for child shell process to complete
waitChild()
child shell process exited with exit code 0
waitChild()
Child shell process exited without printing environment; //STDENV should not contain 'exit'
adoptEnvironment()
run()
cleanup()
JZOS batch launcher elapsed time=0 seconds, cpu time=0.040000 seconds
JZOS batch launcher failed, return code=101
cleanup()
~JzosVM()
~JzosVM()
After looking this up and reading more,we then tried to run the job with superuser access and the job ran fine. So we need to know what privileges we require without being a superuser which will enable us to run the job successfully.
I am trying to build a simple auto updater for my application. I am currently checking the local application version against my remote version. If there is a newer version I want to start my updater.jar - which basically downloads and replaces the old application.
My problem is that I cannot seem to get the updater.jar to start if there is a new version.
The code I am currently using is:
Runtime runtime = Runtime.getRuntime();
try {
Process proc = runtime.exec("java -jar updater.jar");
} catch (IOException ex) {
Logger.getLogger(Splash.class.getName()).log(Level.SEVERE, null, ex);
}
System.exit(0);
The application exits but updater.jar is never launched..
Any ideas?
Your child process is likely exiting when your parent process exits.
When you launch a process you should usually:
consume stdout/stderr from the child process. If you don't do this your child process can block waiting for its output to be consumed. You should consume in separate threads. See this answer for more details
use Process.waitFor() to capture the exit code from the child process
It looks to me like you want to spawn the updater, let it perform a download and then exit your parent process. Anything more complex would likely be platform dependent (e.g. spawning a background process and disowning it)
Maybe the path to updater.jar should be specified in the java -jar command.
You better use the URLClassLoader since jre1.2
see How to load a jar file at runtime
Situation: I have a keep-alive shell script that restarts an application whenever it shuts down. However I do not want it to do this if the application was closed via a SIGTERM or SIGINT (kill, Ctrl+C, etc.) i.e. a shutdown hook. However I have no way of setting the exit code, hence communicating to the keep-alive script, when exiting from a shutdown hook as calling exit is illegal.
From Javadocs for exit:
If this method is invoked after the virtual machine has begun its shutdown sequence then if shutdown hooks are being run this method will block indefinitely. If shutdown hooks have already been run and on-exit finalization has been enabled then this method halts the virtual machine with the given status code if the status is nonzero; otherwise, it blocks indefinitely.
Is this possible?
If the process has been killed by a signal, the $? variable will be set to 128 + signal:
bash$ sleep 3;echo $?
0
bash$ sleep 3;echo $?
^C
130
Here, 130 is 128 + SIGINT.
Grab the PID of the process in a variable and use the wait builtin: if the process has been terminated by a signal, the return code of wait will be 128 + the signal number.
#
# Note: output from shell trimmed
#
# Launch cat in the background, capture the PID
$ cat & PIDTOCHECK=$!
$ echo $PIDTOCHECK
27764
#
# Call wait a first time: the program is halted waiting for input (SIGTTIN)
#
$ wait $PIDTOCHECK ; echo $?
149
#
# Now kill cat, and call wait again
#
$ kill %1
$ wait $PIDTOCHECK ; echo $?
143
Here's what I do:
Runtime.getRuntime().halt(0);
Note that this will exit the program immediately, so you need to do it after the last shutdown hook has finished.
I have set up an automated deployment script (in shell script) for my web application.
It uses java, tomcat, maven and a postgres database.
The deployment script does this:
builds the deployable application from source repository
stops tomcat
applies database migration patches
deploys the war files in tomcat
starts tomcat (by invoking $TOMCAT_HOME/bin/startup.sh)
exits with a success message
It's all working and it's pretty neat - but it needs a little improvement.
You see, even though it exits with a success message, sometimes the deploy was not successful because the web application did not start correctly.
I would like to refactor steps 5 and 6 so that after bring up the tomcat server, the deployment script would "tail -f" in the catalina.out file, looking either for a "server started successfully" message or an exception stack trace.
The tail -f output up to that point should be part of the output of the deployment script, and step 6 would "exit 0" or "exit 1" accordingly.
I know that should be possible, if not in shell script, maybe with python.
The problem is I'm a java specialist - and by specialist I mean I suck at everything else :-)
Help please? :-)
Maybe something like this?
tmp=$(mktemp -t catalina.XXXXXXX) || exit 136
trap 'rm "$tmp"' 0
trap 'exit 255' 2 15
tail -n 200 catalina.out >"$tmp"
if grep -q error "$tmp"; then
cat "$tmp"
exit 1
fi
exit 0
On the other hand, if startup.sh were competently coded, you could just
if startup.sh; then
tail -f catalina.out
else
exit $?
fi
which can be shortened to
startup.sh || exit $?
tail -f catalina.out
As an alternative, you might want to take a look at the Apache Tomcat Manager application. It supports, amongst other things:
Deploying applications remotely, and from local paths
Listing currently deployed applications
Reloading existing applications
Starting an existing application
Stopping an existing application
Undeploying an existing application
The manager provides a web interface that can be called via curl, and which returns simple, parseable messages to indicate the status of the invoked command. Management functions can also be invoked via JMX, or Ant scripts. All in all, a very handy tool.
I ended up implementing a solution using Python's subprocess.Popen, as suggested by #snies.
Here's what it looks like:
waitForIt.py
#! /usr/bin/env python
import subprocess
import sys
def main(argv):
filename = argv[1]
match=argv[2]
p = subprocess.Popen(['tail', '-n', '0', '-f', filename], stdout=subprocess.PIPE)
while True :
line = p.stdout.readline()
print line ,
if match in line :
break
p.terminate()
if __name__ == "__main__":
main(sys.argv)
tailUntil.sh
#!/bin/bash
set -e
filename=$1
match=$2
thisdir=$(dirname $0)
python $thisdir/waitForIt.py "$filename" "$match"
and then
startTomcat.sh
${TOMCAT_HOME}/bin/startup.sh
logDeploy.sh "Agora vamos dar um tail no catalina.out..."
util_tailUntil.sh "$TOMCAT_HOME/logs/catalina.out" 'INFO: Server startup in '
It doesn't do what I originally intended (it still exits with return code 0 even when there is a stacktrace - but that could be changed with a little bit more of Python magic),
but all of tomcat's initialization log is part of the automated deploy out (and easily viewable on Jenkins' deploy job) -
so that's good enough.