Develop Reference

Install4j: Is is possible to configure an install4j launcher to respond to kill -3 and generate a thread dump?

I would like to automatically create a thread dump as part of a log collection script I have written.
I know that it is possible to generate a thread dump using jstack or kill -3 . The customer running the log collection script will only have a JRE installed so jstack is not an option.
If I use jstack and the pid of the JVM that the install4j launcher creates I get the thread dump...however invoking kill -3 with the same pid generates no output. I am currently directing both stderror and stdout to files using the install4j launcher configuration.
This is a linux launcher that is configured with the service option.
FWIW...I have tried not directing the output and also running my application as a console program instead of a service and none of these allow 'kill -3' to work.
I'm using install4j 6.0
Lastly, I do not want to use -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=dump.log because of the unknown overhead it might create.
Any help would be appreciated!

I would suggest to use the ThreadMXBean to get this information programmatically in the same process rather than use signal handlers from another process. The API documentation is at
http://docs.oracle.com/javase/8/docs/api/java/lang/management/ThreadMXBean.html
Here's a full example of how to use this MBean.
In that way you can write the information explicitly to wherever you want it to go.

Java monitoring tool which will allow to see the stacktrace at "real time"

I work on very large web project which is written in java.
when I click some button or do other actions it is hard to me to understand what methods called in application code(because I am new in project and application is really really big). So I would like to know is there a tool which will allow to get stacktrace of some threads with given interval (say every 100 milliseconds ).
I know about VisualVm but it does not allow to do this, I can get thread dumb only at one point of time( there is no way to get stack trace continuously).
Can someone suggest tool or any technique which will allow me to monitor methods call at run-time.?
Thanks

For such cases I use Java Mission Control. The full features works on Oracle JDK, for OpenJDK not everything works properly. More info
From the website:
Starting with the release of Oracle JDK 7 Update 40 (7u40), Java Mission Control is bundled with the HotSpot JVM.
You need to add the following parameters in your JVM to be able to use it. note: I normally add also the debug options.
JAVA_DEBUG="-Xdebug -Xrunjdwp:transport=dt_socket,address=4000,server=y,suspend=n"
JAVA_JMC="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=3614 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UnlockCommercialFeatures -XX:+FlightRecorder"
Then you'll need to remote attach to port 3614 and you'll be able to see inside the JVM. There you'll be able to profile CPU, check allocation and deadlock detection + select the thread and see what is currently executing. And some other graphs and valuable information.

There are multiple ways in which you could check stacktrace:
Using jconsole's thread tab where you will get to see which all threads are alive and what state they are at.
Using JvisualVm (which comes fee with jdk installation) or you could use any of profilers like jprofiler/yourkit etc. to view stack trace when you run in development mode.
You could get stack trace say every minute by running kill -3 pid in unix or control + break on windows
You could use jstack command to get trace.
You could debug the code using say IDE (eclipse/netbeans/Intellij etc.)and after each and every method call trace the method call.

Many tools for Java VM monitoring and application monitoring exist. For instance, see the eG Java Application Monitor:
...the eG Java Monitor gives you a comprehensive view of the
activities within a JVM:
It lets you see which threads are running in the JVM and what state
they are in (such as runnable, blocked, waiting, timed waiting,
deadlocked, or high CPU).
You also have access to a stack trace for
each thread showing class, method, and line of code (to troubleshoot
problems down to the line of code level).
And you can monitor the
performance of garbage collection processes, CPU and memory usage, and
JVM restarts.

Java (hotspot) verbose:gc output use in application and process on runtime

Any way to hook on stdout of JVM currently running process or redirect stdout of JVM process to non file location?
I need verbose:gc output of JVM for my application, currently i can see this output in console but unable to store it in database or process it by any means, it invisible to running java application.
System.err
System.out
Both are wrappers over natives, internal JVM output not passes here.
Both can't see this output (ever if redirected or set to null, gc output shows in console).
Writing verbose:gc to file and then reading file not valid option due performance reasons and filesystem locks, checking hotspot sources not given any way to redirect output to nonfile location via JVM launch flags.
Also i was unable to get Process object of running JVM, there are no methods for this, also i was unable to find native that expose Process object, soo still no valid way to read this data.
Any clues about howto read verbose:gc in runtime?

Writing verbose:gc to file and then reading file not valid option due performance reasons and filesystem locks,
The cost of doing this is trivial compared to the cost of actually GCing. Unless you have a system which should never GC normally, I wouldn't worry about it. If you concern is that you can't read the file due to windows locks, you may have a problem (or you could use an OS like Linux which doesn't do this)
You should note that the GC output to a file is buffered, so it is not real time (but close enough for most use cases)
Reading the output programmatically is very difficult esp as it is written by a multi-threaded GC i.e. you get strange re-ordering of information.
I would consider getting the output of jstat which is designed to be read by a program though doesn't get as much detail.

You can get information GC pause information via JMX. It is not as full as GC logs, but will safe you from parsing. SJK tool is a CLI tool which could track GC events of JVM running processes. You could use it or use its code to build customized solution.
jstat is another option, but it exposes less information compared to JMX and its formatting rather misleading in case of CMS collector.
Problematically using PerfCounter API is third option (jstat is using this internally). See sun.management.counter.perf.PerfInstrumentation class for details.

Taking thread dumps in production

I am analyzing the differences between approaches for taking thread dumps. Below are the couple of them I am researching on
Defining a jmx bean which triggers jstack through Runtime.exec() on clicking a declared bean operation.
Daemon thread executing "ManagementFactory.getThreadMXBean().dumpAllThreads(true, true)" repeatedly after a predefined interval.
Comparing the thread dump outputs between the two, I see the below disadvantages with approach 2
Thread dumps logged with approach 2 cannot be parsed by open source thread dump analyzers like TDA
The ouput does not include the native thread id which could be useful in analyzing high cpu issues (right?)
Any more?
I would appreciate to get suggestions/inputs on
Are there any disadvantages of executing jstack through Runtime.exec() in production code? any compatibility issues on various operating systems - windows, linux?
Any other approach to take thread dumps?
Thank you.
Edit -
A combined approach of 1 and 2 seems to be the way to go. We can have a dedicated thread running in background and printing the thread dumps in the log file in a format understood by the thread dump analyzers.
If any extra information is need (like say probably the native thread id) which is logged only by the jstack output, we do it manually as required.

You can use
jstack {pid} > stack-trace.log
running as the user on the box where the process is running.
If you run this multiple times you can use a diff to see which threads are active more easily.
For analysing the stack traces I use the following sampled periodically in a dedicated thread.
Map<Thread, StackTraceElement[]> allStackTraces = Thread.getAllStackTraces();
Using this information you can obtain the thread's id, run state and compare the stack traces.

With Java 8 in picture, jcmd is the preferred approach.
jcmd <PID> Thread.print
Following is the snippet from Oracle documentation :
The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jstack utility for enhanced diagnostics and reduced performance overhead.
However, shipping this with the application may be licensing implications which I am not sure.

If its a *nix I'd try kill -3 <PID>, but then you need to know the process id and maybe you don't have access to console?

I'd suggest you do all the heap analysis on a staging environment if there is such an env, then reflect your required Application Server tuning on production if any. If you need the dumps for analysis of your application's memory utilization, then perhaps you should consider profiling it for a better analysis.
Heap dumps are usually generated as a result of OutOfMemoryExceptions resulting from memory leaks and bad memory management.
Check your Application Server's documentation, most modern servers have means for producing dumps at runtime aside from the normal cause I mentioned earlier, the resulting dump might be vendor specific though.

How do I connect to a Java command-line tool with the YourKit Java Profiler?

I've build a command-line tool in Java, which I would now like to profile with YourKit. I launch the command-line tool with something like:
$ java -classpath .:foo.bar.jar com.foobar.tools.TheTool arg1 arg2 arg3
It runs to completion in less than 2 seconds.
After reading http://www.yourkit.com/docs/80/help/agent.jsp, I tried the following:
$ java -agentpath:/home/dspitzer/yjp-8.0.24/bin/linux-x86-32/libyjpagent.so -classpath .:foo.bar.jar com.foobar.tools.TheTool arg1 arg2 arg3
...and I get:
[YourKit Java Profiler 8.0.24] JVMTI version 3001016d; 14.3-b01; Sun Microsystems Inc.; mixed mode, sharing; Linux; 32-bit JVM
[YourKit Java Profiler 8.0.24] Profiler agent is listening on port 10001...
[YourKit Java Profiler 8.0.24] *** HINT ***: To get profiling results, connect to the application from the profiler UI
...
(And then the tool runs to completion and exits.)
I guess (by default, at least) YourKit is designed to only connect to running application.
How should I modify my command-line tool to allow connection from YourKit?
I could add a command-line option
that will have it pause for input,
and I won't press return for it to
continue until I've connected to it
from YourKit.
Is there a YourKit API that I could
add to my tool that would cause it
to block until I've connected with
YourKit?
Is there a YourKit API or a java
command-line option that would
create a profiling "snapshot" that I
could load and analyze later (after
the command-line tool has completed)
with YourKit?

There is no way to pause profiled application until profiler
"connects"; you simply do not need to do so.
Instead, please do the following:
Turn on desired profiling modes from startup. For example, if you
need to profile CPU usage, start either CPU sampling or CPU tracing
with corresponding startup options "sampling" or "tracing".
See http://www.yourkit.com/docs/80/help/additional_agent_options.jsp
If profiled application is short-running, enable snapshot capture on
exit with "onexit=snapshot" startup option. Later open captured
snapshot in the profiler UI for analysis.

I'm not aware of a YourKit option to do this (which isn't to say that there isn't one), but there is a Java remote-debugging option to do this.
Try adding the following to your command line, in addition to the YourKit agent options:
-Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y
This should cause the VM to suspend as soon as it starts up, waiting for a debugger to connect on port 8000. YourKit itself isn't a debugger, but an IDE like Eclipse or Netbean will do the job, as will the basic jdb command line.
So run the command, with both options set, connect YourKit, then use a debugger to connect to the debug port, which will trigger the VM to continue execution, allowing YourKit to get the data you need.
It would be better to do this all using YourKit, but I know of no such way.

You could launch it via Eclipse, which has a YourKit plugin that launches on app launch.

YourKit's onexit=snapshot option saves a snapshot file under $HOME/Snapshots on JVM exit. You can load this snapshot into the YourKit GUI for later analysis.
See this page on their website for more details.

The answer to #3 is yes, there's an API you can use to have your application produce profiling results that YourKit can then examine.
Take a look at the API docs, and also the Command line tool to control profiling
Here's an example of using the API (I haven't tried this code):
Controller ykController = new Controller();
ykController.enableStackTelemetry();
ykController.enableExceptionTelemetry();
ykController.startCPUProfiling(ProfilingModes.CPU_TRACING,Controller.DEFAULT_FILTERS);
ykController.startAllocationRecording(false,0,false,0);
YourMainClass.main(args);
String snapshotLocation = ykController.captureMemorySnapshot();