Develop Reference

Command-line-based daemon for Java Mission Control? Alternatives?

I have been asked to investigate Oracle Java Mission Control, so that server-side Java applications may be monitored and actions taken (e.g., alerts emitted and logged, flight recordings saved) under certain conditions. Java Mission Control's trigger system, where you specify conditions and actions, meets our needs, but it seems to depend on the GUI application ("Oracle Java Mission Control") being running, implying that triggers are not the monitored JMX server's responsibility. Is this the case? There are a number of servers usually accessed via terminal...
Is there a way of running Java Mission Control as a daemon, from a terminal session, unattended, while retaining and obeying any specified trigger rules (e.g., imported from an XML file)?
If not, are there competing tools with a similar trigger system that can fill the void?
Thanks! :)

Currently no, you can't run JMC without a GUI.
You are not the first person that wants to do this.
One option is to run JMC in another machine, and make it connect to many servers, which of course requires running the remote JMX agent etc.
We have been discussing server side triggers/rules, but AFAIK, it is not planned for any JDK release.
It is possible to dump flight recordings from code, so you could write your own little agent that uses the DiagnosticMBean to do this on another JVM on the same machine or on remotely. I'm pretty sure this how some people solve the same sort of problem. It is also possible to parse and analyze flight recordings in code. If you're interested in this approach, I'm sure there's some sample code around, of course it's more work than if JMC could run as a daemon :/

You should probably have a look at an APM tool instead of monitoring with JMC. The product is extremely weak, introduces a lot of overhead (making it unsuitable for production) and creates a lot of issues. There are also developer focused tools available out there.
APM : AppDynamics (deepest of the bunch), New Relic, Ruxit
Java Developer Tools : Takipi, Fusion Reactor, Javosize

Gather profiling statistics from desktop client applications

I have a numerous client instances of some desktop application.
Some users of this application encounter performance problems while using some specific reproducing steps and their private execution context (i.e. let say using some private kittens photos that they do not want to share with anybody).
I would like to minimize the number of communication with users and reproduce their problems successfully on my development environment.
I cannot use their execution context because of legal reasons.
So the only option I see here is to gather statistics of application usage (i.e. method calls, CPU load factor).
Ideally I would like to simplify the life for users and just ask them to enable/disable statistics gathering in the application when they see some problems. All other (i.e. capturing customized statistics, transferring statistics to the support) would be done in background.
Looks like a rather common need.
Are there any solutions that can help to achieve described behaviour?

JProfiler allows you to distribute the profiling agent at no cost and operate it in offline mode. The profiling agent is activated by adding a special VM parameter to the invocation of the JVM (-agentpath:...).
Then you can use the Controller class to record data and save snapshots to disk. The start/stop button for recording statistics in your desktop application would call these methods.
If the application is obfuscated, JProfiler can deobfuscate the snapshot when you open it.
To set this all up, create a locally launched session in JProfiler and then choose
Sesssion->Conversion Wizards->Convert Application Session To Redistributed Session
Disclaimer: My company develops JProfiler.

Distributed tracing is really what I needed.
There is a Dapper project from Google.
And Zipkin from Twitter with many integrations including Spring via Sleuth.

How to attach VisualVM and start profiling immediatedly with application start?

I can attach VisualVM to a running application, then start profiling.
All this time while I am performing manual operations, the target application is running and statistics is loosing.
How to begin collecting statistic exactly at the same moment application starts?
UPDATE
With Profile Startup what is the way to set which class to run?
Very small cell for application setup says
-agentpath:D:/Apps/visualvm_138/profiler/lib/deployed/jdk16/windows-amd64/profilerinterface.dll=D:\Apps\visualvm_138\profiler\lib,5140
i.e. contains no clues to application location.

You can try the startup profiler plugin:
https://visualvm.java.net/startupprofiler.html

How to profile a headless Java application, running remotely?

I have a headless Java application, running on a remote server as a daemon-style process.
I want to extract Java level profiling information from the process, of the sort displayed by JVisualVM. For example, it should show method invocation times and so on.
What is the best way of doing this? My understanding is that JVisualVM does not profile when connecting remotely.
Ideally, the profiling information would be stored in a file for later inspection, in a manner similar to Java heap generation (with jmap) and later analysis (with a heap inspector).

You can use NetBeans profiler remote profiling capability.
Remote Profiling
Profile an application that is running on a different system than your NetBeans IDE. The profiler's remote pack can be installed on a remote system, allowing you to profile an application that is started on that system.
In fact, VisualVM is based on it.
Beside various monitoring features, the tool contains a built-in profiler based on the NetBeans profiler. While the profiler UI in VisualVM looks simple (especially when compared to the NetBeans profiler), the profiling capabilities are almost as powerful as in NetBeans.
Here is a detailed blog post about Profiling a Java remote server using Netbeans.

According to the link you will have to set up jstatd and jmx:
http://javadevsoup.blogspot.de/2012/02/remote-java-profiling-using-visual-vm.html

I just found another product that I think does this: http://chrononsystems.com/products/chronon-recording-server

Best way to deploy a Java application on a cluster of servers?

I have a cluster of 32 servers and I need a tool to distribute a Java service, packaged as a Jar file, to each machine and remotely start the service. The cluster consists of Linux (Suse 10) servers with 8 cores per blade. The application is a data grid which uses Oracle Coherence. What is the best tool for doing this?

I asked something similar once, and it seems that the Java Parallel Processing Framework might be what you need:
http://www.jppf.org/
From the web site:
JPPF is an open source Grid Computing
platform written in Java that makes it
easy to run applications in parallel,
and speed up their execution by orders
of magnitude. Write once, deploy once,
execute everywhere!

Have a look at OpenMOLE: http://www.openmole.org/
This tool enables you to distribute a computing workflow to several resources: from multicores machines, to clusters and computing grids.
It is nicely documented and can be controlled through groovy code or a GUI.
Distributing a jar on a cluster should be very easy to do with OpenMOLE.

Is your service packaged as an EJB? JBoss does a fairly good job with clustering.

Use Bit Torrent. Using Peer to Peer sharing style on clusters can really boost up your deployment speed.

It depends on which operating system you have and how security is setup on your network.
If you can use NFS or Windows Share, I suggest you put the software on an NFS drive which is visible to all machines. That way you can run them all from one copy.
If you have remote shell or secure remote shell you can write a script which runs the same command on each machine e.g. start on all machines, or stop on all machines.
If you have windows you might want to setup a service on each machine. If you have linux you might want to add a startup/shutdown script to each machine.
When you have a number of machines, it may be useful to have a tool which monitors that all your services are running, collects the logs and errors in one place and/or allows you to start/stop them from a GUI. There are a number of tools to do this, not sure which is the best these days.