I have been asked to investigate Oracle Java Mission Control, so that server-side Java applications may be monitored and actions taken (e.g., alerts emitted and logged, flight recordings saved) under certain conditions. Java Mission Control's trigger system, where you specify conditions and actions, meets our needs, but it seems to depend on the GUI application ("Oracle Java Mission Control") being running, implying that triggers are not the monitored JMX server's responsibility. Is this the case? There are a number of servers usually accessed via terminal...
Is there a way of running Java Mission Control as a daemon, from a terminal session, unattended, while retaining and obeying any specified trigger rules (e.g., imported from an XML file)?
If not, are there competing tools with a similar trigger system that can fill the void?
Thanks! :)
Currently no, you can't run JMC without a GUI.
You are not the first person that wants to do this.
One option is to run JMC in another machine, and make it connect to many servers, which of course requires running the remote JMX agent etc.
We have been discussing server side triggers/rules, but AFAIK, it is not planned for any JDK release.
It is possible to dump flight recordings from code, so you could write your own little agent that uses the DiagnosticMBean to do this on another JVM on the same machine or on remotely. I'm pretty sure this how some people solve the same sort of problem. It is also possible to parse and analyze flight recordings in code. If you're interested in this approach, I'm sure there's some sample code around, of course it's more work than if JMC could run as a daemon :/
You should probably have a look at an APM tool instead of monitoring with JMC. The product is extremely weak, introduces a lot of overhead (making it unsuitable for production) and creates a lot of issues. There are also developer focused tools available out there.
APM : AppDynamics (deepest of the bunch), New Relic, Ruxit
Java Developer Tools : Takipi, Fusion Reactor, Javosize
Related
This question is not about how JMX works or what JMX does. As far as I know On top of using JMX we can get OS level metrics and JVM specific metrics (such as: Garbage collection time & frequency, heap utilisation etc)
My Question is What are the aspects can be monitored with JMX in Java applications(internal metric)?
Monitor your app’s status at runtime
Java Management Extensions (JMX) is a standard way for you to embed code within your own app to report at runtime the state of your app’s operations.
This embedding of a reporting agent (a “probe”) within a larger piece of software is known as “instrumenting” your code. JMX enables you a framework to surface those pieces of static information at runtime, so you need not invent that reporting-system plumbing yourself. At runtime, you, or your system administrator, can use any of a number of standard monitoring apps sometimes known as “consoles” or “dashboards”.
JMX handles transporting the updates to the monitoring apps without you needing to do any additional programming. The monitoring app need not be local, instead could be running remotely over the network. Which monitoring app is chosen by your sysadmin, and where they choose to run it, has no effect on your code in your app. JMX is a buffer, a layer of indirection, separating your compile-time code from these practical run-time configuration issues.
The purpose is to provide a control room like this, but for your software:
The key advantage here is using standard protocols for reporting status, rather than you inventing your own protocols.
App servers
If your app is a web app, then your Jakarta EE server or web container such as Tomcat or Jetty may be instrumented with JMX. So you can monitor its operations. For example, you can see what user-sessions are currently open.
JVM
Some JVM implementations are themselves instrumented using JMX to report the status of various aspects of the JVM’s operations. As your Question mentioned, some of those reports may be on memory usage, garbage collector activity, etc. You sysadmin’s monitoring app can watch both the JVM and your app, each reporting a stream of status updates.
Operating system
Your operating system may also be instrumented to report on its internal operations as well, though not likely using JMX. One powerful dynamic tracing framework for this purpose is DTrace, built into macOS, FreeBSD, and Solaris.
So your sysadmin may be watching all four sets of status information on her monitoring app: the OS, the JVM, the app server, and your app.
Read the Wikipedia page for basic info.
Read and write
JMX provides not only read-access to monitor current status, but also writing. Your chosen external monitoring app can be used to alter the state within your app in whatever way you choose in your programming. For example, you could change the size of thread pools or caches.
Continuing our metaphor of the control room seen above, you can think of read-access via JMX as watching the gauges seen on those control room panels.
Think of write-access as turning the knobs, switches, and sliders on those panels.
Other than the more 'advanced' GUI from Java mission control, how are they different?
At first glance they seem to offer very similar functionality (Interpreting JMX data and Memory/CPU profiling).
However, as they are both shipped with the JDK (I'm using JDK 1.7.0_51 SE) I'm assuming there are significant differences, otherwise they would be combined into a single solution. Especially as this increases the size of the JDK significantly.
Is Java Mission Control ultimately going to replace JVisualVM in the future?
One important point is that Mission Control is potentially not free to use on production environments. It is free for applications running in DEV & QA and Oracle are not currently enforcing the charges for production applications (as of Nov 2014). However, their executives have made it clear this may change in time.
The JMX Console part of Java Mission Control is just like any other JMX console. I'm of course biased, but in my opinion it's one of the more feature rich consoles available. The more unique part of JMC is the Java Flight Recorder part.
JMC is targeting production systems, and is very careful to avoid introducing unnecessary overhead. With the Java Flight Recorder you can do production time profiling and diagnostics with an almost unmeasurable overhead.
I have found the DTrace intriguing but have personally failed to see a use-case that allow me to get information that I cannot get otherwise anyway without using DTrace.
Hence, I would like to hear what I have overlooked. What can I do on my vmware OpenIndiana build 148 with DTrace that can make a difference when creating stand-alone applications and Java EE web applications (most of which communicate heavily with a legacy backend using sockets)?
Non-trivial Dtrace scripts are very welcome.
I had some good experiences with DTrace.
I had a client running our Java code on a production server running 24x7. We had some performance issues with the applications. It was impossible to stop the JVMs in order to attach the profiler. Moreover the behavior was not present in our lab under the same load.
We solved the issue using DTrace with the JVM related probes as I could attach it to the running JVMs and the overhead introduced was minimal (1.3% on a Netra T2000 SPARC machine).
The bonus of the method was that all the debugging was done via a dial-up (33kbps) line to the customer's lab. It is almost impossible to run any other profiler/debugger with this constraints (JDWP is quite verbose for this bandwidth). With my DTrace script I filtered only what it was interesting for me.
For some hints see:
http://java.dzone.com/articles/java-profiling-dtrace?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+javalobby/frontpage+(Javalobby+/+Java+Zone)
http://download.oracle.com/javase/6/docs/technotes/guides/vm/dtrace.html
I have adapted those scripts
http://blogs.oracle.com/ahl/date/20050418#dtracing_java
Another big plus are the aggregated values that can be easily used to create custom statistics.
But, if you have appropriate conditions, everything can be done easier with othe tools. DTrace is very powerfull but it targets more bare-metal conditions (maybe DLight project would help).
I want to find or develop an application that can run as a daemon, notify the administrator by email or sms when the Java applications running on a host get any exceptions or errors. I know JVMTI can achieve part of my goal, but it will impact performance of the monitored applications(I don't know how much will it be, it will be acceptable if it's slight), besides it seems to be a troublesom job to develop a JVMTI agent and I'm not sure what would happen if several applications running at the same time using the same agent. Is there any better solutions? Thanks in advance.
One way would be to use a logging system like log4j that publishes all errors occuring on system A to a logging server on system B from which you can monitor the errors occured. This isn't a completely generic solutation however, since only exceptions propagated to log4j (or any other logging system) would be handled - but it may be a good start.
The best solution is to have the Java application send its errors via email/sms. The problem is that programs will generate exceptions and handle correctly in normal operation. You only want particular exception.
Failing this you could write a log reader, which reads the logs of the application. This is tricky to get right, but it can be done.
An application can generate 1000+ exception per days and still be behaving normally because the application knows how to handle these exceptions. e.g. every time a socket connection is closed an exception can be thrown.
IMO, the best approach is to deploy an external monitoring system. This can:
monitor multiple applications
monitor infrastructure services
monitor network availability and machine accessibility,
monitor resources such as processor and file system usage.
Applications can be monitored in a variety of ways, including:
by processing log events,
by watching for application restarts,
by "pinging" the application's web apis to check service liveness, and
by using the application's JMX interfaces.
This information can be filtered and prioritized in an intelligent fashion, and critical events can be reported by whatever means is most appropriate.
You don't want individual applications sending emails, because they don't have sufficient information to do a decent job. Furthermore, putting the reporting logic into individual applications is likely to lead to inconsistent implementation, poor configurability, and so on.
There is a nearby alternative to JVMTI : JPDA. This infrastructure allows you to create a remote "debugger" (yes, that's what you're planning to do) using Java code, and connect it to the VM using either local or remote connection.
There will be, like for JVMTI, an overhead to program execution. However, as the Trace.java example shows, it's quite simple to both implement and connect to target VM.
Finally, notice if you want to instrument code run by application server (JBoss, Glassfish, Tomcat, you name it) there are various other means available.
I follow the pattern where every exception gets logged to a table.
Then an RSS feed selects from that table.
I subscribe to the RSS feed in MS Outlook at work and also on my Android phone with a program called NewsRob. NewsRob let me set my phone to alert me when there is something new.
I blog about how to do this HERE. It is in .net, but you get the idea.
As a related step I found a way to notify myself when something DIDN'T happen. That blog is HERE.
There are loads of applications out there that do what you are looking for in a way that does not impact performance. Have you had a look at Kibana/ElasticSearch, or Splunk or Logscape for enterprise solutions ( they both also have free versions).
I'm going to echo what has already been said and highlight what java already provides and what you can do with an external monitoring system. Java already provides:
log4j - log ERRORS, WARNINGS, FATAL and Exceptions to a file
JMX - Create custom application metrics and you also have access to java.lang/* which will give you heap memory usage , garbage collection, thread counters etc.
JVM gc logging - you can log all your garbage collection events to a file and watch for any long Full GC collections.
An external monitoring system will allow you to set alerts triggered off different operational scenarios. You will also get visualisation of your system performance through charts. I've used Logscape's java app in the past to monitor 30 java processes spread out over3 hosts.
I've written a Java file, using Jsp,servlets, that I would like to perform run-time tests on. I've never done this before and was just curious on how to go about it.
What I'm interested in knowing, besides the actual timings, is how to find cpu,memory and io utilization when running the application.Your thoughts are appreciated.
Typically you wouldn't measure these from within the application, but by running another tool on the same host.
If you just want to see the impact on the host operating system, you can use a program like top (on *nix boxes), or good old Task Manager on Windows, to see the CPU/memory/IO utilisation of your Java process (typically the servlet container such as Tomcat).
If you want more detailed information on the actual Java process itself, you can connect JConsole or jvisualvm to get VM information (including memory and CPU) for the process itself. (With Java 6 you should be able to do this from the local machine without passing any parameters to the Java process at startup; for Java 5, or remote connections, you'll need to pass command-line arguments to the Java process to allow (remote) JMX connections.)
Finally, if you want really in-depth details of the resource usage, down to the performance of various methods (which it sounds like you're after), you'll need to use a profiler. There are several of these for Java - with YourKit and JProfiler being the biggest commercial ones (in my unqualified opinion). I believe that the NetBeans IDE also has a decent profiler built-in. The process for connecting these to your application would vary depending on the app itself, but these will all typically allow you to "drill down" into the CPU time to see which classes/methods took the most cycles to execute, and likewise to drill down into memory use to see which classes are taking up the most memory.
The standard way to monitor running Java applications these days is sing JMX through the JConcole
If your a using a commercial application server like Weblogic or WebSphere these have custom and powerful management consoles that provide the monitoring information you are looking for. The technology at the heart of these consoles is still JMX so these can also be monitored and managed using the standard JConsole. This article shows how to do this for Weblogic.
I guess you need this info in the client side (browser). So it's not Java based question.
If so, here is my answer:
I prefer using FireBug and ySlow extensions. They give performance information, memory information and much more.
I combine it with using regular task-manager to view more information about the browser.
BR