Any way to hook on stdout of JVM currently running process or redirect stdout of JVM process to non file location?
I need verbose:gc output of JVM for my application, currently i can see this output in console but unable to store it in database or process it by any means, it invisible to running java application.
System.err
System.out
Both are wrappers over natives, internal JVM output not passes here.
Both can't see this output (ever if redirected or set to null, gc output shows in console).
Writing verbose:gc to file and then reading file not valid option due performance reasons and filesystem locks, checking hotspot sources not given any way to redirect output to nonfile location via JVM launch flags.
Also i was unable to get Process object of running JVM, there are no methods for this, also i was unable to find native that expose Process object, soo still no valid way to read this data.
Any clues about howto read verbose:gc in runtime?
Writing verbose:gc to file and then reading file not valid option due performance reasons and filesystem locks,
The cost of doing this is trivial compared to the cost of actually GCing. Unless you have a system which should never GC normally, I wouldn't worry about it. If you concern is that you can't read the file due to windows locks, you may have a problem (or you could use an OS like Linux which doesn't do this)
You should note that the GC output to a file is buffered, so it is not real time (but close enough for most use cases)
Reading the output programmatically is very difficult esp as it is written by a multi-threaded GC i.e. you get strange re-ordering of information.
I would consider getting the output of jstat which is designed to be read by a program though doesn't get as much detail.
You can get information GC pause information via JMX. It is not as full as GC logs, but will safe you from parsing. SJK tool is a CLI tool which could track GC events of JVM running processes. You could use it or use its code to build customized solution.
jstat is another option, but it exposes less information compared to JMX and its formatting rather misleading in case of CMS collector.
Problematically using PerfCounter API is third option (jstat is using this internally). See sun.management.counter.perf.PerfInstrumentation class for details.
Related
This might be a long shot of a question, but I have ran into a very complicated issue and I am unsure on how to solve it.
Long story short, we have a Java application running, it's currently using JDBC to pull in data from a MysQL Database on startup.
We have had a meltdown and that database is no longer active and has been lost forever and so has the data to go along with it which internally is very valuable.
However the data is still stored in the heap of the running JVM that pulled it in.
My only hope now is to somehow extract the data from the running JVM, in an ideal world i would be able to attach to it and have the flexibility to run code which could access the internal running classes..
So my questions today are:
Is my approach reasonable and possible?
If so how can I attach to the JVM and 'Inject' code
Thank you for reading
It seems that what you want to use is the jmap command. jmap can be used to dump the heap of a running JVM into a file, which you can then analyze "off-line", using tools such as jhat or JVisualVM.
It allows you to do so without killing the JVM and/or injecting code into it, and since the heap dump file is "inert", you can analyze it at your leisure without fear of harming the running VM by probing it further. Admittedly, I haven't used it extensively, so I'm not sure exactly what its capabilities are, but theoretically, you could perhaps also use JVisualVM's OQL language to run automated sequences on data in the heap and dump it to files in a format you want.
See, for instance, this question for usage examples.
In a situation like this the Eclipse Memory Analyzer Tool can be a good solution. It works on heap dumps too and shows you which objects take up memory.
In addition to this it can show the content of objects / memory locations.
I sometimes found MAT goes beyond what VisualVM does, but perhaps a view like this helps you find your data already:
(This is a screenshot of a made-up example where I create some custom objects in
order to show them with their value in the heapdump)
Perhaps you can even attach Eclipse to the running application. There is a certain trick where you can run custom code in a breakpoint. This one could somehow dump your data to disk.
Everyone knows that a heap dump can be obtained from a running JVM. Is the other way possible? Can we start a JVM using a heap dump?
I have been having this question in mind for a long time now. If this is possible it would solve a lot of time and make thinks easy for a support engineer. It helps big time in cases where if we have to recreate some the rare problems our customer face. [Just imagine that the underlying hardware and Java runtime are the same and also all the supporting files are also present in the respective location in file system].
Added note: The intention of doing this is not when OOM occurs but at any given point after JVM starts.
No you can't.
You would need things like the current position in each open file. That affects what data is returned on a simple sequential read. The restorer would need to open each file and get it to the correct position. That may not be possible for non-seekable streams.
Program specific serialization is a much more feasible path, then setup the program from there.
Also, as Heap Dumps are usually from OutOfMemory situtaions, recreating JVM from same would again throw OutOfMemoryException. If you are taking heap dump in between, then serialize your objects and restore them when you bring up jvm.
(contents copied from comments of this question, Authors almas-shaikh and patricia-shanahan)
To create a heap dump from a running JVM you can also use jhat, or jcmd (with the GC.heap_dump command), both exist in the JDK/bin folder. MAT is one way of analyzing the contents of the dump. Java Mission Control has a tool called JOverflow that analyzes heap dumps, but only to look at memory waste patterns.
I have never heard of any way to restart a JVM from sort of image, a heap dump would not be enough at all, since it only contains the Java objects, and not the compiled code and other things.
I think you are looking for tools like Java Mission Control and Chronon DVR (commercial). These help you incident analysis, event collection and profiling, time travel debugging (as phrased by chronon)
As per their documentation:
Java Mission Control
Java Flight Recorder and Java Mission Control together create a
complete tool chain to continuously collect low level and detailed
runtime information enabling after-the-fact incident analysis. Java
Flight Recorder is a profiling and event collection framework built
into the Oracle JDK. It allows Java administrators and developers to
gather detailed low level information about how the Java Virtual
Machine (JVM) and the Java application are behaving. Java Mission
Control is an advanced set of tools that enables efficient and
detailed analysis of the extensive of data collected by Java Flight
Recorder. The tool chain enables developers and administrators to
collect and analyze data from Java applications running locally or
deployed in production environments.Starting with the release of
Oracle JDK 7 Update 40 (7u40)
Some key features of Chronon Recording Server which will be useful in your case:
The Recording Server is specifically designed for long running, server
side applications that run for weeks or months at a time. The
Recording Server will take care of splitting the recording if it get
too large and flushing out old recordings.
Get rid of the need to look at long sparsely detailed log files to
debug your program. Just play back the entire execution and see
exactly what took place in your program.
The Recording Server makes it share recordings on different machines
among team members or across multiple teams.
I have a Java process which has been running for over a week. In the process, I have been processing some data and have been storing some intermediate result into a hashmap in memory.
Now, I need to stop the process due to some bugs in the code. But if i kill the process, then i lose the data in the hashmap n will have to reprocess it again, the next time i run the code.
Is their a way by which i can take a dump of the hashmap present in the memory ?
You can trigger a memory dump which you can read with various tools such as profilers. This is very difficult to read and I have never heard of some one using this to restart a program.
Restarting a program is not something you can do as an afterthought, it needs to designed into your application and thoroughly tested. One way people get around having to worry about this issue is to use a database. This is because databases are, designed and tested professionally, to be restarted without losing data.
You can use JConsole, which is included with the Standard JDK, or visualvm to attach to an already-running process and trigger a heap dump.
However, as Peter said, actually reading this dump is not going to be fun unless you can find a tool to assist you.
lsof is a nice tool for Unix, showing all currently open file handles.
Does anyone know a similar tool that would show all open files inside a running JVM (via JVMTI or any similar interface)?
In this particular case, it would be sufficient for me to know which class has a handle open. Method/line or even an entire chain to GC root would be fantastic, but handler owner class is already a good start.
I know I could make a heap dump, open it in a profiler and find this out, but this is a tedious task, especially for the big heaps.
The JVMTI option sounds like it wouldn't be a bad choice. The big issue would be ensuring you wrap everything that may open a file handle: you would basically have to go through the JDK source code and find every native function that did a file open (littered throughout java.io., java.nio., I'd think java.net.* as well if you consider sockets as file handles, and just about everywhere else that a file handle may be opened by a native function) and then wrap them all with the SetNativeMethodPrefix call.
I'm assuming that is what some of the profiling folks do: however if you're not required to do this listing in real time then I'd think it would be WAY easier to use lsof or handle (on Windows platforms) and filter for your JVM's process id.
I have an application running on Websphere Application Server 6.0 and it crashes nearly every day because of Out-Of-Memory. From verbose GC is certain there are the memory leaks(many of them)
Unfortunately the application is provided by external vendor and getting things fixed is slow & painful process. As part of the process I need to gather the logs and heapdumps each time the OOM occurs.
Now I'm looking for some way how to automate it. Fundamental problem is how to detect OOM condition. One way would be to create shell script which will periodically search for new heapdumps. This approach seems me a kinda dirty. Another approach might be to leverage the JMX somehow. But I have little or no experience in this area and don't have much idea how to do it.
Or is in WAS some kind of trigger/hooks for this? Thank you very much for every advice!
You can pass the following arguments to the JVM on startup and a heap dump will be automatically generated on an OutOfMemoryError. The second argument lets you specify the path for the heap dump file. By using this at least you could check for the existence of a specific file to see if a heap dump has occurred.
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=<value>
I see two options if you want heap dumping automated but #Mark's solution with heap dump on OOM isn't satisfactory.
You can use the MemoryMXBean to detect high memory pressure, and then programmatically create a heap dump if the usage (or usage delta) seems high.
You can periodically get memory usage info and generate heap dumps with a cron'd shell script using jmap (works both locally and remote).
It would be nice if you could have a callback on OOM, but, uhm, that callback probably would just crash with an OOM error. :)
Have you looked at JConsole ? It uses JMX to give you visibility of a variety of JVM metrics, including memory info. It would probably be worth monitoring your application using this to begin with, to get a feel for how/when the memory is consumed. You may find the memory is consumed uniformly over the day, or when using certain features.
Take a look at the detecting low memory section of the above link.
If you need you can then write a JMX client to watch the application automatically and trigger whatever actions required. JConsole will indicate which JMX methods you need to poll.
And alternative to waiting until the application has crashed may be to script a controlled restart like every night if you're optimistic that it can survive for twelve hours..
Maybe even websphere can do that for you !?
You could add a listener (Session scoped or Application scope attribute listener) class that would be called each time a new object is added in session/app scope.
In this - you can attempt to check the total memory used by app (Log it) as as call run gc (note that invoking it will not imply gc will always run)
(The above is for the logging part and gc based on usage growth)
For scheduled gc:
In addition you can keep a timer task class that runs after every few hrs and does a request for gc.
Our experience with ITCAM has been less than stellar from the monitoring perspective. We dumped it in favor of CA Wily Introscope.
Have you had a look on the jvisualvm tool in the latest Java 6 JDK's?
It is great for inspecting running code.
I'd dispute that the you need the heap dumps when the OOM occurs. Periodic gathering of the information over time should give the picture of what's going on.
As has been observed various tools exist for analysing these problems. I have had success with ITCAM for WebSphere, as an IBMer I have ready access to that. We were very quickly able to indentify the exact lines of code in out problem situation.
If there's any way you can get a tool of that nature then that's the way to go.
It should be possible to write a simple program to get the process list from the kernel and scan it to see if your WAS process is still running. On a Unix box you could probably whip up something in Perl in a few minutes (if you know Perl), not sure how difficult it would be under Windows. Run it as a scheduled task every five minutes or so, and if the process doesn't show up you could have it fork off another process that would deal with the heap dump and re-start WAS.