I understand that we can use killall -3 java to get the threaddump.
My question is:
If multiple java processes are running, which process's thread dump is taken ?
Or are thread dumps taken for all processes ?
Thread dumps aren't "taken" in the situation you describe. Java JVMs generally respond to the signal by writing the dump to stdout. It may be captured and stored, but that doesn't change the fundamental principle.
Consequently, you can signal all JVMs on a host to produce a thread dump if you wish.
In many cases, it's more productive to use a utility like jstack to collect the thread dump, because it will allow better control of where the dump is actually written.
Related
I was reading about the concepts of Thread Dumps and found various ways to take Thread Dumps but no articles have mentioned on what basis/issues/reasons the dumps should be taken. We perform a load test on a particular server targeting a JVM and take the Thread Dumps when we observe high CPU utilization or hogging threads. Is this correct ? Can someone throw some light on the reasons on when Thread Dumps should be taken normally or during any load tests.
We use the jstack command to capture the dump:
/app/jdk/jdk1.7.0_111/bin/jstack -l <ProcessID> > <PathToSaveTheFile>
TIA.
Thread dumps are used for post-mortem debugging. IMO, what you are doing is right. I don't see a reason for taking a dump in normal conditions.
Non- Less invasive debugging
You can do thread dumps whenever you see fit to analyse anything to do with thread lock contention, deadlock detection, system resource contention, ...
This is why there are tools to facilitate thread dump whenever we see fit, not only after a JVM crash. Analysis of multiple dumps over time paints fuller picture, than the last crash dump.
This means you can perform less invasive thread debugging without attaching a profiler, which in most cases slows down (and possibly alters some dynamic properties) of the application execution.
Hadoop jvm process hangs without any error message,
I want to take a look into what JVM processes are doing (where they are stuck).
When I program in C++, I used GDB that can be attached to a running process and show the call stack of the threads.
How can I do the same thing for JVM?
You may use following command
kill -3 [PID]
This will print stack traces of all threads to the console of your java process. Another option is to use jstack utility which is bundled with jdk. Jstack does the same thing.
If it doesn't help then profilers should help. They can gather a lot more data than one thread dump.
I am analyzing the differences between approaches for taking thread dumps. Below are the couple of them I am researching on
Defining a jmx bean which triggers jstack through Runtime.exec() on clicking a declared bean operation.
Daemon thread executing "ManagementFactory.getThreadMXBean().dumpAllThreads(true, true)" repeatedly after a predefined interval.
Comparing the thread dump outputs between the two, I see the below disadvantages with approach 2
Thread dumps logged with approach 2 cannot be parsed by open source thread dump analyzers like TDA
The ouput does not include the native thread id which could be useful in analyzing high cpu issues (right?)
Any more?
I would appreciate to get suggestions/inputs on
Are there any disadvantages of executing jstack through Runtime.exec() in production code? any compatibility issues on various operating systems - windows, linux?
Any other approach to take thread dumps?
Thank you.
Edit -
A combined approach of 1 and 2 seems to be the way to go. We can have a dedicated thread running in background and printing the thread dumps in the log file in a format understood by the thread dump analyzers.
If any extra information is need (like say probably the native thread id) which is logged only by the jstack output, we do it manually as required.
You can use
jstack {pid} > stack-trace.log
running as the user on the box where the process is running.
If you run this multiple times you can use a diff to see which threads are active more easily.
For analysing the stack traces I use the following sampled periodically in a dedicated thread.
Map<Thread, StackTraceElement[]> allStackTraces = Thread.getAllStackTraces();
Using this information you can obtain the thread's id, run state and compare the stack traces.
With Java 8 in picture, jcmd is the preferred approach.
jcmd <PID> Thread.print
Following is the snippet from Oracle documentation :
The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jstack utility for enhanced diagnostics and reduced performance overhead.
However, shipping this with the application may be licensing implications which I am not sure.
If its a *nix I'd try kill -3 <PID>, but then you need to know the process id and maybe you don't have access to console?
I'd suggest you do all the heap analysis on a staging environment if there is such an env, then reflect your required Application Server tuning on production if any. If you need the dumps for analysis of your application's memory utilization, then perhaps you should consider profiling it for a better analysis.
Heap dumps are usually generated as a result of OutOfMemoryExceptions resulting from memory leaks and bad memory management.
Check your Application Server's documentation, most modern servers have means for producing dumps at runtime aside from the normal cause I mentioned earlier, the resulting dump might be vendor specific though.
I searched previous postings on the similar topic but could not find a befitting answer therefore asking this question. Your help in answering it is highly appreciated.
I am aware of setting a process's affinity to a particular CPU core by taskset command in Linux. But I want to set a Java thread's affinity to a particular cpu core so that other threads belonging to the same process can run on all remaining cores. For example if I have a process containing 10 threads with 4-core machine, I would like to reserve core-1 for a thread and let remaining 9 threads run on remaining 3-cores. Can it be done and how?
Thanks
Sachin
Say 2241 is the pid of your java process. Run:
jstack 2241
This gives you a list of threads. Find yours there and note the nid field. Say nid=0x8e9, which converts to base 10 as 2281. Then run:
taskset -p -c 0 2281
Done.
Unfortunately, you cannot assign Java Threads to a specific core. What you can do, however, is set Thread Priorities to prioritize the threads (assuming that this would accomplish the same thing)
Alternatively, you could use JNI, but that would be completely overkill.
Remember the Java application that your running is actually running in a JVM which is in turn running on the OS. For you to be able to directly interact with the CPU you'd need a low level programming language (e.g. C).
As suggested in another answer you can use JNI to interact with lower level language (like C) to do what you want however you'd have to delegate concurrency (threads managed within that lower level langaguge) to it...
You can do that in plain Java using JNA. There is no need to use taskset. Just remember that thread affinity is pointless unless you have previously isolated the core from kernel/user threads and hardware interrupts. I am affiliated with Coral Blocks which has developed CoralThreads which does exactly that.
I have recently started multi-threaded programming with Java
in case of Linux threads , i know that the kernel schedules them(as they are the unit entities that are scheduled)but java programs are run on JVM which in my system (RHEL 6.1) is implemented as a program that is run as a user space instance .So, without the kernel being aware of the java threads, how come preemptive multitasking is done in JVM? it will be helpful if the whole mechanism of JVM and kernel interaction in doing this thing is given.
Please cite possible sources of info
Threads in the java/JVM process maps to a native thread and you can see both the java thread id and the native thread id in a thread stack trace dump. Get the thread stack of all java threads using your favorite tool:
command line signal like ctrl+break (windows) or ctrl+\ linux) in the console where the java program is running
command line tool (kill -QUIT or jstack from the jdk)
visual vm in the jdk and/or jmx etc
Example extract from the first line of such a thread dump:
... tid=0x0000002adaba9c00 nid=0x754c ...
tid = java thread id
nid = native id (the OS thread id)
Use the operating system's tools to find out more about the thread using the native id (it is in hex).
Inside the java code you have ThreadMXBean to retrieve more thread information programatically if you want
http://docs.oracle.com/javase/6/docs/api/java/lang/management/ThreadMXBean.html
Read Distinguish Java threads and OS threads?
As I said in the comment Java threads are ordinary OS threads just running JVM code
The jvm is just a normal process which starts with one thread and can spawn as much threads it likes afterwards. Scheduling is done on two levels - between processes and between threads inside processes. All this is done by the OS (via libs) - the jvm just hooks in. Google posix threads for more details - thats whats exposed (API) to the jvm.
This goes a bit into details:
http://www.ibm.com/developerworks/java/library/j-rtj3/
"but java programs are run on JVM which in my system (RHEL 6.1) is
implemented as a program that is run as a user space instance.So,
without the kernel being aware of the java threads ..."
This statement is incorrect for all modern JVM's that use native threads. I think thats been the default since Java 1.2.
Native Thread implementation by a JVM means that each time a thread instantiates/runs a thread in Java code, the JVM asks the OS to create the thread. Since these are native threads, the kernel knows about them and treats them accordingly. Furthermore, Linux supports/implements POSIX threads, and as such on a Linux based systems you will get pthread behavior for your Java apps' threads