My java process stopped responding. I tried to jstack but failed with below error.
21039: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
Then I used -F option, but "No deadlocks found."
Other info:
java version: java version
jmap: jmap
jstat: jstat
jinfo: jinfo
Can anyone help have a look and share some links on troubleshooting this kind of java "not responding" issue?
Possible reasons of Unable to open socket file problem:
The target PID is not a HotSpot JVM process.
This is obviously not your case, since jinfo PID works fine.
JVM is started with -XX:+DisableAttachMechanism option.
This can be also verified by jinfo PID.
Attach socket /tmp/.java_pidNNN has been deleted.
There is a common practice to clean /tmp automatically with some scheduled script. In this case you should configure the cleanup software not to delete .java_pid* files.
How to check: run lsof -p PID | grep java_pid
If it lists a socket file, but the file does not exist, then this is exactly the described problem.
Credentials of the current user (euid/egid) do not match the owner of the attach socket. Make sure you run jstack by the same user as JVM. Attach won't work if you run jstack by a different user, even if this user is root.
/tmp directory of the target process is not the same as /tmp of your shell. This may happen in the following cases:
JVM is started in a different mount namespace. Typically this happens when JVM runs in a Docker container. Running jstack from within the same container will help.
JVM is started in chroot environment. For example, LXC containers may use chroot.
How to check: run readlink -f /proc/PID/root/tmp to see if it points to /tmp or to some other directory.
Current working directory of the target JVM belongs to a file system that does not allow to change permissions. CIFS and DrvFs (WSL) are examples of such file systems.
How to check: run umask 077; touch /proc/PID/cwd/somefile.tmp, then verify that file owner is yourself, and file permissions are 600.
JVM is busy and cannot reach a safepoint. For instance, JVM is in the middle of long-running garbage collection.
How to check: run kill -3 PID. JVM should print a thread dump and heap info in its console. If JVM does not dump anything, but the process consumes almost 100% CPU or shows high I/O utilization, then this looks like the described problem.
JVM process is suspended.
How to check: run ps PID. The STAT column of alive JVM process should be Sl.
More about the internals of jstack.
There is also jattach project which is a better alternative to jstack / jmap. It can automatically handle credentials issue, it works with Docker containers, supports chroot'ed JVMs and handles uncommon file systems.
Related
I am trying to generate a thread dump of a java process being run on a Linux instance in AWS. I am using the jstack command on OpenJDK version 1.8.0. The current command I am running is sudo -u <user> jstack -l <java pid> where <user> is the user that started the JVM.
When I run this, I receive the error Unable to open socket file: target process not responding or HotSpot VM not loaded
Potential Problem:
While reading about how jstack works, I noticed that jstack is supposed to generate a socket file /tmp/.java_pidXXX in order to attach to the process. This file is not generated.
My potential solution is that if I can get the socket file to generate, hopefully jstack will be able to run properly.
I am unsure why this error is occurring, but my only idea is that this could this be some kind of permissions error to create files in the /tmp directory. I tried testing my permissions by creating text files in the /tmp directory and I was able to create text files.
How can I get this socket file to generate? Any potential solutions would be greatly appreciated.
Edit
Here I have added the command that was used to create the JVM. The command used to get this command was ps -aux | grep java
java -server -Xmx8192m -XX:+HeapDumpOnOutOfMemoryError ->XX:HeapDumpPath="/tmp" -XX:MaxPermSize=256M -Djava.awt.headless=true ->Dsling.run.modes=dynamicmedia_scene7,,
<instance_name>,samplecontent,crx3,crx3ta>r -Djava.locale.providers=CLDR,JRE,SPI -jar crx-quickstart/app/cq-?>quickstart-6.5.0-standalone-quickstart.jar start -c crx-quickstart -i >launchpad -p 4502 -Dsling.properties=conf/sling.properties
Solution:
An update for anyone who comes across this in the future. I found a solution that worked for me by changing the command that I was using to initially start the JVM. I added the flag -XX:+StartAttachListener which forces the process to generate the /tmp/.java_pidXXX socket file during the booting of the JVM.
Other tips I came across in my journey to finding this solution:
Make sure the user executing the jstack command is the same user that who ran the process that you are trying to take the thread dump of.
Also, make sure the socket file /tmp/.java_pidXXX is not being automatically cleaned up by any background cleanup processes in the /tmp directory.
Is there a way to create a heap dump on a remote machine without JDK installed?
I can't change the installation / settings and it's running on Windows.
So I have pnly access to commandline tools.
Problem is that a Java app on a remote machine freezes (no out of memory exception so -XX:-HeapDumpOnOutOfMemoryError is useless) and we need to create a dump.
-XX:+HeapDumpOnCtrlBreak
is no option too, because it's not supported anymore on JDK6+.
JMX is not allowed due to security reasons.
Any Ideas? Thank you for your help!
Edit:
Windows
No JDK
No JMX
I think I solved the problem.
You have to "patch" your JRE with some files of the JDK (the same version of course - if you are running jre6uXX you need the corresponding files from jdk6uXX )
Copy the following files:
\JDK6uXX\bin\attach.dll --> %JAVAJRE_HOME%\bin\
\JDK6uXX\bin\jmap.exe --> %JAVAJRE_HOME%\bin\
\JDK6uXX\lib\tools.jar --> %JAVAJRE_HOME%\lib\
No files are overwritten, JRE shouldn't be affected by this.
Now you can use jmap just fine to take dumps ;-)
I appreciate your help! Bye
The simplest solution is to use jmap -dump:liv,format=b,file=app.dump on the command line. You can use jps -lvm to find the process id.
An alternative is to connect to it to jvisualvm This will take the dump and analyse it for you. You can also use this tool to read a dump written by jmap so you may end up using it anyway.
Where jvisualvm struggles is for large heap dumps i.e. more than about half you main memory size. I have found using YourKit to handle larger dumps and also give more useful information. An evaluation license might be all you need to diagnose this.
jmx is not allowed due to security reasons
In that case, you can't do this remotely, unless you use YourKit or some other commercial profiler.
You have start your application with jmx console enabled in a port to debug your application. Execute jconsole and connect to the port which you have enabled for debugging. You can also use of jmap to collect heapdump.
JProfiler has a command line utility bin/jpdump that can take an HPROF heap dump. There is no need to install JDK. There is also no need to run the GUI installer of JProfiler, just extract the ZIP distribution and execute jpdump on the command line.
Disclaimer: My company develops JProfiler.
Update 2016-06-23
As of JProfiler 9.2, jpdump and jpenable run with Java 6 as well.
You could use jvisualvm, just enable jmx port and connect to your application, then you will be able to generate a heap file.
You can do that by adding the following parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=8484
-Dcom.sun.management.jmxremote.ssl=false
Then You need to add your tomcat process manually, So right click on you localhost node -> Add JMX Connection -> type your port -> OK.
Your tomcat process will be listed in under localhost node.
jmap -dump:format=b,file=snapshot.jmap
process-pid
Regardless of how the Java VM was started, the jmap tool will produce a head dump snapshot, in the above example in a file called snapshot.jmap. The jmap output files should contain all the primitive data, but will not include any stack traces showing where the objects have been created.
I had to run jmap in order to take heap dump of my process. but jvm returned:
Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
So I used the -F:
./jmap -F -dump:format=b,file=heap.bin 10330
Attaching to process ID 10331, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.51-b03
Dumping heap to heap.bin ...
Using -F is allright for taking heap dump?
I am waiting 20 minutes and not finished yet. Any ideas why?
jmap vs. jmap -F, as well as jstack vs. jstack -F use completely different mechanisms to communcate with the target JVM.
jmap / jstack
When run without -F these tools use Dynamic Attach Mechanism. This works as follows.
Before connecting to Java process 1234, jmap creates a file .attach_pid1234 at the working directory of the target process or at /tmp.
Then jmap sends SIGQUIT to the target process. When JVM catches the signal and finds .attach_pid1234, it starts AttachListener thread.
AttachListener thread creates UNIX domain socket /tmp/.java_pid1234 to listen to commands from external tools.
For security reasons when a connection (from jmap) is accepted, JVM verifies that credentials of the socket peer are equal to euid and egid of JVM process. That's why jmap will not work if run by different user (even by root).
jmap connects to the socket, and sends dumpheap command.
This command is read and executed by AttachListener thread of the JVM. All output is sent back to the socket. Since the heap dump is made in-process directly by JVM, the operation is really fast. However, JVM can do this only at safepoints. If a safepoint cannot be reached (e.g. the process is hung, not responding, or a long GC is in progress), jmap will timeout and fail.
Let's summarize the benefits and the drawbacks of Dynamic Attach.
Pros.
Heap dump and other operations are run collaboratively by JVM at the maximum speed.
You can use any version of jmap or jstack to connect to any other version of JVM.
Cons.
The tool should be run by the same user (euid/egid) as the target JVM.
Can be used only on live and healthy JVM.
Will not work if the target JVM is started with -XX:+DisableAttachMechanism.
jmap -F / jstack -F
When run with -F the tools switch to special mode that features HotSpot Serviceability Agent. In this mode the target process is frozen; the tools read its memory via OS debugging facilities, namely, ptrace on Linux.
jmap -F invokes PTRACE_ATTACH on the target JVM. The target process is unconditionally suspended in response to SIGSTOP signal.
The tool reads JVM memory using PTRACE_PEEKDATA. ptrace can read only one word at a time, so too many calls required to read the large heap of the target process. This is very and very slow.
The tool reconstructs JVM internal structures based on the knowledge of the particular JVM version. Since different versions of JVM have different memory layout, -F mode works only if jmap comes from the same JDK as the target Java process.
The tool creates heap dump itself and then resumes the target process.
Pros.
No cooperation from target JVM is required. Can be used even on a hung process.
ptrace works whenever OS-level privileges are enough. E.g. root can dump processes of all other users.
Cons.
Very slow for large heaps.
The tool and the target process should be from the same version of JDK.
The safepoint is not guaranteed when the tool attaches in forced mode. Though jmap tries to handle all special cases, sometimes it may happen that target JVM is not in a consistent state.
Note
There is a faster way to take heap dumps in forced mode. First, create a coredump with gcore, then run jmap over the generated core file. See the related question.
I just found that jmap (and presumably jvisualvm when using it to generate a heap dump) enforces that the user running jmap must be the same user running the process attempting to be dumped.
in my case the jvm i want a heap dump for is being run by linux user "jboss". so where sudo jmap -dump:file.bin <pid> was reporting "Unable to open socket:", i was able to grab my heap dump using:
sudo -u jboss jmap -dump:file.bin <pid>
If your application is runing as a systemd service.You should open service file that under /usr/lib/systemd/system/ and named by your service name. Then check whether privateTmp attribute is true.
If it is true,you shoud change it to false,then refresh service by command as follow:
systemctl daemon-reload
systemctl restart [servicename]
If you want runing jmap/jcmd before restart, you can make use of the execStop script in the service file. Just put command in it and to execute systemctl stop [service name]
Just like ben_wing said, you can run with:
sudo -u jboss-as jmap -dump:file.bin <pid>
(in my case the user is jboss-as, but yours could be jboss or some other.)
But it was not enough, because it asked me for a password ([sudo] password for ec2-user:), although I could run sudo without prompting me for a password with other commands.
I found the solution here, and I just needed to add another sudo first:
sudo sudo -u jboss-as jmap -dump:file.bin <pid>
It works with other commands like jcmd and jinfo too.
It's usually solved with -F.
As stated in the message:
The -F option can be used when the target process is not responding
I encountered a situation where the full GC made it impossible to execute the command.
I start up a jvm process on a linux platform, like tomcat web container.
start up command-line :
nohup sh > nohup.out 2>&1 &
When i take a "kill -3 " command, the thread dump information will output to the nohup.out file.
How can i redirect the output to a file singly ?
The Sun HotSpot JVM it's own toolkit from jdk 1.5 which is called jstack, it's very useful by using Sun HotSpot JVM, but the JRockit JVM and HP HotSpot JVM, IBM J9 VM are unsuited.
How can wirte a good generality toolkit to redirect the thread dump information to a single file, and not be stdout file ?
It is part of the standard unix process protection policy that after the start of a program no more redirections for its streams can be done without the active help of that program.
But if you don't care that the threaddump is still in nohup.out and additionally in some other file you can do it like this:
tail -f nohup.out > my_new_threaddump &
kill -3 $pid_target_process
kill %+
Of interest is the related kill -3 to get java thread dump, including Vadzim answer:
-XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=jvm.log
However, this isn't (or at least didn't use to be) compatible with JRockit etc - e.g. see http://blog.eisele.net/2011/01/running-glassfish-with-jrockit.html
In Linux (CentOS 5.4), how do you force a process to stop locking a file system without killing the process?
I am trying to get my Java Webstart Application, running locally, to eject a CD. I do not have this problem if I am just browsing through the files using a JFileChooser, but once I read the contents of a file, I can no longer eject the CD...even after removing ALL references to any files.
Hitting the eject button will give the error (Title - "Cannot Eject Volume"):
"An application is preventing the volume 'volume name' from being ejected"
Thus, my goal is to tell the process to stop targeting the CD-ROM in order to free it up.
Thank you for any help or direction!!
Attempted Fix:
-running the commands:
sudo umount -l /media/Volume_Name //-l Lazy Unmount forces the unmount
sudo eject
Problem:
When a new CD is inserted, it is no longer mounted automatically probably because the process is still "targeting" it.
Assuming that lsof lists your java application as the only application accessing the file, you might try forcing a garbage collection (using jvisualvm or a similar tool) to see if that frees it up.