I have a basic java server app that has 100 worker threads that do simple HEAD requests on urls. I'm using HttpClient 4.x for this.
A few minutes into the run my program just freezes for a couple minutes and I cannot figure out why. Check out the screen shot of what visual vm monitor reports. You can see it flatline. During this time I'm unable to get a good thread dump and visual vm just freezes until it's unblocked. Does anyone have any ideas on what I can do to try and start debugging this guy?
Visual VM: http://tinypic.com/view.php?pic=2i915bs&s=7
Here is the output when I tried to take a jstack dump while it was frozen:
jstack -F 4325
Attaching to process ID 4325, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16.3-b01
Deadlock Detection:
No deadlocks found.
Thread 4557: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
I've seen several bug reports about jstack on Linux with a similar trace:
JVM Bug Id: 6494722 (is supposed to be fixed)
Ubuntu Bug #597098 (this one is not)
Do you get the same result with a kill -3 <pid>?
Very likley due to too much memory usage causing GC. Add the params to java:
-verbosegc -XX:+PrintGCDetails
And see if you notice anything obvious in the output/logs
What worked for me was running jstack as the process owner without -F.
Related
macOS Monterey version is 12.5
and jdk version is 11
I was going with sudo jhsdb jmap --heap --pid xxxx to analyze java process. Then I got the error followed
Password:
Attaching to process ID 47413, please wait...
ERROR: attach: task_for_pid(47413) failed: '(os/kern) failure' (5)
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process. Could be caused by an incorrect pid or lack of privileges.
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process. Could be caused by an incorrect pid or lack of privileges.
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.bsd.BsdDebuggerLocal$BsdDebuggerLocalWorkerThread.execute(BsdDebuggerLocal.java:169)
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.bsd.BsdDebuggerLocal.attach(BsdDebuggerLocal.java:283)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:672)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerDarwin(HotSpotAgent.java:660)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:342)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:369)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:538)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process. Could be caused by an incorrect pid or lack of privileges.
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.bsd.BsdDebuggerLocal.attach0(Native Method)
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.bsd.BsdDebuggerLocal$1AttachTask.doit(BsdDebuggerLocal.java:274)
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.bsd.BsdDebuggerLocal$BsdDebuggerLocalWorkerThread.run(BsdDebuggerLocal.java:144)
I'v tried use jdk14,18,and didn't work out for me.
so any solution?
My application uses javafx 11.0.1 and is shipped bundled with a jlinked version of openjdk 11. It runs fine for the vast majority of the users but few of them are getting this stack:
Exception in thread "WindowsNativeRunloopThread" java.lang.NoSuchMethodError: <init>
at com.sun.glass.ui.win.WinApplication.staticScreen_getScreens(Native Method)
at com.sun.glass.ui.Screen.initScreens(Screen.java:412)
at com.sun.glass.ui.Application.lambda$run$1(Application.java:152)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
at java.base/java.lang.Thread.run(Unknown Source)
Exception in thread "JavaFX Application Thread" java.lang.NullPointerException
at com.sun.javafx.tk.quantum.QuantumToolkit.assignScreensAdapters(QuantumToolkit.java:695)
at com.sun.javafx.tk.quantum.QuantumToolkit.runToolkit(QuantumToolkit.java:313)
at com.sun.javafx.tk.quantum.QuantumToolkit.lambda$startup$10(QuantumToolkit.java:258)
at com.sun.glass.ui.Application.lambda$run$1(Application.java:153)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
at java.base/java.lang.Thread.run(Unknown Source)
I found some discussion related to the same exception but regarding Maven+Eclipse, here and here. The issue is very similar, user reporting it, do have other java installations and uninstalling them solves the issue, so basically leaving my bundled openjdk as the only option the application starts, but if there is another Java installed on the system, the wrong .dll is picked up and the application crashes with the above stacktrace.
I tried the suggested java.library.path workaround but users are saying it doesn't solve. Unfortunately I cannot reproduce it myself, any idea on how to solve it or what to ask the users reporting it?
EDIT: we fixed the exe generated by lauch4j here and the bash script here. The idea is basically to restrict/change the PATH env variable to avoid the wrong dll being picked up.
We fixed the exe generated by lauch4j here and the bash script here. The idea is basically to restrict/change the PATH env variable to avoid the wrong dll being picked up.
I am using Ubuntu 12.04 with jenkins as a continuous integration server and have SonarQube installed and setup. Sonar worked for a large amount of time, but recently the service will shutdown immediately after starting the service. I am using Sonar 5.3 and my server has 4GB RAM where most of the time only 1-2GB are occupied.
The sonar.log says:
WrapperSimpleApp: Encountered an error running main: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149)
at java.lang.StringCoding.decode(StringCoding.java:193)
at java.lang.String.<init>(String.java:426)
at java.lang.String.<init>(String.java:491)
at java.io.UnixFileSystem.list(Native Method)
at java.io.File.list(File.java:1122)
at java.io.File.listFiles(File.java:1207)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1645)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
at org.apache.commons.io.FileUtils.deleteQuietly(FileUtils.java:1566)
at org.sonar.application.PropsBuilder.initTempDir(PropsBuilder.java:102)
at org.sonar.application.PropsBuilder.build(PropsBuilder.java:73)
at org.sonar.application.App.main(App.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
at java.lang.Thread.run(Thread.java:745)
<-- Wrapper Stopped
I googled for solutions but they suggest to increase the heap-size when starting the .jar file via command line, but the file that defines the sonar-service in /etc/init.d doesnt even run a .jar file.
This looks like SONAR-7125, fixed in SonarQube 5.4. Workaround is to delete the temp folder manually before restart.
I'm trying to extract a java heap dump from a native dump taken by WER when it crashed:
jstack -m -l "c:\Program Files\Java\jre6\bin\java.exe" WER.tmp.hdmp
but I get the following exception:
Attaching to core c:\Users\xxx\Desktop\WER.tmp.hdmp from executable c:\Program Files\Java\jre6\bin\java.exe, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypes" in any of the known library names (jvm.dll, jvm_g.dll)
at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:389)
at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:104)
at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:85)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:565)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:494)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:348)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:169)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Debugger attached successfully.
jstack requires a java VM process/core!
I've double checked that:
I'm running the same version of java as the system that the crash occurred on
The bitness of Windows (32) matches the system that the crash occurred on
jvm.dll is in the path
windbg is installed
I get exactly the same error if I try to extract a java heap dump using jmap.
Does anyone know what could be going wrong?
I have a java application running on Solaris. This application regularily launches external processes using Runtime.exec. It seems that after a while, having successfully launched such processes many time over, a launching of a process will hang. A thread dump taken at this point (and several minutes later) reveals that java.lang.UNIXProcess.forkAndExec is "stuck". Following is the top of the relevant stack trace taken from the thread dump:
"Thread-85305" prio=3 tid=0x0000000102aae800 nid=0x21499 runnable [0x7fffffff2a3fe000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
at java.lang.ProcessBuilder.start(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
I have read through some forums where others have experienced forAndExec throwing an IOException due to not enough space or not enough memory, but I'm not getting this error here. I'm now waiting to get the results of pstack in the hope that it will reveal more information.
Does anyone have any idea on how to resolve this issue?
thanks,
Mike
Installing the Sun Alert Patch Cluster will do the trick.
As the thread live as long as your executable is running, maybe it's only your external executable which is hanging.
You should try to find the parameters passed to your executable and try to launch it manually.
Is stdout and stderr from the process being consumed? You might be looking at the results of a full buffer.
You can test this by adding output redirection to a tempfile for the command that is spawned.