Develop Reference

Cannot use storm command line interface to kill topology

I have a storm cluster, my storm version is 0.10.1, and it run well. I can kill topology using ui in browser which is good. But sometime I want to just kill topology by shell. When I run this: bin/storm kill topology_name, I get errors.
Exception in thread "main" NotAliveException(msg:com_adsame_mydmp is not alive)
at backtype.storm.generated.Nimbus$killTopologyWithOpts_result$killTopologyWithOpts_resultStandardScheme.read(Nimbus.java:7482)
at backtype.storm.generated.Nimbus$killTopologyWithOpts_result$killTopologyWithOpts_resultStandardScheme.read(Nimbus.java:7468)
at backtype.storm.generated.Nimbus$killTopologyWithOpts_result.read(Nimbus.java:7410)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_killTopologyWithOpts(Nimbus.java:283)
at backtype.storm.generated.Nimbus$Client.killTopologyWithOpts(Nimbus.java:269)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at backtype.storm.command.kill_topology$_main.doInvoke(kill_topology.clj:27)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at backtype.storm.command.kill_topology.main(Unknown Source)
But I can see this topology activate when I execute bin/storm list.
Topology_name Status Num_tasks Num_workers Uptime_secs
-------------------------------------------------------------------
topology_name ACTIVE 73 4 423894
Any idea ? Thanks.

"can not get class data for sun/java2d/Disposer"

I have used a script successfully in the past and the JDK did not change, but for some reason in this instance I was unable to convert a core dump to a hprof;
Attaching to core my.core from executable /opt/jdk1.8.0_31/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.31-b07
Dumping heap to my.hprof ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at sun.tools.jmap.JMap.runTool(JMap.java:201)
at sun.tools.jmap.JMap.main(JMap.java:130)
Caused by: sun.jvm.hotspot.utilities.AssertionFailure: can not get class data for sun/java2d/Disposer$$Lambda$10x00000001002e8428
at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeInstance(HeapHprofBinWriter.java:803)
at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:95)
at sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:353)
at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:171)
at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:433)
at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:62)
at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:83)
... 6 more
The Partial heap won't load with visualvm.
What could cause this?
Is this a known bug?
Is there a workaround?

This is a known bug and has been fixed starting with Java 8 Update 60.
See https://bugs.openjdk.java.net/browse/JDK-8044416
Java 8 ticket: https://bugs.openjdk.java.net/browse/JDK-8077507

NullPointerException while getting thread dump using jstack

I'm trying to debug a CPU spike issue. For fetching thread dump, I'm using Jstack which is giving below mentioned error:
Thread 1780: (state = IN_JAVA)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:78)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Has anyone seen this issue before ?
JDK version: 1.7

unable to convert java core dump to hprof with jmap

When trying to convert java core dump with jmap to hprof, i get this exection:
Attaching to core core.26045 from executable /usr/java/jdk1.6.0_21/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 17.0-b16
Dumping heap to core.hprof ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Caused by: sun.jvm.hotspot.debugger.UnmappedAddressException
at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readOopHandle(LinuxDebuggerLocal.java:436)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getOopHandleAt(LinuxAddress.java:120)
at sun.jvm.hotspot.oops.Oop.getKlassForOopHandle(Oop.java:237)
at sun.jvm.hotspot.oops.ObjectHeap.newOop(ObjectHeap.java:362)
at sun.jvm.hotspot.utilities.HashtableEntry.literal(HashtableEntry.java:53)
at sun.jvm.hotspot.memory.SymbolTable.symbolsDo(SymbolTable.java:106)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeSymbols(HeapHprofBinWriter.java:838)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:396)
at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:56)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)
... 6 more
I used command:
jmap -dump:format=b,file=core.hprof /usr/java/jdk1.6.0_21/bin/java core.26045
Any help on this?
I am trying to create hprof in same box, where i got the core dump.

Unable to get thread dump? Any ideas why my app blocks?

I have a basic java server app that has 100 worker threads that do simple HEAD requests on urls. I'm using HttpClient 4.x for this.
A few minutes into the run my program just freezes for a couple minutes and I cannot figure out why. Check out the screen shot of what visual vm monitor reports. You can see it flatline. During this time I'm unable to get a good thread dump and visual vm just freezes until it's unblocked. Does anyone have any ideas on what I can do to try and start debugging this guy?
Visual VM: http://tinypic.com/view.php?pic=2i915bs&s=7
Here is the output when I tried to take a jstack dump while it was frozen:
jstack -F 4325
Attaching to process ID 4325, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16.3-b01
Deadlock Detection:
No deadlocks found.
Thread 4557: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)

I've seen several bug reports about jstack on Linux with a similar trace:
JVM Bug Id: 6494722 (is supposed to be fixed)
Ubuntu Bug #597098 (this one is not)
Do you get the same result with a kill -3 <pid>?

Very likley due to too much memory usage causing GC. Add the params to java:
-verbosegc -XX:+PrintGCDetails
And see if you notice anything obvious in the output/logs

What worked for me was running jstack as the process owner without -F.