I have a problem that appears in pseudo-distributed mode, but not in standalone mode, and I'm hoping to scratch up some ideas on how to debug this.
Some of my mapper tasks are returning code 143. I'd love to drop a breakpoint on System.exit() and see who's calling this why, but I have to get the debugger running on that mapper.
I can get the task tracker up in the debugger by modifying my bin/hadoop script and remotely connecting to localhost:5000:
...
elif [ "$COMMAND" = "tasktracker" ] ; then
CLASS=org.apache.hadoop.mapred.TaskTracker
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_TASKTRACKER_OPTS"
# TBMark!
HADOOP_OPTS="$HADOOP_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,address=5000,server=y,suspend=n"
...and I can get the first mapper (or by a minor tweak, reducer) into Eclipse by adding this into my conf/mapred-site.xml and remotely connecting to localhost:5001:
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xdebug -Xrunjdwp:transport=dt_socket,address=5001,server=y,suspend=y</value>
</property>
My problem is that the failure happens at random and not on the first mapper.
Unsatisfactory ideas that come to mind include:
Somehow replace System.exit() with my own method that does a stack trace. (How does one hook a system call?)
Just keep trying to debug the mappers one by one and run each one to completion before debugging the next. (It might work...)
Track down every last place in hadoop that calls System.exit() and write a distinct signature to a log. (Yuck)
Make the debugger port number variable such that, if I can guess which one is going to fail and the delay doesn't make the bug go away, I can attach to that jvm and debug it. (Many if's, and I don't know any way to make this variable in the .xml file.)
If failure can be predicted to happen on a certain attempt, break the task tracker just before the jvm launch and hand edit the script file. (Desperate times call for desperate measures)
Any suggestions or ideas for how to make my bad ideas above work?
You could try to re-run a failed map task with the IsolationRunner
In case it fails again you should be able to add the debug options!
Related
I started to work with JMH lately and wondered if there is a possible way to debug it.
First thing I did was try to debug it like any other program, but it threw "No transports initialized", so I couldn't debug it in the old fashioned way.
Next thing I did is to try to search on the internet and found someone who said that you need to put the forks to 0, tried it and it worked.
Unfortunately, I couldn't really understand why the forks are impacting the debug, or how forks impact the way I see things in the JMH.
All I know so far is that when you put .forks(number) on the OptionBuilder it says on how many process the benchmark will run. But if I put .forks(3) it's running each #Benchmark method on 3 process async?
An explanation about the .forks(number), .threads(number) how they are changing the way benchmarks run and how they impact debug would really clear things.
So normal JMH run has two processes running at any given time. The first ("host") process handles the run. The second ("forked") process is the process that runs one trial of a given benchmark -- achieving isolation. Requesting N forks either via annotation or command line (which takes precedence over annotations) makes N consecutive invocations for forked process. Requesting zero forks runs the workload straight in the hosted process itself.
So, there are two things to do with debugging:
a) Normally, you can attach to the forked process, and debug it there. It helps to configure workload to run longer to have plenty of time to attach and look around. The forked process usually has ForkedMain as its entry point, visible in jps.
b) If the thing above does not work, ask -f 0, and attach to the host process itself.
This was quite a pain in the soft side to get to work (now that I wrote this, it sounds trivial)
First of all I was trying to debug DTraceAsmProfiler (perfasm, but for Mac), using gradle.
First of all, I have a gradle task called jmh that looks like this:
task jmh(type: JavaExec, dependsOn: jmhClasses) {
// the actual class to run
main = 'com.eugene.AdditionPollution'
classpath = sourceSets.jmh.compileClasspath + sourceSets.jmh.runtimeClasspath
// I build JMH from sources, thus need this
repositories {
mavenLocal()
maven {
url '/Users/eugene/.m2/repository'
}
}
}
Then I needed to create a Debug Configuration in Intellij (nothing un-usual):
And then simply run:
sudo gradle -Dorg.gradle.debug=true jmh --debug-jvm
When I use remote debugger in IntelliJ to debug a Java application on server, it stops on breakpoints successfully but when I try to evaluate any expressions or variables it hangs and shows nothing (usually with "collecting data" message). From that point I can't even continue stepping through code anymore. I have to click resume so it at least runs, but it will never stop at other breakpoints too until I restart the debug session and usually even the Java application being debugged.
I can step through the code after stopping on breakpoint, also I see the variables in the debugger panel, it only starts to behave weirdly when I try to evaluate an expression or add a watcher. Then it stops working and restart of the debugger and the app is needed.
Did anybody experience something similar? Is it IntelliJ or server problem?
(sorry this is so vaguely described, but I have no idea what to share or what the problem might be)
Expression evaluation during remote debug need more data to be sync then the other operations ( breakpoints add/remove, step managements ecc ).
So this kind of issue should be just related to:
a slow connection
an huge complexity and amount of data involved into the operation to execute in the remote server
My apologizes if this question is asked before but i couldn't find a correct match with my situation.
I have some batch files, they are always running because of the java program running inside them. However, sometimes they crash and since i am not monitoring them, some amount of time passes till i restart them.
I want to ask that is there a way to monitor those batch files, or how can i restart those batch files even if they crash. I am running those files on Windows XP.
Thanks
Have you tried this?
http://www.ghacks.net/2009/08/31/monitor-and-restart-crashed-windows-processes/
You can start a batch file from another batch file in a loop. When it is started, the loop waits. Once it crashes - the loop starts it over.
Replace each call to java with a goto loop that calls java via cmd.exe. Conditionally loop back only if the java program exited with an error.
.
.
:javaLoop
cmd /c java yourJavaProg || goto :javaLoop
.
.
You might want to guard against an endless rapid loop in the case of a complete failure by adding a counter to limit the number of times it restarts. Another possibility is to note the time and only restart if a minimum amount of time has elapsed since the last restart. Or perhaps a combination of both.
EDIT - On second thought, you may not need the CMD /C. It may work fine just by invoking java directly and conditionally looping upon error.
Some CPU intensive routines get dramatically slower when run through a debugger. Why is this?
Currently I'm just using IntelliJ to step through code running in JBoss. When I start JBoss, I use these options:
set JAVA_OPTS=-Xms512m -Xmx1024m -XX:MaxPermSize=256m -Xdebug -Xrunjdwp:transport=dt_socket,address=5005,server=y,suspend=n %JAVA_OPTS%
Is there a way to speed up the execution? Or to speed up certain method executions that I don't need to step through?
Update: Seems if I don't step over/into the CPU intensive routines (ie: Just run til a breakpoint set right after the routine), then the execution time is as if not in a debugger.
Some CPU intensive routines get dramatically slower when run through a debugger. Why is this?
Because the JITter won't optimize code as much (often, not at all) when debugging is enabled.
It also depends on the "breakpoints-style". E.g. having watchpoints on variables or putting breakpoints on interface level (debugger will stop on all method-implementations when they're executed) often dramatically slows down the process time.
When debugging, in addition to running your application, you are also running a debugger.
The code is compiled in debug mode with metadata symbols about local variables and other source-level information. The debugger reads to know which line of source code corresponds with the current instruction. The process is called symbolic debugging. The stored symbols increase code size and interpreting them increases execution time.
Some debuggers actually interpret the code on the fly, which is almost always a major performance hit.
More information about Java's debug compilation mode, which is performed by javac and includes debug information in class files: Java Language Compiler Options.
For example: -g generates all debugging information, including local variables.
You do need to consider that another program -- the debugger -- is hooked into your program and is watching it for things like exceptions. It's also monitoring the current line in order to react to breakpoints or user requested interruptions (like a pause request or watch condition).
Top Tip: in IDEA you can use ALT+F9 to run to where you have the cursor placed rather than set an extra breakpoint.
I have found anecdotally that debugging becomes very slow in IDEA if you are walking through code where there is a lot of data accessible from the stack. Don't forget, IDEA collects this data (anything currently in the lexical scope) and presents it up to you as a tree of objects to browse whether you are "watching" or not and does this at every subsequent step (maybe it re-creates the tree each time?).
This is especially apparent when, for example, there is a large collection as an instance variable of the "current" object.
Debugging the optimized code produced by the JIT would be very hard, because there isn't a direct relationship between a range of native instructions and a line of Java code like there is a relationship between a range of Java bytecode and a line of Java code.
So breaking into a function in the debugger forces the JVM to deoptimize the method you are stepping through. Hotspot doesn't generate native code at all and just interprets the bytecode of the method.
Prior to JDK 1.4.1 starting with debugging enabled forced the JVM to only use the interpreter: http://java.sun.com/products/hotspot/docs/whitepaper/Java_Hotspot_v1.4.1/Java_HSpot_WP_v1.4.1_1002_3.html#full
If you use Java 5, the parameter for debugging is :
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=
and before Java 5
-Xdebug -Xrunjdwp:transport=dt_socket,address=5005,server=y,suspend=n
anyone have any experience using this?
if so, is it worth while?
I just used jdb for the first time yesterday and am really pleased with the results. You see, I program in Eclipse on my laptop, then deploy to a VM to make sure the whole shebang still works. Very occasionaly, I'll have to work on something that gets executed standalone, as a commandline. These things sometimes need debugging.
This has always been a problem, because I don't want to go to the trouble of installing Eclipse on the VM (it's slow enough already!), yet I don't know of an easy way to get it to connect to my commandline-running class before it finishes running.
jdb to the rescue! It works a treat - small and functional, almost to the point where it is bare... this forces you to apply your mind more than you apply the tool (like I said here).
Make sure to print out the reference (solaris, windows, java 1.5 - I think they're all about the same, really) and have your source code open and browsable on your second screen. I hope you have a second screen, or you'll be alt-tabbing a lot.
Assume your program is started by the following command:
java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=<port> <class>
You can attach to this process by jdb:
jdb -attach <port>
In some cases you need to use the following command .
jdb -sourcepath \.src -connect com.sun.jdi.SocketAttach:hostname=localhost,port= <port>
JDB is incredibly difficult to use. Placing System.outs or using an IDE debugger will produce better results. And for the more interesting features (e.g. tracking threads, heap size, etc.), you can get the information graphically with the JConsole tool.