I have a Java application that uses a C++ DLL via JNA. The C++ DLL is proprietary, therefore, I cannot share the code unless I can make a simplified reproducible example. It is not straight forward to make a reproducible example until I further debug.
The application crashes sporadically with the error message Java Result: -1073740940. I am running the Java application from Netbeans, although it crashes without Netbeans. Since there is no hs_err_.log, I guess crash is in the C++ layer. How can I begin debug this crash?
The "Java Result" output from Netbeans simply tells you the exit code of the java program. You could generate the same with a System.exit(-1073740940);. A successful program exits with a code of 0. Anything else is a failure that requires documentation to interpret.
You have not given us any indication what DLL you are using, so the only information we have to work with is this exit code. Converting that int to hex digits results in 0xc0000374 which you can enter into your favorite search engine and find out is a Heap Corruption Exception. Some examples are provided but in general this means you are accessing non-allocated native memory.
Without having any idea what code you're using, I would guess you're doing something wrong with native memory, invoking native functions, or incorrectly manipulating pointers or handles somewhere in your application.
You should start by looking closely at arguments to native functions. Type mapping could be a problem if the number of bytes is mismatched. Investigate any Pointer-based arguments to native functions, including ByReference arguments. Trace back in the code and find when/how these Pointers were associated with native-allocated memory. If it was never allocated, that's one possibility for the problem. If it was allocated, see if you can find a point where that memory was freed, possibly by a different native function.
The root cause of the crash was heap corruption in the C++ layer. If a random crash occurs due to heap corruption, sometimes, it is complicated to pinpoint the cause of crash because the crash can happen later, when the program tries to manipulate the corrupted memory. Hence, it is also complicated to provide an SSCCE, especially when we work on the proprietary legacy code.
How I debugged this crash:
Reproduction: Try to find a consistent use case for the crash. If the crash is random then try to figure out a set of user actions that always leads to the crash.
Assumption: Guess which feature/component contains the crash.
Validation: Make sure that crash is not happening when you disable this feature/component.
Verification: Skimm through and slice the code. Review the small piece of code.
Documentation: Write everything.
Daniel's answer was very helpful in fixing this crash!
Related
This is intended to be a general-purpose question to assist new programmers who have a problem with a program, but who do not know how to use a debugger to diagnose the cause of the problem.
This question covers three classes of more specific question:
When I run my program, it does not produce the output I expect for the input I gave it.
When I run my program, it crashes and gives me a stack trace. I have examined the stack trace, but I still do not know the cause of the problem because the stack trace does not provide me with enough information.
When I run my program, it crashes because of a segmentation fault (SEGV).
A debugger is a program that can examine the state of your program while your program is running. The technical means it uses for doing this are not necessary for understanding the basics of using a debugger. You can use a debugger to halt the execution of your program when it reaches a particular place in your code, and then examine the values of the variables in the program. You can use a debugger to run your program very slowly, one line of code at a time (called single stepping), while you examine the values of its variables.
Using a debugger is an expected basic skill
A debugger is a very powerful tool for helping diagnose problems with programs. And debuggers are available for all practical programming languages. Therefore, being able to use a debugger is considered a basic skill of any professional or enthusiast programmer. And using a debugger yourself is considered basic work you should do yourself before asking others for help. As this site is for professional and enthusiast programmers, and not a help desk or mentoring site, if you have a question about a problem with a specific program, but have not used a debugger, your question is very likely to be closed and downvoted. If you persist with questions like that, you will eventually be blocked from posting more.
How a debugger can help you
By using a debugger you can discover whether a variable has the wrong value, and where in your program its value changed to the wrong value.
Using single stepping you can also discover whether the control flow is as you expect. For example, whether an if branch executed when you expect it ought to be.
General notes on using a debugger
The specifics of using a debugger depend on the debugger and, to a lesser degree, the programming language you are using.
You can attach a debugger to a process already running your program. You might do it if your program is stuck.
In practice it is often easier to run your program under the control of a debugger from the very start.
You indicate where your program should stop executing by indicating the source code file and line number of the line at which execution should stop, or by indicating the name of the method/function at which the program should stop (if you want to stop as soon as execution enters the method). The technical means that the debugger uses to cause your program to stop is called a breakpoint and this process is called setting a breakpoint.
Most modern debuggers are part of an IDE and provide you with a convenient GUI for examining the source code and variables of your program, with a point-and-click interface for setting breakpoints, running your program, and single stepping it.
Using a debugger can be very difficult unless your program executable or bytecode files include debugging symbol information and cross-references to your source code. You might have to compile (or recompile) your program slightly differently to ensure that information is present. If the compiler performs extensive optimizations, those cross-references can become confusing. You might therefore have to recompile your program with optimizations turned off.
I want to add that a debugger isn't always the perfect solution, and shouldn't always be the go-to solution to debugging. Here are a few cases where a debugger might not work for you:
The part of your program which fails is really large (poor modularization, perhaps?) and you're not exactly sure where to start stepping through the code. Stepping through all of it might be too time-consuming.
Your program uses a lot of callbacks and other non-linear flow control methods, which makes the debugger confused when you step through it.
Your program is multi-threaded. Or even worse, your problem is caused by a race condition.
The code that has the bug in it runs many times before it bugs out. This can be particularly problematic in main loops, or worse yet, in physics engines, where the problem could be numerical. Even setting a breakpoint, in this case, would simply have you hitting it many times, with the bug not appearing.
Your program must run in real-time. This is a big issue for programs that connect to the network. If you set up a breakpoint in your network code, the other end isn't going to wait for you to step through, it's simply going to time out. Programs that rely on the system clock, e.g. games with frameskip, aren't much better off either.
Your program performs some form of destructive actions, like writing to files or sending e-mails, and you'd like to limit the number of times you need to run through it.
You can tell that your bug is caused by incorrect values arriving at function X, but you don't know where these values come from. Having to run through the program, again and again, setting breakpoints farther and farther back, can be a huge hassle. Especially if function X is called from many places throughout the program.
In all of these cases, either having your program stop abruptly could cause the end results to differ, or stepping through manually in search of the one line where the bug is caused is too much of a hassle. This can equally happen whether your bug is incorrect behavior, or a crash. For instance, if memory corruption causes a crash, by the time the crash happens, it's too far from where the memory corruption first occurred, and no useful information is left.
So, what are the alternatives?
Simplest is simply logging and assertions. Add logs to your program at various points, and compare what you get with what you're expecting. For instance, see if the function where you think there's a bug is even called in the first place. See if the variables at the start of a method are what you think they are. Unlike breakpoints, it's okay for there to be many log lines in which nothing special happens. You can simply search through the log afterward. Once you hit a log line that's different from what you're expecting, add more in the same area. Narrow it down farther and farther, until it's small enough to be able to log every line in the bugged area.
Assertions can be used to trap incorrect values as they occur, rather than once they have an effect visible to the end-user. The quicker you catch an incorrect value, the closer you are to the line that produced it.
Refactor and unit test. If your program is too big, it might be worthwhile to test it one class or one function at a time. Give it inputs, and look at the outputs, and see which are not as you're expecting. Being able to narrow down a bug from an entire program to a single function can make a huge difference in debugging time.
In case of memory leaks or memory stomping, use appropriate tools that are able to analyze and detect these at runtime. Being able to detect where the actual corruption occurs is the first step. After this, you can use logs to work your way back to where incorrect values were introduced.
Remember that debugging is a process going backward. You have the end result - a bug - and find the cause, which preceded it. It's about working your way backward and, unfortunately, debuggers only step forwards. This is where good logging and postmortem analysis can give you much better results.
This is intended to be a general-purpose question to assist new programmers who have a problem with a program, but who do not know how to use a debugger to diagnose the cause of the problem.
This question covers three classes of more specific question:
When I run my program, it does not produce the output I expect for the input I gave it.
When I run my program, it crashes and gives me a stack trace. I have examined the stack trace, but I still do not know the cause of the problem because the stack trace does not provide me with enough information.
When I run my program, it crashes because of a segmentation fault (SEGV).
A debugger is a program that can examine the state of your program while your program is running. The technical means it uses for doing this are not necessary for understanding the basics of using a debugger. You can use a debugger to halt the execution of your program when it reaches a particular place in your code, and then examine the values of the variables in the program. You can use a debugger to run your program very slowly, one line of code at a time (called single stepping), while you examine the values of its variables.
Using a debugger is an expected basic skill
A debugger is a very powerful tool for helping diagnose problems with programs. And debuggers are available for all practical programming languages. Therefore, being able to use a debugger is considered a basic skill of any professional or enthusiast programmer. And using a debugger yourself is considered basic work you should do yourself before asking others for help. As this site is for professional and enthusiast programmers, and not a help desk or mentoring site, if you have a question about a problem with a specific program, but have not used a debugger, your question is very likely to be closed and downvoted. If you persist with questions like that, you will eventually be blocked from posting more.
How a debugger can help you
By using a debugger you can discover whether a variable has the wrong value, and where in your program its value changed to the wrong value.
Using single stepping you can also discover whether the control flow is as you expect. For example, whether an if branch executed when you expect it ought to be.
General notes on using a debugger
The specifics of using a debugger depend on the debugger and, to a lesser degree, the programming language you are using.
You can attach a debugger to a process already running your program. You might do it if your program is stuck.
In practice it is often easier to run your program under the control of a debugger from the very start.
You indicate where your program should stop executing by indicating the source code file and line number of the line at which execution should stop, or by indicating the name of the method/function at which the program should stop (if you want to stop as soon as execution enters the method). The technical means that the debugger uses to cause your program to stop is called a breakpoint and this process is called setting a breakpoint.
Most modern debuggers are part of an IDE and provide you with a convenient GUI for examining the source code and variables of your program, with a point-and-click interface for setting breakpoints, running your program, and single stepping it.
Using a debugger can be very difficult unless your program executable or bytecode files include debugging symbol information and cross-references to your source code. You might have to compile (or recompile) your program slightly differently to ensure that information is present. If the compiler performs extensive optimizations, those cross-references can become confusing. You might therefore have to recompile your program with optimizations turned off.
I want to add that a debugger isn't always the perfect solution, and shouldn't always be the go-to solution to debugging. Here are a few cases where a debugger might not work for you:
The part of your program which fails is really large (poor modularization, perhaps?) and you're not exactly sure where to start stepping through the code. Stepping through all of it might be too time-consuming.
Your program uses a lot of callbacks and other non-linear flow control methods, which makes the debugger confused when you step through it.
Your program is multi-threaded. Or even worse, your problem is caused by a race condition.
The code that has the bug in it runs many times before it bugs out. This can be particularly problematic in main loops, or worse yet, in physics engines, where the problem could be numerical. Even setting a breakpoint, in this case, would simply have you hitting it many times, with the bug not appearing.
Your program must run in real-time. This is a big issue for programs that connect to the network. If you set up a breakpoint in your network code, the other end isn't going to wait for you to step through, it's simply going to time out. Programs that rely on the system clock, e.g. games with frameskip, aren't much better off either.
Your program performs some form of destructive actions, like writing to files or sending e-mails, and you'd like to limit the number of times you need to run through it.
You can tell that your bug is caused by incorrect values arriving at function X, but you don't know where these values come from. Having to run through the program, again and again, setting breakpoints farther and farther back, can be a huge hassle. Especially if function X is called from many places throughout the program.
In all of these cases, either having your program stop abruptly could cause the end results to differ, or stepping through manually in search of the one line where the bug is caused is too much of a hassle. This can equally happen whether your bug is incorrect behavior, or a crash. For instance, if memory corruption causes a crash, by the time the crash happens, it's too far from where the memory corruption first occurred, and no useful information is left.
So, what are the alternatives?
Simplest is simply logging and assertions. Add logs to your program at various points, and compare what you get with what you're expecting. For instance, see if the function where you think there's a bug is even called in the first place. See if the variables at the start of a method are what you think they are. Unlike breakpoints, it's okay for there to be many log lines in which nothing special happens. You can simply search through the log afterward. Once you hit a log line that's different from what you're expecting, add more in the same area. Narrow it down farther and farther, until it's small enough to be able to log every line in the bugged area.
Assertions can be used to trap incorrect values as they occur, rather than once they have an effect visible to the end-user. The quicker you catch an incorrect value, the closer you are to the line that produced it.
Refactor and unit test. If your program is too big, it might be worthwhile to test it one class or one function at a time. Give it inputs, and look at the outputs, and see which are not as you're expecting. Being able to narrow down a bug from an entire program to a single function can make a huge difference in debugging time.
In case of memory leaks or memory stomping, use appropriate tools that are able to analyze and detect these at runtime. Being able to detect where the actual corruption occurs is the first step. After this, you can use logs to work your way back to where incorrect values were introduced.
Remember that debugging is a process going backward. You have the end result - a bug - and find the cause, which preceded it. It's about working your way backward and, unfortunately, debuggers only step forwards. This is where good logging and postmortem analysis can give you much better results.
I am running a JAVA program on RHEL 6.4 Server. The program terminates abnormally displaying the messaeg "Segmentation fault(Core dumped)". But i do not find any file indicating the reason for termination in the current user directory from where the program was run.
How can i debug to find the error in such case?
DOUBT
As per my understandin segmentation fault occurs when program tries to access a memory address outside the programs range. I would expect to see such faults in C or C++ programs, but in Java since there are no pointers , how is segmentation fault possible.
how is segmentation fault possible.
There are several possible reasons for this. There could be a bug in the JVM itself, or in a package (some of these are written in C or C++). It could also be due to a misconfiguration where incompatible components are used together.
From experience, a JVM bug is the least likely of these (although I've seen some).
If you capture the stack trace at the point of the crash, this might give you some clues as to where exactly the crash is occurring.
I am trying to build a SIP client in Java. As SIP-framework I use doubango. To display the video I'm using OpenGL through JOGL, but that should not be important, because during testing I actually detached the ByteStream from JOGL.
First of all, the video works fine on Win 7 with a 32 bit VM. The libraries do not compile for 64 bit, so I have to use a 32-bit-VM on 64-bit-system, which I thought should work just fine. Unfortunately I get to see the video only for a few seconds, before it crashes with an "Unhandled exception at 0x774315de in javaw.exe: 0xC0000005: Access violation reading location 0xdddddde9."
Debugging in VisualStudio revealed, that the exception occurs in the following line:
memcpy((void*)m_ConsumeBuffer.pConsumeBufferPtr, pBuffer, nRetsize);
(http://code.google.com/p/doubango/source/browse/branches/2.0/doubango/bindings/_common/ProxyConsumer.cxx#710)
So apparently the error happens after a few hundred calls to the method (because the video displays correctly for a while).
To narrow down the possible causes, I only implemented incoming video in my test.
The doubango gets the pointer to the ByteBuffer from my implementation in Java, like so:
videoFrame = ByteBuffer.allocateDirect(frameSize);
consumer.setConsumeBuffer(videoFrame, frameSize);
The consumer is doubango's own ProxyVideoConsumer which I get from it at construction.
I am not too familiar myself with C++-Debugging, seeing as I'm more experienced with Java and Delphi, but I think I'm confident to say that there seems to happen some kind of heap corruption.
My current assumption is that the JVM's ByteBuffer.allocateDirect(..)-function does not behave correctly when running a 32-bit JVM on a 64-bit machine, but of course I can be mistaken.
It might be interesting to note that in two of about 50 test runs, the video was stable for 5-10 minutes, without any degradation or apparent memory leak. But this "pseudo-randomness" is consistent with my experience with memory allocation issues in Delphi.
I'd be glad if someone could help me, as I've run out of ideas how to get this to run stable.
Regards,
CSp
I think I somewhat solved the problem myself. Apparently when the error occurs, the pBuffer is already freed/invalidated at this point. I traced the error so far, that the sipSession which contains the buffer is being freed by the JVM at random intervals, even if there is still a reference being hold within the JVM.
Putting the garbage collector into debug-mode also didn't reveal much, as it didn't log any intention to dispose any object at this point. Actually the error occurs after only a few seconds, so the GC didn't try to free anything at that point.
I found one instance, where I freed the sipSession prematurely, which reduced the frequency of the error, but all the other disposals have their stack-trace leading to the GC.
I started looking into JNI and from what I understand is that if a problem occurs with the loaded dll, the jvm is possible to terminate on the spot.
I.e. the process can not be protected e.g. like when catching an exception.
So if my understanding is correct, my question is if there is a standard approach/pattern for this situation when using jni.
Or to state it differently, are processes using jni designed in way to avoid these issues?
Or such problems are not expected to occur?
Thank you.
Yes, the JVM will just terminate which is one of the reasons why JNI code is really hard to debug. If you are using C++ code you can use exceptions and then map them to a Java exception which at least gives you some level of security but doesn't help with things like bad memory access etc.
From an architecture point of view I suggest to decouple you code from JNI as much as possible. Create a class / procedure structure that is entirely testable from C++/ C and let the JNI code do only all the conversion stuff. If the JVM then crashes you at least know where you have to look.
The principles are no different from any multi-threaded C application:
Always check all your input thoroughly.
Always free up temporary memory you allocated.
Make sure your functions are re-entrant.
Don't rely on undefined behaviour.
The Java virtual machine offers you no extra protection for your native code, if it fails or is leaking, your VM will fail or leak.
You can have exactly the same spectrum of error handling in a JNI library as in anything else.
You can use try/catch. If you are on Windows, you can use SEH. If you are on Linux, you can call sigaction.
Still, if you mess up and there's a SIGSEGV, your JVM is probably toast whether you try to catch that signal or not.