I've run into the issue where I have a program (not written by me, by someone else) I want to run 24/7, but sometimes it crashes. Normally, this wouldn't be an issue because I can simply create a process watcher that checks if it crashed, and then restarts it if necessary.
But, this particular program sometimes throws an exception and outputs it into the graphical interface that's integrated into it. In this instance, the program doesn't crash at all. The interface stays up, but the actual server functionality is unavailable.
Is there any way I can intercept this information from this process?
You want to use the Java Virtual Machine Tools Interface. I can't give you the code to catch your exception, but this is where to look. You'll have to do some detective work to find the class that throws the exception, or at least to find some indicator that it has been thrown.
Edit: You can also try calling the vendor to see if they know of a way. You can also look to see if it is writing the exception to a log file, which you could then watch.
This may or may not work, but if when the application displays it's error and the server stops working does the memory usage drop? If so you could probably just add some logic to your process monitor to call the windows command tasklist to see if the memory usage drops below some threshold. You'll have to check how much memory the program normally uses and how much it uses after the error though.
Since you said the server functionality stops working, another option could be to write a simple program that basically just pings the server how ever often you want to make sure it is still up. If not, kill the process and restart it.
I assume you have no access to the source code, so if it is outputting to the GUI the answer is no. Even if you could attach to the running process you would need to intercept the exception, but it is caught and sent to the GUI, not thrown from the application.
In theory, you could screen scrape the application. I don't know of any specific tools for doing this, but they may be out there.
Edit: I may have been wrong above, check out a post here where they get the stack from a running thread. You probably won't be able to capture the exception this way, but if you're lucky the stack trace will look very different when the program is operating normally compared to when an exception has been thrown.
Edit 2: I submitted a second, more accurate answer. See below.
Is the other program Java? Look at AspectJ, you may be able to hack something using it if you have control on the program startup.
Without ability to rebuild the app you are generally out of luck unless you do some extensive hacking. Here is one option I can think of.
Most likely the application replaces System.out and/or System.err with its own stream implementation. If that's the case you can try to locate the class for this stream and replace it with your own wrapper with the same name. You may rename original class using jarjar. In the wapper you can provide console output to detect the exception.
Related
I have problem similar to my previous one presented here.
This time I want use program written in c/c++ to track execution of JAVA program. So as I stated before same code which track stdout printing for c/c++ and register syscall 4 haven't done it for JAVA. I assume it's because execlp which I trace is used just to run jvm. And later on there are created more processes (by inner mechanism of jvm) which I do not track. I found this topic which seems to be partial solution. If I got it right every child will be traced. But that's is a problem as well I want to track only that process which handles my application and not all others that jvm might create. Is there any chance to get know which jvm thread/process handles my program and track only it?
For make it a bit easier let's assume my JAVA program is one-thread.
If you start the binary through your tracer app, all threads will be traced.
But if you attach to a process, then you won't attach to all it's threads. You have to attach to all of its threads using the threadids, that you can found listed eg. in /proc/%d/task/.
Also, I suggest reading through strace's source code, I've learnt a lot from it. If you can use strace to successfully follow java threads as you want, you can get the logic from it.
I am running a Java Program in command prompt
The normal course is after successfully executing the program it comes back to prompt .. what are the possible reasons it will not come back to prompt after successfully executing the program
why is it not coming back to prompt after execution
usually it comes back but sometimes it doesn't...
This sounds like a race condition. Something in your application's shutdown sequence is non-deterministic, and it works or does not work depending on various platform specific (and possibly external) factors. There is probably no point figuring out what those factors are (or might be), since it won't help you fix the problem.
Only difference is in RAM hard disk capacity mine is slower.. Can it be possible reason?
These could be factors, but they are not the cause of the problem. So focus on figuring out what makes your application non-deterministic.
As others have said, without more information (and relevant code) we can only guess.
When the application has failed to shut down, get it to give you a thread dump. Or try shutting it down while it is attached to a debugger. These may allow you to get some clues as to what is going wrong.
Finally, the brute force solution is simply to have the main method (or whatever) call System.exit(0) on its way out. But beware of the possibility of files not being flushed, etc if you do that.
Because it's not finishing. If it's sometimes happening and sometimes not, my instinct is that you have some sort of race condition. Probably one of your cleanup steps is hanging if another action has or hasn't been taken.
Without source code this will be hard to debug.
There could be an active thread still running which is not in "daemon" mode. For example, if you have a Swing GUI and all of the frames are closed the Event Dispatch thread is still active so the JVM will not exit.
What is the best practice when dealing with Errors within a server application?
In particular, how do you think an application should handle errors like OutOfMemoryError?
I'm particularly interested in Java applications running within Tomcat, but I think that is a more general problem.
The reason I'm asking is because I am reviewing a web application that frequently throws OOME, but usually it just logs them and then proceeds with execution. That results, obviously, in more OOMEs.
While that is certainly bad practice, in my opinion, I'm not entirely sure that stopping the Server would be the best solution.
There is not much you can do to fix OutOfMemoryError except to clean up the code and adjust JVM memory (but if you have a leak somewhere it's just a bandaid)
If you don't have access to the source code and/or are not willing to fix it, an external solution is to use some sort of watch dog program that will monitor java application and restart it automatically when it detects OOMEs. Here is a link to one such program.
Of course it assumes that the application will survive restarts.
The application shouldn't handle OOM at all - that should be the server's responsibility.
Next step: Check if memory settings are appropriate. If they aren't, fix them; if they are, fix the application. :)
Well, if you have OOME then the best way would be to release as many resources (especially cached ones) as possible. Restarting the web-app (in case it's web-apps fault) or the web server itself (in case something else in the server does this) would do for recovering from this state. On the development front though it'd be good to profile the app and see what is taking up the space, sometimes there are resources that are attached to a class variable and hence not collected, sometimes something else. In the past we had problems where Tomcat wouldn't release the classes of previous versions of the same app when you replace the app with a newer version. Somewhat solved the problem by nullifying class variables or re-factoring not to use them at all but some leaks still remained.
An OutOfMemoryError is by no means always unrecoverable - it may well be the result of a single bad request, and depending on the app's structure it may just abandon processing the request and continue processing others without any problems.
So if your architecture supports it, catch the Error at a point where you have a chance to stop doing what caused it and continue doing something else - for an app server, this would be at the point that dispatches requests to individual app instances.
Of course, you should also make sure that this does not go unnoticed and a real fix can be implemented as soon as possible, so the app should log the error AND send out some sort of warning (e.g. email, but preferably something harder to ignore or get lost). If something goes wrong during that, then shutting down is the only sensible thing left to do.
#Michael Borgwardt, You can't recover from an OutOfMemoryError in Java. For other errors, it might not stop the application, but OutOfMemoryError literally hangs applications.
In our application which deals with Documents heavily, we do catch OOM errors where one bad request can result in OOM but we dont want to bring down the application because of this. We catch OOM and log it.
Not sure if this is best practice but seems like its working
I'm not an expert in such things, but I'll take a chance to give my vague opinion on this problem.
Generally, I think that there's two main ways:
Server is stopped.
Resources are thus gracefully degrading throughput, reducing memory consumption, but staying alive. For this case application must have appropriate architecture, I think.
According to the javadoc about a java.lang.Error:
An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. Most such errors are abnormal conditions. The ThreadDeath error, though a "normal" condition, is also a subclass of Error because most applications should not try to catch it.
A method is not required to declare in its throws clause any subclasses of Error that might be thrown during the execution of the method but not caught, since these errors are abnormal conditions that should never occur.
So, the best practice when dealing with subclasses of Error is to fix the problem that is causing them, not to "handle" them. As it's clearly stated, they should never occur.
In the case of an OutOfMemoryError, maybe you have a process that consumes lots of memory (e.g. generating reports) and your JVM is not well sized, maybe you have a memory leak somewhere in your application, etc. Whatever it is, find the problem and fix it, don't handle it.
I strongly disagree with the notion that you should never handle an OutOfMemoryError.
Yes, it tends to be unrecoverable most of the time. However, one of my servers got one a few days ago and the server was still mostly working for more than an hour and a half. Nobody complained so I didn't notice until my monitoring software got a failure and hour and a half after the first OutOfMemoryError. I need to know as soon as possible when there is an OutOfMemoryError on my server. I need to handle it so that I can set up a notification so that I can know to restart my server ASAP.
I'm still trying to figure out how to get Tomcat to do something when it gets an Error. error-page doesn't seem to be working for it.
I need to use deprecated stop () because I need to run Runnable classes which were developed by other programmers and I can't use while (isRunning == true) inside method run.
The question is: Is it safety enough to use method stop ()? Theads don't work with any resources (like files, DB, or Internet connections). But I want to be sure that JVM wouln't be corrupted after I stop a dozen of threads with stop () method.
P.S.: yes, I can write some code to test it, but I hope somebody knows the answer)
Sort of. There's nothing inherently "corrupting" about Thread.stop(). The problem is that it can leave objects in a damaged state, when the thread executing them suddenly stops. If the rest of your program has no visibility to those objects, then it's alright. On the other hand, if some of those objects are visible to the rest of the program, you might run into problems that are hard to diagnose.
If you use Thread.stop you'll probably get away with, assuming you have few users. It is exceptionally hard to test for. It can cause an exception anywhere in executing code. You can't test all possible situations. On your machine in your set up you might never find a problem; come the next JRE update your program might start failing with a highly obscure intermittent bug.
An example problem case is if the thread is loading a class at the time. The class fails to load and will not be retried again. You program is broken.
The JVM won't be corrupt, but read the javadocs closely to make sure that you don't meet their conditions for "disaster."
You'll need to take a close look at any synchronization monitors that the thread is holding onto. You mentioned files and sockets as resources being hung onto, but you'll also need to consider any shared data structures. Also make sure your exception handling doesn't catch RuntimeExceptions (see stop()).
I'm trying to debug a problem in my Java application that throws no errors, no exceptions and doesn't even crash the app (it seems the failure happens in a separate thread).
The problem seems to be inside a call to a library function (it's JAXBContext.newInstance(String) if that matters). The program will reach the line just before the call, but not the one just after it. My catch blocks are not entered and the program just continues to run.
The problem happens while trying to render an XML response to a web request that came in via Struts. The request has been handled and the code should marshal the response object. The client gets a response right away (so the code doesn't seem to hang in a loop), but it's just empty.
I have set a breakpoint just before the problematic line but the debugger just runs over it, I haven't a clue why.
I'm using eclipse and the application runs inside an OSGi container (Apache Felix) that was started with -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y. From within Eclipse I then use the Debug settings for "Remote Java application" to connect the debugger.
What are techniques to get at such a problem?
Probably an obvious question, but are you sure you are catching Throwable? An unchecked exception could easily cause the thread in question to die (assuming no one above you in the call stack is catching it either.)
Since you are suspending the VM on startup with your debug arguments, I assume you have confirmed that the debugger is attaching correctly. The fact that you say the debugger skips right past the call is very suspect. Are you able to hit any breakpoints in this application? What about in this Class? What about in this thread?
How did you narrow down the line in question without the debugger? println/debugging to a file?
Can you paste a code snippet of the method in question?
You could confirm the theory that the thread is dying by creating a second thread before the problem occurs and joining it to the thread you think is dying. Then the second thread's run() method would be invoked when the thread in question exits, and you'd know it died (but would still not know why.)
In answer to your general question, when I have a bug in a Java app that I can't reproduce in the debugger (which happens from time to time for various reasons), I incrementally modify my code with sysout printlns or output to files. If necessary, I may also modify the code my code is invoking. If you don't have the source code to the code you are invoking, you can try one of the many BCI frameworks to inject your byte code into the methods in question. It's a tedious process, but only happens occasionally.
You could try getting a Thread Dump - that will tell you if any methods are blocking (e.g. waiting for input). [Edit: re-reading your original question, getting a thread dump probably won't help as it looks like nothing is actually blocking. But I'm leaving it here as I find it useful in many other situations!]
If you think the error is happening in another thread you could also set an UncaughtExceptionHandler to try and catch it.
If you're sure the problem is somewhere within that method, you could try looking at the JAXB source code.
EDIT:
Well, if it gets really bad you can build your own private copy with debugging instrumentation. I hope you won't have to resort to that.
perhaps inside the call there is an infitite loop happening and this is why you get no further - but this might not cause a crash (unless memory is being used in each loop).