I made a TwitchBot that I have been running for a long time, and every now and then it would just terminate itself. (Stop running).
I assumed these were internet or connection drops but it's happening a lot more often today. Does anyone have an idea what could cause it and how to debug it in the future?
What it looks like after it stops itself:
Chat log at time of termination:
There is nothing that is being specifically ran on any of those messages. So I have no way of specifically try-catching a throwable.
Unless I made it try catch on an onMessage listener so it checked every chat message that is seen. - Other than that, there is no reason there should be a hard coded error.
I use TwitchIRC API if anyone thinks it may be something within the API.
https://github.com/lmariscal/twitchirc
EDIT:
Just happened again whilst writing this:
What it looks like after it stops itself:
I am trying to debug a program in Java using SpringSource ToolSuite. Every time it gets stuck at AWTAutoShutdown.class. I have tried hitting F8 to step out of whatever is happening but it seems like there is a thread locking up somewhere. I understand this is kind of a vague question but has anyone seen this before? Where can I start looking to solve this problem?
So one thread is suspended with a ThreadDeath exception. Perhaps STS is suspending because there is an uncaught exception, instead of on a breakpoint. There is a setting so that you can turn off suspension on uncaught exceptions, which I only want about half the time anyway. Turn that off and see if your thread dies and gives you something more useful in terms of operation (or an error).
What is the best practice when dealing with Errors within a server application?
In particular, how do you think an application should handle errors like OutOfMemoryError?
I'm particularly interested in Java applications running within Tomcat, but I think that is a more general problem.
The reason I'm asking is because I am reviewing a web application that frequently throws OOME, but usually it just logs them and then proceeds with execution. That results, obviously, in more OOMEs.
While that is certainly bad practice, in my opinion, I'm not entirely sure that stopping the Server would be the best solution.
There is not much you can do to fix OutOfMemoryError except to clean up the code and adjust JVM memory (but if you have a leak somewhere it's just a bandaid)
If you don't have access to the source code and/or are not willing to fix it, an external solution is to use some sort of watch dog program that will monitor java application and restart it automatically when it detects OOMEs. Here is a link to one such program.
Of course it assumes that the application will survive restarts.
The application shouldn't handle OOM at all - that should be the server's responsibility.
Next step: Check if memory settings are appropriate. If they aren't, fix them; if they are, fix the application. :)
Well, if you have OOME then the best way would be to release as many resources (especially cached ones) as possible. Restarting the web-app (in case it's web-apps fault) or the web server itself (in case something else in the server does this) would do for recovering from this state. On the development front though it'd be good to profile the app and see what is taking up the space, sometimes there are resources that are attached to a class variable and hence not collected, sometimes something else. In the past we had problems where Tomcat wouldn't release the classes of previous versions of the same app when you replace the app with a newer version. Somewhat solved the problem by nullifying class variables or re-factoring not to use them at all but some leaks still remained.
An OutOfMemoryError is by no means always unrecoverable - it may well be the result of a single bad request, and depending on the app's structure it may just abandon processing the request and continue processing others without any problems.
So if your architecture supports it, catch the Error at a point where you have a chance to stop doing what caused it and continue doing something else - for an app server, this would be at the point that dispatches requests to individual app instances.
Of course, you should also make sure that this does not go unnoticed and a real fix can be implemented as soon as possible, so the app should log the error AND send out some sort of warning (e.g. email, but preferably something harder to ignore or get lost). If something goes wrong during that, then shutting down is the only sensible thing left to do.
#Michael Borgwardt, You can't recover from an OutOfMemoryError in Java. For other errors, it might not stop the application, but OutOfMemoryError literally hangs applications.
In our application which deals with Documents heavily, we do catch OOM errors where one bad request can result in OOM but we dont want to bring down the application because of this. We catch OOM and log it.
Not sure if this is best practice but seems like its working
I'm not an expert in such things, but I'll take a chance to give my vague opinion on this problem.
Generally, I think that there's two main ways:
Server is stopped.
Resources are thus gracefully degrading throughput, reducing memory consumption, but staying alive. For this case application must have appropriate architecture, I think.
According to the javadoc about a java.lang.Error:
An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. Most such errors are abnormal conditions. The ThreadDeath error, though a "normal" condition, is also a subclass of Error because most applications should not try to catch it.
A method is not required to declare in its throws clause any subclasses of Error that might be thrown during the execution of the method but not caught, since these errors are abnormal conditions that should never occur.
So, the best practice when dealing with subclasses of Error is to fix the problem that is causing them, not to "handle" them. As it's clearly stated, they should never occur.
In the case of an OutOfMemoryError, maybe you have a process that consumes lots of memory (e.g. generating reports) and your JVM is not well sized, maybe you have a memory leak somewhere in your application, etc. Whatever it is, find the problem and fix it, don't handle it.
I strongly disagree with the notion that you should never handle an OutOfMemoryError.
Yes, it tends to be unrecoverable most of the time. However, one of my servers got one a few days ago and the server was still mostly working for more than an hour and a half. Nobody complained so I didn't notice until my monitoring software got a failure and hour and a half after the first OutOfMemoryError. I need to know as soon as possible when there is an OutOfMemoryError on my server. I need to handle it so that I can set up a notification so that I can know to restart my server ASAP.
I'm still trying to figure out how to get Tomcat to do something when it gets an Error. error-page doesn't seem to be working for it.
I'm trying to debug a problem in my Java application that throws no errors, no exceptions and doesn't even crash the app (it seems the failure happens in a separate thread).
The problem seems to be inside a call to a library function (it's JAXBContext.newInstance(String) if that matters). The program will reach the line just before the call, but not the one just after it. My catch blocks are not entered and the program just continues to run.
The problem happens while trying to render an XML response to a web request that came in via Struts. The request has been handled and the code should marshal the response object. The client gets a response right away (so the code doesn't seem to hang in a loop), but it's just empty.
I have set a breakpoint just before the problematic line but the debugger just runs over it, I haven't a clue why.
I'm using eclipse and the application runs inside an OSGi container (Apache Felix) that was started with -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y. From within Eclipse I then use the Debug settings for "Remote Java application" to connect the debugger.
What are techniques to get at such a problem?
Probably an obvious question, but are you sure you are catching Throwable? An unchecked exception could easily cause the thread in question to die (assuming no one above you in the call stack is catching it either.)
Since you are suspending the VM on startup with your debug arguments, I assume you have confirmed that the debugger is attaching correctly. The fact that you say the debugger skips right past the call is very suspect. Are you able to hit any breakpoints in this application? What about in this Class? What about in this thread?
How did you narrow down the line in question without the debugger? println/debugging to a file?
Can you paste a code snippet of the method in question?
You could confirm the theory that the thread is dying by creating a second thread before the problem occurs and joining it to the thread you think is dying. Then the second thread's run() method would be invoked when the thread in question exits, and you'd know it died (but would still not know why.)
In answer to your general question, when I have a bug in a Java app that I can't reproduce in the debugger (which happens from time to time for various reasons), I incrementally modify my code with sysout printlns or output to files. If necessary, I may also modify the code my code is invoking. If you don't have the source code to the code you are invoking, you can try one of the many BCI frameworks to inject your byte code into the methods in question. It's a tedious process, but only happens occasionally.
You could try getting a Thread Dump - that will tell you if any methods are blocking (e.g. waiting for input). [Edit: re-reading your original question, getting a thread dump probably won't help as it looks like nothing is actually blocking. But I'm leaving it here as I find it useful in many other situations!]
If you think the error is happening in another thread you could also set an UncaughtExceptionHandler to try and catch it.
If you're sure the problem is somewhere within that method, you could try looking at the JAXB source code.
EDIT:
Well, if it gets really bad you can build your own private copy with debugging instrumentation. I hope you won't have to resort to that.
perhaps inside the call there is an infitite loop happening and this is why you get no further - but this might not cause a crash (unless memory is being used in each loop).
I'm working on a cross platform application in Java which currently works nicely on Windows, Linux and MacOS X. I'm trying to work out a nice way to do detection (and handling) of 'crashes'. Is there an easy, cross-platform way to detect 'crashes' in Java and to do something in response?
I guess by 'crashes' I mean uncaught exceptions. However the code does use some JNI so it'd be nice to be able to catch crashes from bad JNI code, but I have a feeling that's JVM specific.
For simple catch-all handling, you can use the following static method in Thread. From the Javadoc:
static void setDefaultUncaughtExceptionHandler(Thread.UncaughtExceptionHandler eh) Set the default handler invoked when a thread abruptly terminates due to an uncaught exception, and no other handler has been defined for that thread.
This is a very broad way to deal with errors or unchecked exceptions that may not be caught anywhere else.
Side-note: It's better if the code can catch, log and/or recover from exceptions closer to the source of the problem. I would reserve this kind of generalized crash handling for totally unrecoverable situations (i.e. subclasses of java.lang.Error). Try to avoid the possibility of a RuntimeException ever going completely uncaught, since it might be possible--and preferable--for the software to survive that.
For handling uncaught exceptions you can provide a new ThreadGroup which provides an implementation of ThreadGroup.uncaughtException(...). You can then catch any uncaught exceptions and handle them appropriately (e.g. send a crash log home).
I can't help you on the JNI front, there's probably a way using a native wrapper executable before calling the JVM, but that executable is going to need to know about all the possible JVMs it could be calling and how the indicate crashes and where crash logs are placed etc.
Not sure if this is what you needing, but you can also detect if an exception has occurred from within your native code. See http://java.sun.com/javase/6/docs/technotes/guides/jni/spec/functions.html#wp5234 for more info.