In node.js world, there are apps like Supervisor - it's a daemon process that checks if your app is running, and if not (closed, crashed), it restarts it instantly. That's very nice way of temporary handling critical errors in production, when one feature could fail, but rest of system is still running.
I come from PHP background where all you had to do is to press back button in browser, when something is broken.
How to achieve this behavior in Spring Boot ? So far what I noticed is that when app faces unhandled exception, it crashes and whole server is down. I know that those kind of errors are the ones that one should fix asap, but sometimes it's just impossible, and system needs to be running.
Are there any tools that work like Node.js supervisor?
In the past I sometimes used the Java Wrapper by Tanuki which worked quite nice. Otherwise you do have the option to either monitor the process and automatically restart if it fails (dependent on your system environment) or on the highest level of you application catch Throwable, which will not be a good idea because you'll catch fatal cases that by intend should kill your jvm execution, e.g. OutOfMemory...
Related
SEVERE: The web application [/webapps/bb-nautilus-BBLEARN] appears to have started a thread named [MessageQueueHandler-bb-nautilus-content-blitz-0] but has failed to stop it. This is very likely to create a memory leak.
If an application starts all kinds of stuff (registering jdbc drivers, starting threads, ...) when it is fired up, it is the responsibility of that application to also clean up after itself when it is stopped.
Are you the author of this application ? Correct your code. Not the author of this application ? Submit a bug report.
In the latter case, until the bug is addressed it might be possible to add a ServletContextListener of your own making to the deployment. But clearing up leftover Threads from "foreign" code is at any rate going to require you to figure out how to find those Thread objects and then subsequently stop() them, which is a deprecated method.
MessageQueue may be busy doing something strange. It refuses to exit. So, reboot the server and try starting Bb Learn after that. Post the new error after you know that no part of Bb Learn was running after a failed app restart.
I have tomcat server running. and it crashes suddenly for some reason... however I am trying to find the error.
Is there a function in tomcat or java like beforeExit() or ifCrashed() which I can override and write some code there like notifying myself if server crashes for some reason.
You could try using shut down hooks which are executed on system exit. It's not guaranteed though that it will be executed on a hard crash like SIGKILL but this may be an option in some cases:
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
// Implementation goes here..;
}
});
I don't think that crashed server can manage something. You can consider this steps:
use servletContextListeners in order to run code before servlet context will shutdown. It will work if shutdown is done gracefully
if it is JVM that is crushing you can try to find tools that provide JVM monitoring services and reporting. Also I recommend to provide flag in order to get reports from OutOfMemoryError and open JMX ports for monitoring
you can right some mock rest endpoint and run some external job that will call that endpoint periodically and if it returns HTTPStatus.OK then it working, if not - then you can report yourself someway.
It could be better if you can provide logs and some information about what is really happening with server. Maybe here people will help but I gues this is topic of another questions.
In case of a fatal crash, you will get a jvm log. You can specify the location by giving tomcat a -XX:ErrorFile on the Oracle JVM
http://www.oracle.com/technetwork/java/javase/felog-138657.html
If it is not that fatal, you can adjust the log settings in tomcat to display better logging levels, such as specified on their doc site
https://tomcat.apache.org/tomcat-6.0-doc/logging.html
If it dies because of out of memory errors, you can get a heap dump on OOM, and analyse it with i.e. jhat
http://docs.oracle.com/javase/6/docs/technotes/tools/share/jhat.html
If profiling is an option for you JProfiler has an offline profiling mode where you can capture information about the running program and log it for viewing at a later time. Check out the docs
JPRofiler Offline Profiling
This might give you some more insight into whats happening in the application. According to this documentation you can also manage the profiling agent from you application.
Can we automatically restart a websphere application server v6.1 on OOM exception after heap dump is created?we have an enterprise application hosted on websphere application server,recently we are facing OOM exceptions,and from time to time the app server gets automatically restarted after the heap dump is generated.But recently the app server restart is not happening automatically but has to be done manually.Can you please let me know what may be the issue
There is no in built/parameter based option in WAS 6.1 answering your question. It comes in v.7.0.
Better way I/Many follow is write a basic java program to monitor the sysout.log/ syserr.log for the particular String "OutOfMemory" or "in total in the server that may be hung". If the log has any of those string, then (i) stop the server, (ii) rotate the logs (iii)start the server.
Schedule this java program for every 2 or 5 mins.
I wont recommend this method, This is not a good practice as well. I would recommend WASADM should inform the data related team to fix the issue and providing the logs, threads, hprofs, etc..
But most of the time, it is difficult and time consuming for data/application team to fix it immediately. So WAS administrator has to follow these kind of methods.
I have a Java application which should run on a server machine in the background (the application can be started by a command without GUI). The problem is: When I log off from the server, the application gets killed. I'm looking for something similar like nohup under Linux. I found some solutions, but I'm unsure, what is the best for my situation (Windows Server 2003, Java Application run from BAT-Skript, restart after booting the machine)? What are the pros and cons of the solutions?
psexec: Do the process really need to run under the SYSTEM account?
Combination of instsrv and srvany: But srvany should should never be used in a production environment
I have started the application with the scheduled tasks with the option run as NT AUTHORITY\SYSTEM. But after log off, the application was still killed. Can I get this working?
The DOS Task Scheduler AT command
Write a windows service (on Java, C# - I don't think it does matter)
Start the app from your service
or
Run your app as a window service using 3d party utility
The only way in which you can get a process to run without a user being logged in is to have the process run as a windows service or at the very least called by a windows service.
Just to add to above answers.
Consider using http://wrapper.tanukisoftware.com.
It's rather mature and popular (in contrast to ServiceEx and RunAsService).
Yes, a Windows Service is definitely the way to go but there are a few things to watch out for when you run a Java application as a Windows Service. Most of them are covered in this tutorial showing how to setup a java application with our commercial run-anything-as-a-service application, AlwaysUp:
http://www.coretechnologies.com/products/AlwaysUp/Apps/RunJavaApplicationAsAService.html
Beware: You will almost surely need the "-Xrs" flag on Windows 2003 to prevent the closing-on-logoff behavior but things can get tricky if you are catching shutdown events. Let me know if that is an issue for your situation.
I want to find or develop an application that can run as a daemon, notify the administrator by email or sms when the Java applications running on a host get any exceptions or errors. I know JVMTI can achieve part of my goal, but it will impact performance of the monitored applications(I don't know how much will it be, it will be acceptable if it's slight), besides it seems to be a troublesom job to develop a JVMTI agent and I'm not sure what would happen if several applications running at the same time using the same agent. Is there any better solutions? Thanks in advance.
One way would be to use a logging system like log4j that publishes all errors occuring on system A to a logging server on system B from which you can monitor the errors occured. This isn't a completely generic solutation however, since only exceptions propagated to log4j (or any other logging system) would be handled - but it may be a good start.
The best solution is to have the Java application send its errors via email/sms. The problem is that programs will generate exceptions and handle correctly in normal operation. You only want particular exception.
Failing this you could write a log reader, which reads the logs of the application. This is tricky to get right, but it can be done.
An application can generate 1000+ exception per days and still be behaving normally because the application knows how to handle these exceptions. e.g. every time a socket connection is closed an exception can be thrown.
IMO, the best approach is to deploy an external monitoring system. This can:
monitor multiple applications
monitor infrastructure services
monitor network availability and machine accessibility,
monitor resources such as processor and file system usage.
Applications can be monitored in a variety of ways, including:
by processing log events,
by watching for application restarts,
by "pinging" the application's web apis to check service liveness, and
by using the application's JMX interfaces.
This information can be filtered and prioritized in an intelligent fashion, and critical events can be reported by whatever means is most appropriate.
You don't want individual applications sending emails, because they don't have sufficient information to do a decent job. Furthermore, putting the reporting logic into individual applications is likely to lead to inconsistent implementation, poor configurability, and so on.
There is a nearby alternative to JVMTI : JPDA. This infrastructure allows you to create a remote "debugger" (yes, that's what you're planning to do) using Java code, and connect it to the VM using either local or remote connection.
There will be, like for JVMTI, an overhead to program execution. However, as the Trace.java example shows, it's quite simple to both implement and connect to target VM.
Finally, notice if you want to instrument code run by application server (JBoss, Glassfish, Tomcat, you name it) there are various other means available.
I follow the pattern where every exception gets logged to a table.
Then an RSS feed selects from that table.
I subscribe to the RSS feed in MS Outlook at work and also on my Android phone with a program called NewsRob. NewsRob let me set my phone to alert me when there is something new.
I blog about how to do this HERE. It is in .net, but you get the idea.
As a related step I found a way to notify myself when something DIDN'T happen. That blog is HERE.
There are loads of applications out there that do what you are looking for in a way that does not impact performance. Have you had a look at Kibana/ElasticSearch, or Splunk or Logscape for enterprise solutions ( they both also have free versions).
I'm going to echo what has already been said and highlight what java already provides and what you can do with an external monitoring system. Java already provides:
log4j - log ERRORS, WARNINGS, FATAL and Exceptions to a file
JMX - Create custom application metrics and you also have access to java.lang/* which will give you heap memory usage , garbage collection, thread counters etc.
JVM gc logging - you can log all your garbage collection events to a file and watch for any long Full GC collections.
An external monitoring system will allow you to set alerts triggered off different operational scenarios. You will also get visualisation of your system performance through charts. I've used Logscape's java app in the past to monitor 30 java processes spread out over3 hosts.