I'm looking for a tool (in linux) that can montior a tomcat/jboss process and if the process fails, I can respawn either or both without having to manually ssh into the box, do any house keeping and then start them up again. I'm not too sure if there is a good tool out there that can monitor the health of jboss/tomcat and report on it's performance. I know jvisualvm gives you various tools, but I'm looking for a disaster recovery solution that is a bit higher level than jvisualvm.
Java Service Wrapper is an application that wraps your Java process and installs it in the system using service (Windows) or daemon (Linux). It pings the VM periodically and restarts it when it does not respond. Worked for us in production with several application, including Tomcat, JBoss, Mule, etc. Actually Mule ESB is even bundled with this application in the distribution.
Also you don't have to run the application manually when the system starts.
I'm currently working on a daemon to do this and more, since JOPR nor naggios didn't do what we needed, but those are good tools you could use. I'm not sure but JOPR (or whatever is called today) can restart your servers in case something goes wrong.
A custom made solution as we're working on shouldn't take you more than a week. The main problem, is that to start either JBoss or Tomcat you have to call the startup scripts. But the startup script will restart the service if the exit code is 10, something like this:
while $? -ne 10; do
start_jboss
done
So, this daemon which is made in Java uses JMX to connect to the JBoss server and tells JBoss to go down and exit with status code 10 using a method in a MBean. I'm at home, so I'm not sure the exact name of the MBean you have to call for this but I'll provide more info tomorrow.
I am using monit to control the launch of Tomcat/JBoss.
Related
As the title says - when launching Spring app (STS) with embedded Tomcat, both java.exe and javaw.exe processes are created on the same port.
That means I have troubles with busy port, even when using the Relaunch option (which typically shuts Tomcat down and then restarts it). Indeed, relaunching does stop one of the two processes but then I get an error stating that the port must be busy. Killing the remaining one does solve the issue but I life is too short to both develop in Java AND kill the process manually.
Is there a convenient way to overcome this? I see that is STS (and Eclipse too, of course) there's a way to specify project specific JRE but I'm unsure on how to proceed.
Note: I have both 32- and 64-bit java8 instances installed.
Try running it as a java application (ie, just run main).
It sounds like you are deploying it on Tomcat, while at the same time there is an embedded Tomcat being started... :)
I have a Java application that uses the Apache Daemon service installer to register it as a Windows service. I am using Puppet to run an exec{} block to register the service, which works, and then chains a service{} block to start the service. Puppet uses "net.exe start" to run the service, but that command reports an error, even though the service starts correctly.
The output from running the command in a powershell shell is:
PS C:\ProgramData\PuppetLabs\puppet\etc\modules> net start myservice
The myservice_descriptive_name service is starting.....
The myservice_descriptive_name service could not be started.
More help is available by typing NET HELPMSG 3523.
As I refresh the Windows service panel while this command is running, I see the state change from:
blank field -> starting -> started
Is this a problem caused by the apache wrapper, which is starting a jvm in a separate shell or some other side effect? And, more importantly, can I get around this problem in Puppet while still using the service{} block? Is it possible to substitute sc.exe, which does not suffer the same problem, short of using an exec{} block?
To take the questions in order:
The net start command reports failure because the service appears to have hung.
Yes, the problem is caused by the Apache wrapper.
Specifically, the wrapper is telling Windows that it will reach the first checkpoint within two seconds. Since there does not appear to be any way for the Java code to implement a checkpoint, or to change the wait hint, this means that the service must start within two seconds to be compliant with the Windows service specification.
(In principle, Windows is entitled to terminate your service at this point. So far as I know, no current versions of Windows do so, though they may log error messages.)
Short of modifying Puppet or (preferably) the Apache wrapper, the only obvious workaround is to ensure that your service "starts" immediately, rather than waiting for initialization to complete.
This is less than ideal, since it means that the service can't provide feedback to Puppet if it really does fail to initialize, but no worse than your suggestion of using sc start instead of net start.
JPBlanc's answer explains why the net.exe times out waiting on the service to start, even though it does end up starting. You can definitely try swapping out net.exe calls for sc.exe (Service Control) instead.
I've created a ticket to address this - https://tickets.puppetlabs.com/browse/PUP-5475
If you find that it doesn't also timeout while waiting, please comment and/or file a pull request containing the change. At any rate, using something better than net.exe would be preferred.
The explanation is that the service takes too much time to start and does not communicate correctly with the starter.
When you write a service that initiate communications or DB connections you have to communicate with the Service Control Manager (SCM) to give the information that you are starting. Doing this kind of "I'am still starting message" the SCM can wait as mus time as you need to start. But much service writer or or tools to encapsulate exe files as services ignore that, so the SCM return "service could not be started". In Win32 this is handled by SetServiceStatus function, you will have much details there.
I've got a Talend process, deployed inside a .war file in a Tomcat app server. The .war has several jobs inside it, that are called using a get request pretty much like "http://server:port/TalendProject/method=runJob".
Not having the commercial license of Talend (using Open Studio at the moment) I don't have any way to control whether a job has failed or not in the middle of the execution.
I know I can check logs but not knowing the system I'm operating with, I can't be sure if a job has failed or is still running (after an exception for example).
My question is if there is any way to monitor the job that is running inside the Tomcat, this is, if there is any way to show the process that is being running.
My guess, but I'm not sure is that there is no way since the Talend Job is running encapsulated by the Talend .war, so the only thing I will see is the java calling this .war and everything below will be hidden.
I don't know anything about Talend, but I think you can use JConsole or better VisualVm to monitor your Tomcat server via JMX Agent. In order to monitor any Java application (as Tomcat) you have to invoke it with -Dcom.sun.management.jmxremote if tomcat is running in localhost.
If Tomcat is running in another host, you need to configure at least port JMX property -Dcom.sun.management.jmxremote.port=XXXX.
In order to add these properties to run tomcat, you have to change CATALINA_OPTS variable located into $TOMACT_HOME\bin\catalina.sh or $TOMACT_HOME\bin\catalina.bat file.
You have full documentation about Java JMX Agent monitoring here
This way you can monitor threads, memory consumption, gc activity, do profiling, thread dump, ... maybe it helps you to understand .
I have a Java application which should run on a server machine in the background (the application can be started by a command without GUI). The problem is: When I log off from the server, the application gets killed. I'm looking for something similar like nohup under Linux. I found some solutions, but I'm unsure, what is the best for my situation (Windows Server 2003, Java Application run from BAT-Skript, restart after booting the machine)? What are the pros and cons of the solutions?
psexec: Do the process really need to run under the SYSTEM account?
Combination of instsrv and srvany: But srvany should should never be used in a production environment
I have started the application with the scheduled tasks with the option run as NT AUTHORITY\SYSTEM. But after log off, the application was still killed. Can I get this working?
The DOS Task Scheduler AT command
Write a windows service (on Java, C# - I don't think it does matter)
Start the app from your service
or
Run your app as a window service using 3d party utility
The only way in which you can get a process to run without a user being logged in is to have the process run as a windows service or at the very least called by a windows service.
Just to add to above answers.
Consider using http://wrapper.tanukisoftware.com.
It's rather mature and popular (in contrast to ServiceEx and RunAsService).
Yes, a Windows Service is definitely the way to go but there are a few things to watch out for when you run a Java application as a Windows Service. Most of them are covered in this tutorial showing how to setup a java application with our commercial run-anything-as-a-service application, AlwaysUp:
http://www.coretechnologies.com/products/AlwaysUp/Apps/RunJavaApplicationAsAService.html
Beware: You will almost surely need the "-Xrs" flag on Windows 2003 to prevent the closing-on-logoff behavior but things can get tricky if you are catching shutdown events. Let me know if that is an issue for your situation.
This will be a network application that will always (or near as always as I can manage) be listening on a given port.
I'm fairly new to Java, and very new to non-web server side programming, so I'd like to get feedback from the community on my assumptions and preliminary plans.
I've read about jsvc ( http://commons.apache.org/daemon/jsvc.html ) and am currently operating on the assumption that this is the "best" way to write a daemon in java for a linux box (likely running centOS).
Can nagios be configured to monitor whether or not my daemon is running, and to alert me or the sys admin when it isn't? (I assume yes, but I'm not a very talented sys admin type)
This will be an SMPP client app (or ESME app I guess) which is why I've chosen Java as it seems to be a very mature platform for SMPP. However, I know that it's more "traditional" to write a daemon in C/C++. With modern Java, performing fairly uncomplicated tasks, am I likely to run into any major disadvantages?
What's the best way to manage deployment of new builds? Just stop the daemon and replace the binary as quickly as possible and restart?
Any other input would be greatly appreciated.
How to write a Java daemon that has 24/7 uptime...
We run a number of 24/365 applications on our Linux servers that just call the Java like the following -- no need for any C wrappers:
nohup java -D... -X... -jar something.jar ... < /dev/null > output.log 2>&1 &
That will put the jar running in the background (nohup ... &) with no input (< /dev/null) and the output (stdout and stderr) redirected to a logfile (> output.log 2>&1). We have distributed logging infrastructure but some console output (such as thread dumps) is still expected. These applications can run for months until we upgrade them.
Can nagios be configured to monitor whether or not my daemon is running, and to alert me or the sys admin when it isn't?
In terms of monitoring, there is much you can do. Nagios looks to have a JMX plugin for testing the information that jconsole displays. There are also a lot of native JMX logging and monitoring utilities out there. We have internal green/yellow/red indicators that can be pulled up using JMX and easily checked. I also have exported a simple JMX/HTTP service from each application to provide status information making it easy for 3rd party monitoring tools to detect failures.
This will be an SMPP client app (or ESME app I guess) which is why I've chosen Java as it seems to be a very mature platform for SMPP.
I assume you mean SMPP? If so then I see no reason why Java couldn't do a good job. Our applications do a wide variety of HTTP, UDP, SMTP, JDBC, LDAP, and other protocols in real time. We use Jgroups at lot which accomplishes a complete authenticated, encrypted, network stack in Java.
What's the best way to manage deployment of new builds? Just stop the daemon and replace the binary as quickly as possible and restart?
In terms of replacing a running binary on the fly, that it more complicated. We have VIPs up front and replace the binaries at our leisure. Our internal protocols are designed to failover. If you do not have a VIP then one thing to consider would be an orderly handoff. You boot the new jar and it talks to the application running the old jar when it is ready to bind to the new port. Then the old application unbinds and the new one binds immediately afterwards. Something like that.
Hope this helps.
If you really want to have something running non-stop on *nix, I recommend you have a look at daemontools.
There are some examples on how to do this here and here.
Basically svscan will spawn a process that monitors your java process from init, and every time it crashes, it gets restarted.