Develop Reference

Why does net.exe start <servicename> report a failure when the service starts?

I have a Java application that uses the Apache Daemon service installer to register it as a Windows service. I am using Puppet to run an exec{} block to register the service, which works, and then chains a service{} block to start the service. Puppet uses "net.exe start" to run the service, but that command reports an error, even though the service starts correctly.
The output from running the command in a powershell shell is:
PS C:\ProgramData\PuppetLabs\puppet\etc\modules> net start myservice
The myservice_descriptive_name service is starting.....
The myservice_descriptive_name service could not be started.
More help is available by typing NET HELPMSG 3523.
As I refresh the Windows service panel while this command is running, I see the state change from:
blank field -> starting -> started
Is this a problem caused by the apache wrapper, which is starting a jvm in a separate shell or some other side effect? And, more importantly, can I get around this problem in Puppet while still using the service{} block? Is it possible to substitute sc.exe, which does not suffer the same problem, short of using an exec{} block?

To take the questions in order:
The net start command reports failure because the service appears to have hung.
Yes, the problem is caused by the Apache wrapper.
Specifically, the wrapper is telling Windows that it will reach the first checkpoint within two seconds. Since there does not appear to be any way for the Java code to implement a checkpoint, or to change the wait hint, this means that the service must start within two seconds to be compliant with the Windows service specification.
(In principle, Windows is entitled to terminate your service at this point. So far as I know, no current versions of Windows do so, though they may log error messages.)
Short of modifying Puppet or (preferably) the Apache wrapper, the only obvious workaround is to ensure that your service "starts" immediately, rather than waiting for initialization to complete.
This is less than ideal, since it means that the service can't provide feedback to Puppet if it really does fail to initialize, but no worse than your suggestion of using sc start instead of net start.

JPBlanc's answer explains why the net.exe times out waiting on the service to start, even though it does end up starting. You can definitely try swapping out net.exe calls for sc.exe (Service Control) instead.
I've created a ticket to address this - https://tickets.puppetlabs.com/browse/PUP-5475
If you find that it doesn't also timeout while waiting, please comment and/or file a pull request containing the change. At any rate, using something better than net.exe would be preferred.

The explanation is that the service takes too much time to start and does not communicate correctly with the starter.
When you write a service that initiate communications or DB connections you have to communicate with the Service Control Manager (SCM) to give the information that you are starting. Doing this kind of "I'am still starting message" the SCM can wait as mus time as you need to start. But much service writer or or tools to encapsulate exe files as services ignore that, so the SCM return "service could not be started". In Win32 this is handled by SetServiceStatus function, you will have much details there.

Start/stop java application from an external script

I have a Stand-alone Java application. At the moment I am running this Java application using a start-script i.e. startApplicatoin.bat in windows and startApplicatoin.sh in Linux which sets up the class-paths and then it executes: java -classpath .
Now I have to add a stopApplication.bat and stopApplication.sh script. This stop script has to shutdown/close this java application gracefully.
To achieve this I am planning to take the following steps:
1. When my java application runs it will store the process-id of the launched application in a file i.e. in a known file myapplication.pid.
Looks like ManagementFactory.getRuntimeMXBean().getName() call will work on both Linux and Windows to get the process ID. So I shall collect process ID in this way and will store it in the specified file myapplication.pid.
2. Then when running stop application script, this script will issue a “kill” request to the process-id as specified by that myapplication.pid file.
For Windows I shall run the "taskkill" command to stop this application. And for Linux environment "kill" command will serve that purpose.
And in my java code I shall add a addShutdownHook which will enable the graceful shutdown operations that I want to run i.e. there I shall handle whatever stuffs I want to persist before this program is going to stop.
http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#addShutdownHook%28java.lang.Thread%29
Now I would like to do a sanity check to ensure the way I am thinking is the proper way to do. Or there is a better way to do this. Any suggestion is appreciated. And thanks in advance.

If you're wanting a "graceful" shutdown, it may be more practical (and easier cross-platform) to open a socket in your long-running process and have your "stop" script connect to it and issue a shutdown command; this might even be practical through JMX, depending on how your application overall is structured. Approaches that are "inline" rather than requiring interaction with the OS are generally easier to reason about and test.

This looks like a Daemon.
The easiest way to run a daemon with start/stop functionality without resorting to a lot of scripting is with jsvc. This allows your code to implement an interface with four methods:
void init(String[] arguments): Here open configuration files, create a trace file, create ServerSockets, Threads
void start(): Start the Thread, accept incoming connections
void stop(): Inform the Thread to terminate the run(), close the ServerSockets
void destroy(): Destroy any object created in init()
You then have platform specific binaries that deal with keeping track of the process and stopping it when requested to do so.
The most useful thing is that jsvc can start a process as a superuser (root on unix) and then drop to a peon user the for auction running of the process.
This is how Tomcat (for example) works, it starts as root and performs privileged actions such as binding to port 80. It then drops down to a peon use called tomcat for security reasons.

How to start a application without getting killed after log off?

I have a Java application which should run on a server machine in the background (the application can be started by a command without GUI). The problem is: When I log off from the server, the application gets killed. I'm looking for something similar like nohup under Linux. I found some solutions, but I'm unsure, what is the best for my situation (Windows Server 2003, Java Application run from BAT-Skript, restart after booting the machine)? What are the pros and cons of the solutions?
psexec: Do the process really need to run under the SYSTEM account?
Combination of instsrv and srvany: But srvany should should never be used in a production environment
I have started the application with the scheduled tasks with the option run as NT AUTHORITY\SYSTEM. But after log off, the application was still killed. Can I get this working?
The DOS Task Scheduler AT command

Write a windows service (on Java, C# - I don't think it does matter)
Start the app from your service
or
Run your app as a window service using 3d party utility

The only way in which you can get a process to run without a user being logged in is to have the process run as a windows service or at the very least called by a windows service.

Just to add to above answers.
Consider using http://wrapper.tanukisoftware.com.
It's rather mature and popular (in contrast to ServiceEx and RunAsService).

Yes, a Windows Service is definitely the way to go but there are a few things to watch out for when you run a Java application as a Windows Service. Most of them are covered in this tutorial showing how to setup a java application with our commercial run-anything-as-a-service application, AlwaysUp:
http://www.coretechnologies.com/products/AlwaysUp/Apps/RunJavaApplicationAsAService.html
Beware: You will almost surely need the "-Xrs" flag on Windows 2003 to prevent the closing-on-logoff behavior but things can get tricky if you are catching shutdown events. Let me know if that is an issue for your situation.

How to write a Java daemon

This will be a network application that will always (or near as always as I can manage) be listening on a given port.
I'm fairly new to Java, and very new to non-web server side programming, so I'd like to get feedback from the community on my assumptions and preliminary plans.
I've read about jsvc ( http://commons.apache.org/daemon/jsvc.html ) and am currently operating on the assumption that this is the "best" way to write a daemon in java for a linux box (likely running centOS).
Can nagios be configured to monitor whether or not my daemon is running, and to alert me or the sys admin when it isn't? (I assume yes, but I'm not a very talented sys admin type)
This will be an SMPP client app (or ESME app I guess) which is why I've chosen Java as it seems to be a very mature platform for SMPP. However, I know that it's more "traditional" to write a daemon in C/C++. With modern Java, performing fairly uncomplicated tasks, am I likely to run into any major disadvantages?
What's the best way to manage deployment of new builds? Just stop the daemon and replace the binary as quickly as possible and restart?
Any other input would be greatly appreciated.

How to write a Java daemon that has 24/7 uptime...
We run a number of 24/365 applications on our Linux servers that just call the Java like the following -- no need for any C wrappers:
nohup java -D... -X... -jar something.jar ... < /dev/null > output.log 2>&1 &
That will put the jar running in the background (nohup ... &) with no input (< /dev/null) and the output (stdout and stderr) redirected to a logfile (> output.log 2>&1). We have distributed logging infrastructure but some console output (such as thread dumps) is still expected. These applications can run for months until we upgrade them.
Can nagios be configured to monitor whether or not my daemon is running, and to alert me or the sys admin when it isn't?
In terms of monitoring, there is much you can do. Nagios looks to have a JMX plugin for testing the information that jconsole displays. There are also a lot of native JMX logging and monitoring utilities out there. We have internal green/yellow/red indicators that can be pulled up using JMX and easily checked. I also have exported a simple JMX/HTTP service from each application to provide status information making it easy for 3rd party monitoring tools to detect failures.
This will be an SMPP client app (or ESME app I guess) which is why I've chosen Java as it seems to be a very mature platform for SMPP.
I assume you mean SMPP? If so then I see no reason why Java couldn't do a good job. Our applications do a wide variety of HTTP, UDP, SMTP, JDBC, LDAP, and other protocols in real time. We use Jgroups at lot which accomplishes a complete authenticated, encrypted, network stack in Java.
What's the best way to manage deployment of new builds? Just stop the daemon and replace the binary as quickly as possible and restart?
In terms of replacing a running binary on the fly, that it more complicated. We have VIPs up front and replace the binaries at our leisure. Our internal protocols are designed to failover. If you do not have a VIP then one thing to consider would be an orderly handoff. You boot the new jar and it talks to the application running the old jar when it is ready to bind to the new port. Then the old application unbinds and the new one binds immediately afterwards. Something like that.
Hope this helps.

If you really want to have something running non-stop on *nix, I recommend you have a look at daemontools.
There are some examples on how to do this here and here.
Basically svscan will spawn a process that monitors your java process from init, and every time it crashes, it gets restarted.

Jboss/Tomcat monitor

I'm looking for a tool (in linux) that can montior a tomcat/jboss process and if the process fails, I can respawn either or both without having to manually ssh into the box, do any house keeping and then start them up again. I'm not too sure if there is a good tool out there that can monitor the health of jboss/tomcat and report on it's performance. I know jvisualvm gives you various tools, but I'm looking for a disaster recovery solution that is a bit higher level than jvisualvm.

Java Service Wrapper is an application that wraps your Java process and installs it in the system using service (Windows) or daemon (Linux). It pings the VM periodically and restarts it when it does not respond. Worked for us in production with several application, including Tomcat, JBoss, Mule, etc. Actually Mule ESB is even bundled with this application in the distribution.
Also you don't have to run the application manually when the system starts.

I'm currently working on a daemon to do this and more, since JOPR nor naggios didn't do what we needed, but those are good tools you could use. I'm not sure but JOPR (or whatever is called today) can restart your servers in case something goes wrong.
A custom made solution as we're working on shouldn't take you more than a week. The main problem, is that to start either JBoss or Tomcat you have to call the startup scripts. But the startup script will restart the service if the exit code is 10, something like this:
while $? -ne 10; do
start_jboss
done
So, this daemon which is made in Java uses JMX to connect to the JBoss server and tells JBoss to go down and exit with status code 10 using a method in a MBean. I'm at home, so I'm not sure the exact name of the MBean you have to call for this but I'll provide more info tomorrow.

I am using monit to control the launch of Tomcat/JBoss.