Develop Reference

Why does net.exe start <servicename> report a failure when the service starts?

I have a Java application that uses the Apache Daemon service installer to register it as a Windows service. I am using Puppet to run an exec{} block to register the service, which works, and then chains a service{} block to start the service. Puppet uses "net.exe start" to run the service, but that command reports an error, even though the service starts correctly.
The output from running the command in a powershell shell is:
PS C:\ProgramData\PuppetLabs\puppet\etc\modules> net start myservice
The myservice_descriptive_name service is starting.....
The myservice_descriptive_name service could not be started.
More help is available by typing NET HELPMSG 3523.
As I refresh the Windows service panel while this command is running, I see the state change from:
blank field -> starting -> started
Is this a problem caused by the apache wrapper, which is starting a jvm in a separate shell or some other side effect? And, more importantly, can I get around this problem in Puppet while still using the service{} block? Is it possible to substitute sc.exe, which does not suffer the same problem, short of using an exec{} block?

To take the questions in order:
The net start command reports failure because the service appears to have hung.
Yes, the problem is caused by the Apache wrapper.
Specifically, the wrapper is telling Windows that it will reach the first checkpoint within two seconds. Since there does not appear to be any way for the Java code to implement a checkpoint, or to change the wait hint, this means that the service must start within two seconds to be compliant with the Windows service specification.
(In principle, Windows is entitled to terminate your service at this point. So far as I know, no current versions of Windows do so, though they may log error messages.)
Short of modifying Puppet or (preferably) the Apache wrapper, the only obvious workaround is to ensure that your service "starts" immediately, rather than waiting for initialization to complete.
This is less than ideal, since it means that the service can't provide feedback to Puppet if it really does fail to initialize, but no worse than your suggestion of using sc start instead of net start.

JPBlanc's answer explains why the net.exe times out waiting on the service to start, even though it does end up starting. You can definitely try swapping out net.exe calls for sc.exe (Service Control) instead.
I've created a ticket to address this - https://tickets.puppetlabs.com/browse/PUP-5475
If you find that it doesn't also timeout while waiting, please comment and/or file a pull request containing the change. At any rate, using something better than net.exe would be preferred.

The explanation is that the service takes too much time to start and does not communicate correctly with the starter.
When you write a service that initiate communications or DB connections you have to communicate with the Service Control Manager (SCM) to give the information that you are starting. Doing this kind of "I'am still starting message" the SCM can wait as mus time as you need to start. But much service writer or or tools to encapsulate exe files as services ignore that, so the SCM return "service could not be started". In Win32 this is handled by SetServiceStatus function, you will have much details there.

Invoke Threads using JMX

How can I use JMX to invoke a thread using jConsole or jManage ?
I want to initially create 5 threads. Let them run. Then when one of them gets stuck, I want to create a new thread to continue operations.
I do not want to kill process until complete data is not processed / until really required.

You question seems a little bit vague; in general thread always runs some logic, so you should do some development here.
Basically JMX provides a way to install component (called MBean) and run it along with JVM process.
Java allows to start a JMX server along with the JVM process, in order to do that you should supply some properties to the process.
Then you can use this server for installing your own MBean that can do whatever you want, and of course run the thread.
Once you have a deployed mbean component and your jvm proces is up and running you can use jConsole and you should see your mbean among others.
Then just call the method.
There is a really good tutorial here
Hope this helps

Sending Signals to a Running JVM

I'm using a custom signal handler to catch TERM, ABRT and INT signals in a custom java daemon. I have this handler in the code so that I can send TERM signals to it and gracefully shutdown the program via the kill command. The signal handler works right now, but when I compile the code I'm receiving the following warning (many times over):
warning: sun.misc.SignalHandler is Sun proprietary API and may be removed in a future release
while using these classes:
import sun.misc.SignalHandler;
import sun.misc.Signal;
Is there a better way to send signals to a running JVM to initiate a shutdown of the main thread? I don't like having my code tied to this API when it could be removed in the future.
This code works on Solaris and HPUX today using 1.5.0_22 JVM. Any help or suggestions would be much appreciated. I used this document, from IBM, to develop the signal handler:
http://www.ibm.com/developerworks/java/library/i-signalhandling/

First of all, understand that this is just a standard warning for sun.misc package. They're letting you know that the API you're using is not a standard Java API. It doesn't mean that they're actively planning to remove this API in the future. http://java.sun.com/products/jdk/faq/faq-sun-packages.html
As far as your question, it's not quite clear to me why the default way java process handles kill signals is not sufficient for you: How to stop java process gracefully?. If you need to add additional logic, you can add a shutdown hook.
Still, If you're looking for other ways to let your java process know it's time to exit, you can have it listen on a socket, or stdin, or a named pipe...
You might also want to look into JVMTI

You could do this via JMX.
JMX is a standard set of apis that can be used to monitor and manage java applications.
Here are some links to get you started :
http://onjava.com/pub/a/onjava/2004/09/29/tigerjmx.html
http://www.ibm.com/developerworks/java/library/j-jtp09196/index.html?ca=drs
The main idea is this :
a) You will have a boolean variable, say isShutDownTrigerred. You will have a thread that will run a infinite loop, sleep for 2s, and keeps checking this variable value. When the value is true, you will execute code that will shutdown the application.
b) Then you write a mxbean (check the links above). This mxbean will be used to change the "isShutDownTrigerred" value to true. You can use a tool like jconsole /jManage to see and modify the mxbeans of a java application. As soon as the "isShutDownTriggered" is set to true, the above thread is going to know it and will execute the shutdown of the application

Is it possible to know if a process is waiting in Blocked state on a Receive() call on Linux?

My main purpose is to execute processes one by one in a round-robin fashion until one calls receive() and is blocked, so that the execution switches to the next process in the queue. There is a controller application which is coded in Java and it executes these processes(which are also Java applications) using Runtime.getRuntime().exec() and keeps the return values which are Process objects.
To achieve this purpose, I need to capture the receive() calls(or their states, which is blocked) and tell them to the controller(master) application.
I can go as low-level as you want if this is possible.. My first thought was to get this information from the driver and then tell it to my controller Java application. I have written a linux kernel network module which captures the send and receive operations, but AFAIK the socket.receive() function does not tell anything to the network driver.
So, I think the options are to get this information from either the JVM, somehow get it from a linux command or so, or possibly through the linux kernel module?
What are your suggestions?

If you want to know if your threads are blocked, or exactly what they are blocked on, you can either take a thread dump or use a tool like jvisualvm to attach to the process and take a look (in jvisualvm you would attach to the process, take a thread dump, and then look at the activity of each thread).

Have you looked at systemtap? Should be readily available on recent Fedora systems.
Best
Anders

I don't know if this will help you, but you could get information about the state of a Java thread on your machine using local attach.
1) Add the tools.jar to your classpath and use VirtualMachine.list() to get a list of the running JVM on you machine.
2) Attach to the JVM processed using VirtualMachine.attach(virtualMachineDescriptor)
3) Get the local connector address, vm.getAgentProperties().get("com.sun.management.jmxremote.localConnectorAddress");
4) Use JMXConnectorFactory.newJMXConnector(...) to connect to the JVM
5) From the JMX connection lookup up the ThreadMXBean
6) From the ThreadMXBean you get an array of ThreadInfos that describes all threads in the JVM.
7) From TheadInfo#getThreadState() you can check if the state is ThreadState.BLOCKED

You should use interprocess communication primitives in your worker processes to notify the controller application that they are ready to receive data.
You can't make assumptions about how the child processes implement their socket read. They could be using recv, or select, or poll, etc., to wait for network data.

There are actually a few points here. The Linux scheduler is smart enough to pre-empt a blocked task. Meaning, if you call receive() and there's nothing waiting to receive, your task will probably be put to sleep until such a time that the call will return. You don't need to handle the scheduling; the Linux kernel will do it for you.
That said, if you need to know whether your task is blocked from some daemon application, if you're willing to write an LKM, why not just get the task in the task list that you're interested in, and check its state?
Of course, simply checking the state of the task might not tell you exactly what you want. If your task state is TASK_INTERRUPTIBLE, it only tells you that your task is waiting on something, but it might not be a trivial matter to figure out what that something is. Similarly, your task can be in a TASK_RUNNING state and not actually be running on the CPU at the current moment (but, at least, in the TASK_RUNNING state you know your task isn't blocked).

You can just send a QUIT signal (Ctrl-\ on the console) to get a thread dump.

Is it possible to kill a Java Virtual Machine from another Virtual Machine?

I have a Java application that launches another java application. The launcher has a watchdog timer and receives periodic notifications from the second VM. However, if no notifications are received then the second virtual machine should be killed and the launcher will perform some additional clean-up activities.
The question is, is there any way to do this using only java? so far I have to use some native methods to perform this operation and it is somehow ugly.
Thanks!

I may be missing something but can't you call the destroy() method on the Process object returned by Runtime.exec()?

You can use java.lang.Process to do what you want. Once you have created the nested process and have a reference to the Process instance, you can get references to its standard out and err streams. You can periodically monitor those, and call .destroy() if you want to close the process. The whole thing might look something like this:
Process nestedProcess = new ProcessBuilder("java mysubprocess").start();
InputStream nestedStdOut = nestedProcess.getInputStream(); //kinda backwards, I know
InputStream nestedStdErr = nestedProcess.getErrorStream();
while (true) {
/*
TODO: read from the std out or std err (or get notifications some other way)
Then put the real "kill-me" logic here instead of if (false)
*/
if (false) {
nestedProcess.destroy();
//perform post-destruction cleanup here
return;
}
Thread.currentThread().sleep(1000L); //wait for a bit
}
Hope this helps,
Sean

You could also publish a service (via burlap, hessian, etc) on the second JVM that calls System.exit() and consume it from the watchdog JVM. If you only want to shut the second JVM down when it stops sending those periodic notifications, it might not be in a state to respond to the service call.
Calling shell commands with java.lang.Runtime.exec() is probably your best bet.

The usual way to do this is to call Process.destroy()... however it is an incomplete solution since when using the sun JVM on *nix destroy maps onto a SIGTERM which is not guaranteed to terminate the process (for that you need SIGKILL as well). The net result is that you can't do real process management using Java.
There are some open bugs about this issue see:
link text

OK the twist of the gist is as follows:
I was using the Process API to close the second virtual machine, but it wouldn't work.
The reason is that my second application is an Eclipse RCP Application, and I launched it using the eclipse.exe launcher included.
However, that means that the Process API destroy() method will target the eclipse.exe process. Killing this process leaves the Java Process unscathed. So, one of my colleagues here wrote a small application that will kill the right application.
So one of the solutions to use the Process API (and remove redundant middle steps) is to get away with the Eclipse launcher, having my first virtual machine duplicate all its functionality.
I guess I will have to get to work.

java.lang.Process has a waitFor() method to wait for a process to die, and a destroy() method to kill the subprocess.

You can have the java code detect the platform at runtime and fire off the platform's kill process command. This is really an refinement on your current solution.
There's also Process.destroy(), if you're using the ProcessBuilder API

Not exactly process management, but you could start an rmi server in the java virtual machine you are launching, and bind a remote instance with a method that does whatever cleanup required and calls System.exit(). The first vm could then call that remote method to shutdown the second vm.

You should be able to do that java.lang.Runtime.exec and shell commands.