I know how to "deamonize" a process (not to confused with Thread.setDaemon). There are some answers here and here and I'm using my own perl wrapper, which works fine.
But what I'd like to get now, is the parent process waiting for the Java process until it says "OK", i.e., until it has really started successfully (not only process started, but everything up and running well).
I could indicate this by writing a file, binding to a socket or alike, but it's ugly. Out of the eight items on the deamonize list, I only need the following three in a slightly simplified form:
Close standard input, standard output and standard error.
Run in the background (i.e., fork)
Ignore SIGHUP.
The first and last item can be done before process start, so only forking the process remains. Googling for "Java fork" is hopeless since the ForkJoinPool exists. Before I get my hands dirty, I'd like to know if
it's supported in Java 9 (then I'd simply wait)
someone did it already using JNA and what problems they ran in
there's a better solution
I don't care about Windows as it's for a Linux server.
I don't know how well this translates to Java, but from a syscall perspective:
When both the parent and child are under your full control, you should call pipe or socketpair to create your own communication channel, and specify it to the child via environment variables or command-line arguments. Remember to immediately close one half in each process (either between fork and exec if you have control there, or via the CLOEXEC flag - this means the child executable never has to know about the parent's end at all).
I'm not sure why you seem hesitant to use sockets - perhaps you're under the impression that local sockets take up ports (they don't) - though if all the data travels in one direction I'd prefer pipes just for clarity.
If it is possible for the child to create its own children, you should set the CLOEXEC flag on the inherited pipe as soon as possible.
You must send a positive message to indicate success; closing the pipe early must be considered an error (though you may also have explicit errors). Note that this means you don't have to track the exit value (useful if the "child" is actually a grandchild).
Alternatively, perhaps you should make something else do the work: your init system, X11, and DBUS? E.g., what were you thinking you should do if the child crashes?
Related
In our Netty application. We are moving all blocking calls in our code to run in a special backgroundThreadGroup.
I'd like to be able to log in production the threadName and the lineNumber of the java code that is about to execute a blocking operation. (i.e. sync File and Network IO)
That way I can grep for the logs looking at places were we might have missed to move our blocking code to the backgroundThreadGroup.
Is there a way to instrument the JVM so that it can tell me that?
Depends on what you mean by a "blocking operation".
In a broad sense, any operation that causes a voluntary context switch is blocking. Trying to do something special about them is absolutely impractical.
For example, in Java, any method containing synchronized is potentially blocking. This includes
ConcurrentHashMap.put
SecureRandom.nextInt
System.getProperty
and many more. I don't think you really want to avoid calling all these methods that look normal at a first glance.
Even simple methods without any synchronization primitives can be blocking. E.g., ByteBuffer.get may result in a page fault and a blocking read on the OS level. Furthermore, as mentioned in comments, there are JVM level blocking operations that are not under your control.
In short, it's impractical if not impossible to find all places in the code where a blocking operation happens.
If, however, you are interested in finding particular method calls that you believe are bad (like Thread.sleep and Socket.read), you can definitely do so. There is a BlockHound project specifically for this purpose. It already has a predefined list of "bad" methods, but can be customized with your own list.
There is a library called BlockHound, that will throw an exception unless you have configured BlockHound to ignore that specific blocking call
This is how you configure BlockHound for Netty: https://github.com/violetagg/netty/blob/625f9d5781ed85bfaca6fa4e826d0d46d70fdbd8/common/src/main/java/io/netty/util/internal/Hidden.java
(You can improve the above code by replacing the last line with builder.nonBlockingThreadPredicate(
p -> p.or(thread -> thread instanceof FastThreadLocalThread)); )
see https://github.com/reactor/BlockHound
see https://blog.frankel.ch/blockhound-how-it-works/
I personally used it to find all blocking call within our Netty based service.
Good Luck
Everywhere I look about how to forcefully stop a thread in Java, I see "just do an exit variable check instead, your program is broken if you need to force kill."
I have a rather unique situation though. I am writing a Java program that dynamically loads and runs other Java classes in a separate thread. (No comments about security risks please, this is a very specific use case).
The trouble is, since other people will have written the classes that need to be loaded, there's no way to guarantee they'll implement the stop checking and whatnot correctly. I need a way to immediately terminate their thread, accepting all the risks involved. Basically I want to kill -9 their thread if I need to. How can I do this in Java?
Update: here's a bit more info:
This is actually an Android app
The user code depends on classes in my application
A user class must be annotated with #UserProgram in order to be "registered" by my application
The user also has the option of building their classes right into the application (by downloading a project with the internal classes already compiled into a libraries and putting their classes in a separate module) rather than having them dynamically loaded from a JAR.
The user classes extend from my template class which has a runUserProgram() method that they override. Inside that method, they are free to do anything they want. They can check isStopRequested() to see if I want them to stop, but I have no guarantee that they'll do that.
On startup, my application loads any JARs specified and scans both all the classes in the application and the classes in those JARs to find any classes annotated with the aforementioned annotation. Once a list of those classes is built, it is fed into the frontend where the UI provides a list of programs that can be run. Once a program is selected, a "start" button must be pressed to actually start it. When it is pressed, the button changes to a "stop" button and a callback is fired into the backend to load up the selected class in a new thread and call the runUserProgram() method. When the "stop" button is pressed, a variable is set which causes isStopRequested() to return true.
You can kill -9 it by running in its own process i.e. start with a ProcessBuilder and call Process.destroyForcibly() to kill it.
ProcessBuilder pb = new ProcessBuilder("java", "-cp", "myjar.jar");
pb.redirectErrorStream();
Process process = pb.start();
// do something with the program.
Scanner sc = new Scanner(process.getOutputStream());
while (sc.hasNextLine()) {
System.out.println(sc.nextLine());
}
// when done, possibly in another thread so it doesn't get blocked by reading.
process.waitFor(1, TimeUnit.SECONDS);
if (process.isAlive())
process.destroyForcibly();
Java 8 had Thread.stop(). The problem is that it could only work reasonably for very limited use cases, so limited you were better off using interrupts, and if the code isn't trusted, neither are any good.
There is the deprecated Thread.stop() but don't use it.
There is no way to cleanly terminate another thread without it cooperating.
The thread can be in a state where it allocated some memory, or added some objects to some global state, locked some mutexes, etc. If you kill it at the wrong moment, you risk leaking memory or even causing a deadlock.
It would be possible through JNI, under Windows there is a TerminateThread API that you can call, there is (hopefully) probably a similar thing under Android. The trouble will be getting the thread's native handle, you would need to obtain that when your user "program" is first loaded, probably by calling another JNI method from the thread in question as part of the initialisation process and getting the current thread handle from that.
I have not tried this myself, best case is that this "works" and kills the thread, but it is going to cause that thread to leak resources. Worst case is that it will leave the JVM in an inconsistent state internally, which will probably crash your entire application.
I really think this is a Bad Idea.
A better design, if you want to allow this, is to run your user code in another process and communicate with it via sockets or pipes. This way you can relatively safely terminate the other process if necessary. It's more work, but it's going to be a lot better in the long run.
You shold use Thread.interrupt().
I am monitoring and Minecraft server and I am making a setup file in Python. I need to be able to run two threads, one running the minecraft_server.jar in the console window, while a second thread is constantly checking the output of the minecraft_server. Also, how would I input into the console from Python after starting the Java process?
Example:
thread1 = threading.Thread(target=listener)
thread2 = minecraft_server.jar
def listener():
if minecraft_server.jarOutput == "Server can't keep up!":
sendToTheJavaProccessAsUserInputSomeCommandsToRestartTheServer
It's pretty hard to tell here, but I think what you're asking is how to:
Launch a program in the background.
Send it input, as if it came from a user on the console.
Read its output that it tries to display to a user on the console.
At the same time, run another thread that does other stuff.
The last one is pretty easy; in fact, you've mostly written it, you just need to add a thread1.start() somewhere.
The subprocess module lets you launch a program and control its input and output. It's easiest if you want to just feed in all the input at once, wait until it's done, then process all the output, but obviously that's not your case here, so it's a bit more involved:
minecraft = subprocess.Popen(['java', 'path/to/minecraft_server.jar', '-other', 'args],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
I'm merging stdout and stderr together into one pipe; if you want to read them separately, or send stderr to /dev/null, or whatever, see the docs; it's all pretty simple. While we're making assumptions here, I'm going to assume that minecraft_server uses a simple line-based protocol, where every command, every response, and every info message is exactly one line (that is, under 1K of text ending in a \n).
Now, to send it input, you just do this:
minecraft.stdin.write('Make me a sandwich\n')
Or, in Python 3.x:
minecraft.stdin.write(b'Make me a sandwich\n')
To read its output, you do this:
response = minecraft.stdout.readline()
That works just like a regular file. But note that it works like a binary file. In Python 2.x, the only difference is that newlines don't get automatically converted, but in Python 3.x, it means you can only write bytes (and compatible objects), not strs, and you will receive bytes back. There are good reasons for that, but if you want to get pipes that act like text files instead, see the universal_newlines (and possibly bufsize) arguments under Frequently Used Arguments and Popen Constructor.
Also, it works like a blocking file. With a regular file, this rarely matters, but with a pipe, it's quite possible that there will be data later, but there isn't data yet (because the server hasn't written it yet). So, if there is no output yet (or not a complete line's worth, since I used readline()), your thread just blocks, waiting until there is.
If you don't want that, you probably want to create another thread to service stdout. And its function can actually look pretty similar to what you've got:
def listener():
for line in minecraft.stdout:
if line.strip() == "Server can't keep up!":
minecraft.stdin.write("Restart Universe\n")
Now that thread can block all day and there's no problem, because your other threads are still going.
Well, not quite no problem.
First it's going to be hard to cleanly shut down your program.
More seriously, the pipes between processes have a fixed size; if you don't service stdout fast enough, or the child doesn't service stdin fast enough, the pipe can block. And, the way I've written things, if the stdin pipe blocks, we'll be blocked forever in that stdin.write and won't get to the next read off stdout, so that can block too, and suddenly we're both waiting on each other forever.
You can solve this by having another thread to service stdout. The subprocess module itself includes an example, in the Popen._communicate function used by all the higher-level functions. (Make sure to look at Python 3.3 or later, because earlier versions had bugs.)
If you're in Python 3.4+ (or 3.3 with a backport off PyPI), you can instead use asyncio to rewrite your program around an event loop and handle the input and output the same way you'd write a reactor-based network server. That's what all the cool kids are doing in 2017, but back in late 2014 many people still thought it looked new and scary.
If all of this is sounding like a lot more work than you signed on for, you may want to consider using pexpect, which wraps up a lot of the tedious details, and makes some simplifying assumptions that are probably true in your case.
My application takes a lot of measurements of it's internal processes. For example I time certain methods, I time external webservice calls and I also have variables which have a changing value, and processes which have a 'state' (e.g. PAUSED, WAITING etc).
The application uses 100 to 200 threads, and each bit of data would be associated with a particular thread.
I am looking for some software that I can channel all this information into that would produce useful metrics and graphs of the data (ideally in real time or close to real time), let me set thresholds to trigger warnings, would allow me to filter the data by thread or thread group, etc etc.
The application is performing time critical tasks so the software/api would need to be very fast and never block.
The application is written in java, and ideally the software/api would be in java as well. I think what I'm looking for is called Event Stream Processing, but I'm really not sure what language to use to describe it.
All I've found so far are Esper and ERMA. Can anyone give me a recommendation? I'm the only one working on this project so I'm hoping for something that is pretty easy to set up and use, and has a workable front end.
In the end I found Graphite which was pretty close to being exactly what I wanted. Not the simplest to set up and configure however, but I got it working in the end.
http://graphite.wikidot.com/
In my case I send data directly from my application to Statsd (via UDP), which collects the data and does some pre processing before it ends up in the whisper back end, there is a simple example of a java interface here https://github.com/etsy/statsd/commit/2253223f3c19d2149d65ec5bc802198ff93da4cb
Alternatively you could send your data directly to graphite, example here http://neopatel.blogspot.co.uk/2011/04/logging-to-graphite-monitoring-tool.html
In my application which runs user submitted code[1] in separate threads, there might be some cases where the code might take very long to run or it might even have an infinite loop! In that case how do I stop that particular thread?
I'm not in control of the user code, so I cannot check for Thread.interrupted() from the inside. Nor can I use Thread.stop() carelessly. I also cannot put those code in separate processes.
So, is there anyway to handle this situation?
[1] I'm using JRuby, and the user code is in ruby.
With the constraints you've provided:
User submitted code you have no control over.
Cannot force checks for Thread.interrupted().
Cannot use Thread.stop().
Cannot put the user code in a process jail.
The answer to your question is "no, there is no way of handling this situation". You've pretty much systematically designed things so that you have zero control over untrusted third-party code. This is ... a suboptimal design.
If you want to be able to handle anything, you're going to have to relax one (or preferably more!) of the above constraints.
Edited to add:
There might be a way around this for you without forcing your clients to change code if that is a(nother) constraint. Launch the Ruby code in another process and use some form of IPC mechanism to do interaction with your main code base. To avoid forcing the Ruby code to suddenly have to be coded to use explicit IPC, drop in a set of proxy objects for your API that do the IPC behind the scenes which themselves call proxy objects in your own server. That way your client code is given the illusion of working inside your server while you jail that code in its own process (which you can ultimately kill -9 as the ultimate sanction should it come to that).
Later you're going to want to wean your clients from the illusion since IPC and native calls are very different and hiding that behind a proxy can be evil, but it's a stopgap you can use while you deprecate APIs and move your clients over to the new APIs.
I'm not sure about the Ruby angle (or of the threading angle) of things here, but if you're running user-submitted code, you had best run it in a separate process rather than in a separate thread of the same process.
Rule number one: Never trust user input. Much less if the input is code!
Cheers
Usually you have a variable to indicate to stop a thread. Some other thread then would set this variable to true. Finally you periodically check, whether the variable is set or not.
But given that you can't change user code , I am afraid there isn't a safe way of doing it.
For Running Thread Thread.Interrupt wont actually stop as sfussenegger mentioned aforth (thanks sfussenegger recollected after reading spec).
using a shared variable to signal that it should stop what it is doing. The thread should check the variable periodically,(ex : use a while loop ) and exit in an orderly manner.
private boolean isExit= false;
public void beforeExit() {
isExit= true;
}
public void run() {
while (!isExit) {
}
}