I have a gobbler that reads the output from a Process.
There is a case where we kill the process programatically using its PID and the external Windows taskkill command.
It is a 16-Bit DOS process
We taskkill because it is a DOS 16-bit process and Process.destroyForcibly() does not work with it because it resides in the ntvdm subsystem and the best way is to get the PID and use 'taskkill /T /F' which does indeed kill it and any children.
Normally, we have no problem with our DOS 16-bit (or 32 bit) processes. This one has some file locks in place. It is especially important that we ensure it is dead to have the OS release the locks.
We close all streams before and after the kill
Prior to calling taskkill, we attempt to flush and close all streams in an executor: in,out,err. After calling taskkill, we verify that all streams are closed by re-closing them.
We call Thread.interrupt() on all gobblers after the kill
Now, after the kill success, which is confirmed in the OS as well, the gobbler is still running and it does not respond to Thread.interrupt().
We even do a last-ditch Thread.stop (gasp!)
And furthermore, we have invoked Thread.stop() on it and it still stays waiting at the read stage ...
So, it seems, we are unable to stop the std-out and std-in gobblers on our Processes streams.
We know Thread.stop() is deprecated. To be somewhat safe, we catch ThreadDeath
then clean any monitors and then rethrow ThreadDeath. However,
ThreadDeath never in fact gets thrown and the thread just keeps on
waiting on inputStream.read ..
so Thread.stop being deprecated in this case is a moot point
... because it does not do anything.
Just so no one flames me and so that I have a clean conscience,
We have removed Thread.stop() from our production code.
I am not surprised that the Thread does not interrupt since that only happens on some InputStreams and not all reads are incorruptible. But I am surprised that the Thread will not stop when Thread.stop is invoked.
Thread trace shows
A thread trace shows that both main-in and main-er (the two outputs from the process) are still running even after the streams are closed, thread is interrupted and last ditch Thread.stop is called.
The task is dead, so why care about idle blocked gobblers?
It is not that we care that the gobblers won't quit. But we hate threads running that just pile up and clog the system. This particular process is called by a webserver and then .. it could amount to several hundred idle threads in a blocking state on dead processes...
We have tried launching the process two ways with no difference ...
run(working, "cmd", "/c", "start", "/B", "/W", "/SEPARATE", "C:\\workspace\\dotest.exe");
run(working, "cmd", "/c", "C:\\workspace\\dotest.exe");
The gobbler is in a read like this:
try (final InputStream is = inputStream instanceof BufferedInputStream
? inputStream : new BufferedInputStream(inputStream, 1024 * 64);
final BufferedReader br = new BufferedReader(new InputStreamReader(is, charset))) {
String line;
while ((line = br.readLine()) != null) {
lineCount++;
lines.add(line);
if (Thread.interrupted()) {
Thread.currentThread().interrupt();
throw new InterruptedException();
}
}
eofFound = true;
}
Our destroyer calls this on the gobbler thread after the taskkill:
int timeLimit = 500;
t.interrupt();
try {
t.join(timeLimit);
if (t.isAlive()) {
t.stop();
// we knows it's deprecated but we catch ThreadDeath
// then clean any monitors and then rethrow ThreadDeath
// But ThreadDeath never in fact gets thrown and the thread
// just keeps on waiting on inputStream.read ..
logger.warn("Thread stopped because it did not interrupt within {}ms: {}", timeLimit, t);
if (t.isAlive()) {
logger.warn("But thread is still alive! {}", t);
}
}
} catch (InterruptedException ie) {
logger.info("Interrupted exception while waiting on join({}) with {}", timeLimit, t, ie);
}
This is a snippet of the log output:
59.841 [main] INFO Destroying process '5952'
04.863 [main] WARN Timeout waiting for 'Close java.io.BufferedInputStream#193932a' to finish
09.865 [main] WARN Timeout waiting for 'Close java.io.FileInputStream#159f197' to finish
09.941 [main] DEBUG Executing [taskkill, /F, /PID, 5952].
10.243 [Thread-1] DEBUG SUCCESS: The process with PID 5952 has been terminated.
10.249 [main] DEBUG java.lang.ProcessImpl#620197 stopped with exit code 0
10.638 [main] INFO Destroyed WindowsProcess(5952) forcefully in 738 ms.
11.188 [main] WARN Thread stop called because it did not interrupt within 500ms: Thread[main-in,5,main]
11.188 [main] WARN But thread is still alive! Thread[main-in,5,main]
11.689 [main] WARN Thread stop because it did not interrupt within 500ms: Thread[main-err,5,main]
11.689 [main] WARN But thread is still alive! Thread[main-err,5,main]
Note: prior to calling taskkill, the Process std-out and std-err will not close. But they are closed manually after the taskkill (not shown in log because success).
Related
I'm new to Java Multithreading. Curious to know the state of idle thread incase of ThreadPoolExecutor. Is it in RUNNABLE/WAITING?
In case the idle threads are in RUNNABLE state, how are the new tasks attached to idle threads? AFAIK we assign a runnable/callable object to thread/pool. But my question is how does ThreadPoolExecutor assign queued runnable objects to idle thread??
It's easy enough to find out:
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;
import java.io.IOException;
public class ThreadExample {
public static void main(String[] args) throws IOException {
Executor executor = Executors.newFixedThreadPool(5);
// force the threads to be started
for (int i = 0; i < 5; i++) {
executor.execute(() -> {
try {Thread.sleep(1000);} catch (InterruptedException e) {
}
});
}
// don't terminate
System.in.read();
}
}
Run it:
$ javac ThreadExample.java
$ java ThreadExample
In another console, having waited at least one second for the tasks to complete:
$ ps
PID TTY TIME CMD
3640 ttys000 0:00.25 -bash
5792 ttys000 0:00.15 java ThreadExample
5842 ttys001 0:00.05 -bash
$ jstack 5792
...
"pool-1-thread-1" #12 prio=5 os_prio=31 cpu=1.77ms elapsed=13.37s tid=0x00007fe99f833800 nid=0xa203 waiting on condition [0x00007000094b2000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base#11.0.2/Native Method)
- parking to wait for <0x000000061ff9e998> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base#11.0.2/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base#11.0.2/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.LinkedBlockingQueue.take(java.base#11.0.2/LinkedBlockingQueue.java:433)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base#11.0.2/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base#11.0.2/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base#11.0.2/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base#11.0.2/Thread.java:834)
...
All the pool threads are in that state.
Curious to know the state of idle thread in case of ThreadPoolExecutor. Is it in RUNNABLE/WAITING?
It will be WAITING. It is waiting (in a Queue.take() call) for a new task to appear on the work queue. In the current implementations this involves a mechanism similar to wait / notify.
Your second question is therefore moot.
However, it is worth noting the following:
No "idle" thread will ever be RUNNABLE.
In current generation HotSpot JVMs, the actual scheduling (deciding which threads get priority and assigning them a core to run on) is handled by the operating system.
In Loom JVMs (Loom is still an Incubator project), light-weight virtual threads ("fibres") are scheduled (to a native thread) by the JVM rather than the OS.
I have a client that needs to disconnect from one server and connect to another. Its taking about 16 seconds. I still haven't debugged the connection logic, but I can see the shutdown of the channel is taking 5 seconds. Is this expected behavior, or should I be looking for thread starvation in my code.
LOG.debug("==============SHUTTING DOWN MANAGED CHANNEL");
long startTime=System.currentTimeMillis();
channel.shutdown().awaitTermination(20, SECONDS);
long endTime=System.currentTimeMillis();
LOG.debug("Time to shutdown channel ms = {}",endTime-startTime);
LOG.debug("==============RETURN FROM SHUTTING DOWN MANAGED CHANNEL");
From the log
2018-07-09 14:41:23,143 DEBUG [com.ticomgeo.ftc.client.FTCClient] (EE-ManagedExecutorService-singleThreaded-Thread-1) ==============SHUTTING DOWN MANAGED CHANNEL
2018-07-09 14:41:28,151 INFO [io.grpc.internal.ManagedChannelImpl] (grpc-default-worker-ELG-1-1) [io.grpc.internal.ManagedChannelImpl-1] Terminated
2018-07-09 14:41:28,152 DEBUG [com.ticomgeo.ftc.client.FTCClient] (EE-ManagedExecutorService-singleThreaded-Thread-1) Time to shutdown channel ms = 5009
2018-07-09 14:41:28,152 DEBUG [com.ticomgeo.ftc.client.FTCClient] (EE-ManagedExecutorService-singleThreaded-Thread-1) ==============RETURN FROM SHUTTING DOWN MANAGED CHANNEL
There are two shutdown functions, shutdown and shutdownNow. Is there any chance you have a calls going that are blocking shutdown? You may be better served by shutdownNow.
shutdown
Initiates an orderly shutdown in which preexisting calls continue but new calls are rejected.
shutdownNow
Initiates a forceful shutdown in which preexisting and new calls are rejected. Although forceful, the shutdown process is still not instantaneous; isTerminated() will likely return false immediately after this method returns.
I am writing an application where it has two threads:
[1] main
[2] threadX
main thread is command line program and starts an application(mongodb) in background and then exits.
threadX is the Server part of my application which requires to be active as long as the application(mongodb) is running.
main takes two arguments: start and stop.
On start: it starts -> start mongod and threadX.
On stop: it starts -> kills mongod and (should)stops threadX.
But even after setting threadX as daemon it stops after main thread exits.
Proof:
public class Index {
public static void main(String [] args) {
Logger log = Logger.getLogger("Index logging");
try {
FileHandler logfile = new FileHandler("test.log");
logfile.setFormatter(new SimpleFormatter());
log.addHandler(logfile);
} catch (SecurityException | IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Thread daemonThread = new Thread() {
public void run() {
while (true) {
try {
log.log(Level.INFO,"Try block executed");
Thread.sleep(1000l);
} catch (Throwable t) {
t.printStackTrace();
}
}
}
#Override
public void finalize() {
log.log(Level.INFO,"Finalize method called");
}
};
daemonThread.setDaemon(true);
daemonThread.start();
try {
Thread.sleep(3500l);
} catch (Throwable t) {
//NO-OP
}
}
}
Output:
Apr 08, 2017 2:30:54 PM num.Index$1 run
INFO: Try block executed
Apr 08, 2017 2:30:55 PM num.Index$1 run
INFO: Try block executed
Apr 08, 2017 2:30:56 PM num.Index$1 run
INFO: Try block executed
Apr 08, 2017 2:30:57 PM num.Index$1 run
INFO: Try block executed
As you can see it does not continues to write logs after main exits
What i want is a thread that can be started in background and continue to run even when main thread exits and that thread can be stopped after i restart my app. Just like console applications which starts processes in background and quit leaving console available to user. The user can then use command to stop that background process.
Daemon threads do exit after the main method exits - if you do not want your thread to exit you should not make it a daemon thread.
Remove the line
daemonThread.setDaemon(true);
or change it to
daemonThread.setDaemon(false);
From the Javadoc of the setDaemon method:
The Java Virtual Machine exits when the only threads running are all
daemon threads.
Create a User Thread and not a Daemon Thread. Why do you need to set the Thread as Daemon?
JVM will not terminate a User Thread and it will wait for it to finish before exiting unlike Daemon Thread.
Hope this Helps
If I get right at what you're trying to accomplish: you want a program to control mongodb. Separate invocations of the program may have diffrent commands as their command line arguments.
In the case of "start": your app should start mongodb as a separate process so it can exit right after mongodb started. You wont need thread to keep mongodb running.
In the case of "stop": You'll have to connect to the mongodb process and tell it to shut down gracefully, wait for it to end and end your own program.
I am doing kill -15 <PID> on my working jvm and it seems completely ignored.
The invironment is:
Linux 2.6 kernel
jdk 1.6.0_20-x86-64
There are no references to sun.misc.SignalHandler in the project. The only (quite lame) clue I have is call to AbstractApplicationContext.registerShutdownHook() in main. JVM startup args do not contain anything related to signal handling.
There is nothing in logs (DEBUG level) and nothing printed out to stdout in reaction to kill -15.
How do I find out what causes ignoring of SIGTERM?
Normally, Signals 1 (SIGHUP), 2 (SIGINT), 4 (SIGILL), 7 (SIGBUS), 8
(SIGFPE), 11 (SIGSEGV), and 15 (SIGTERM) on JVM threads cause the JVM
to shut down; therefore, an application signal handler should not
attempt to recover from these unless it no longer requires the JVM.
Since your jvm doesn't exit, you may need to check whether there is:
Any use of Runtime.addShutdownHook
Existence of the -Xrs option on JVM startup
Any use of sun.misc.SignalHandler.
Here is the AbstractApplicationContext.registerShutdownHook() in Spring source code.
public void registerShutdownHook() {
if (this.shutdownHook == null) {
// No shutdown hook registered yet.
this.shutdownHook = new Thread() {
#Override
public void run() {
doClose();
}
};
Runtime.getRuntime().addShutdownHook(this.shutdownHook);
}
}
I have an embedded Jetty 6.1.26 instance.
I want to shut it down by HTTP GET sent to /shutdown.
So I created a JettyShutdownServlet:
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
resp.setStatus(202, "Shutting down.");
resp.setContentType("text/plain");
ServletOutputStream os = resp.getOutputStream();
os.println("Shutting down.");
os.close();
resp.flushBuffer();
// Stop the server.
try {
log.info("Shutting down the server...");
server.stop();
} catch (Exception ex) {
log.error("Error when stopping Jetty server: "+ex.getMessage(), ex);
}
However, when I send the request, Jetty does not stop - a thread keeps hanging in org.mortbay.thread.QueuedThreadPool on the line with this.wait():
// We are idle
// wait for a dispatched job
synchronized (this)
{
if (_job==null)
this.wait(getMaxIdleTimeMs());
job=_job;
_job=null;
}
...
2011-01-10 20:14:20,375 INFO org.mortbay.log jetty-6.1.26
2011-01-10 20:14:34,756 INFO org.mortbay.log Started SocketConnector#0.0.0.0:17283
2011-01-10 20:25:40,006 INFO org.jboss.qa.mavenhoe.MavenHoeApp Shutting down the server...
2011-01-10 20:25:40,006 INFO org.mortbay.log Graceful shutdown SocketConnector#0.0.0.0:17283
2011-01-10 20:25:40,006 INFO org.mortbay.log Graceful shutdown org.mortbay.jetty.servlet.Context#1672bbb{/,null}
2011-01-10 20:25:40,006 INFO org.mortbay.log Graceful shutdown org.mortbay.jetty.webapp.WebAppContext#18d30fb{/jsp,file:/home/ondra/work/Mavenhoe/trunk/target/classes/org/jboss/qa/mavenhoe/web/jsp}
2011-01-10 20:25:43,007 INFO org.mortbay.log Stopped SocketConnector#0.0.0.0:17283
2011-01-10 20:25:43,009 WARN org.mortbay.log 1 threads could not be stopped
2011-01-10 20:26:43,010 INFO org.mortbay.log Shutdown hook executing
2011-01-10 20:26:43,011 INFO org.mortbay.log Shutdown hook complete
It blocks for exactly one minute, then shuts down.
I've added the Graceful shutdown, which should allow me to shut the server down from a servlet; However, it does not work as you can see from the log.
I've solved it this way:
Server server = new Server( PORT );
server.setGracefulShutdown( 3000 );
server.setStopAtShutdown(true);
...
server.start();
if( server.getThreadPool() instanceof QueuedThreadPool ){
((QueuedThreadPool) server.getThreadPool()).setMaxIdleTimeMs( 2000 );
}
setMaxIdleTimeMs() needs to be called after the start(), becase the threadPool is created in start(). However, the threads are already created and waiting, so it only applies after all threads are used at least once.
I don't know what else to do except some awfulness like interrupting all threads or System.exit().
Any ideas? Is there a good way?
Graceful doesn't do what you think it does - it allows the server to shutdown gracefully, but it does not allow you to shutdown from inside a servlet.
The problem is as described in the mailing-list post you linked to - you're trying to stop the server, while you're still processing a connection inside the server.
You should try changing your servlet's implementation to:
// Stop the server.
new Thread()
{
public void run() {
try {
log.info("Shutting down the server...");
server.stop();
log.info("Server has stopped.");
} catch (Exception ex) {
log.error("Error when stopping Jetty server: "+ex.getMessage(), ex);
}
}
}.start();
That way the servlet can finished processing while the server is shutting down, and will not hold up the shutdown process.