How to programmatically shutdown embedded jetty server?
I start jetty server like this:
Server server = new Server(8090);
...
server.start();
server.join();
Now, I want to shut it down from a request, such as http://127.0.0.1:8090/shutdown
How do I do it cleanly?
The commonly proposed solution is to create a thread and call server.stop() from this thread.
But I possibly need a call to Thread.sleep() to ensure that the servlet has finished processing the shutdown request.
I found a very clean neat method here
The magic code snippet is:-
server.setStopTimeout(10000L);;
try {
new Thread() {
#Override
public void run() {
try {
context.stop();
server.stop();
} catch (Exception ex) {
System.out.println("Failed to stop Jetty");
}
}
}.start();
Because the shutdown is running from a separate thread, it does not trip up over itself.
Try server.setGracefulShutdown(stands_for_milliseconds);.
I think it's similar to thread.join(stands_for_milliseconds);.
Having the ability for a Jetty server to be shutdown remotely through a HTTP request is not recommended as it provides as potential security threat. In most cases it should be sufficient to SSH to the hosting server and run an appropriate command there to shutdown a respective instance of a Jetty server.
The basic idea is to start a separate thread as part of Jetty startup code (so there is no need to sleep as required in one of mentioned in the comment answers) that would serve as a service thread to handle shutdown requests. In this thread, a ServerSocket could be bound to localhost and a designated port, and when an expected message is received it would call server.stop().
This blog post provides a detailed discussion using the above approach.
Related
I have a product service in Java. In our code I am creating shut down hook, but when I stop service it is not calling shut down hook consistently. Out of 5 stop calls it has called shutdown hook only once.
Runnable shutdownHandler = new Runnable() {
#Override
public void run() {
s_log.info("Shutting down thread..");
}
};
Runtime.getRuntime().addShutdownHook(
new Thread(shutdownHandler, "shutdownthread"));
Can anybody please tell me what could be the reason behind this not getting called consistently?
Check the following code:
Runnable shutdownHandler = new Runnable() {
#Override
public void run() {
System.out.println("Shutting down thread..");
}
};
Runtime.getRuntime().addShutdownHook(
new Thread(shutdownHandler, "shutdownthread"));
and if it gives you expected output, you need to check the documentation of your logging framework.
I am also finding that my framework (Jooby) and Java shutdown hooks work fine on my Mac on IntelliJ which sends a kill SIGINT (-2) however on Ubuntu Server 20.04 LTS they don't run.
As my Java app is a webapp I came up with a simple workaround:
Setup a controller to listen to some url that isn't easily guessable e.g.
/exit/fuuzfhuaBFDUWYEGLI823y82941u9y47t3u45
Have the controller simply do the following:
System.exit(0)
Do a curl or wget from a script to the URL and the shutdown hooks all fire as JVM comes down.
I suspect for some reason on Linux there is a bug and no matter what interrupt that I use besides SIGKILL they all effectively behave like SIGKILL and the JVM comes down hard/abruptly.
I am trying to come up with a microservice using dropwizard.
The documentation tells how to start the application, but says nothing about terminating it gracefully. Fir example, apache tomcat has both startup and shutdown scripts.
So does anyone know how to terminate a dropwizard application other than pressing Ctrl+C of kill ?
Dropwizard Jetty has shutdown hooks. So kill -SIGINT <pid> works out really well.
Inspired by praveenag's answer, I dug into Jetty's code.
If you start DropWizard providing:
-DSTOP.PORT=xxxx -DSTOP.KEY=my_secret_key
as Java options,
It tells Jetty to listen on that port for a stop request.
You can then write to that socket to tell Jetty to shutdown. I did this from R like this:
socket = make.socket("localhost", 8082)
write.socket(socket, "my_secret_key\r\n")
write.socket(socket, "stop\r\n")
close.socket(socket)
I guess you can do the same from any other language.
The other answers here are great, but if you want to go a bit further up the stack and easily add custom logging / security / arbitrarily complex shutdown logic, then adding a shutdown hook via a dropwizard admin Task is a nice pattern.
Create a shutdown Task with any logic you like
import io.dropwizard.servlets.tasks.Task;
public class ShutdownTask extends Task {
public ShutdownTask() {
super("shutdown"); // the task name, used in the endpoint to execute it
}
public void execute(
ImmutableMultimap<String, String> immutableMultimap,
PrintWriter printWriter
) throws Exception {
// kill the process asynchronously with some nominal delay
// to allow the task http response to be sent
new Timer().schedule(new TimerTask() {
public void run() {
// any custom logging / logic here prior to shutdown
System.exit(0);
}
}, 5000);
}
}
Register the task in Application.run()
environment.admin().addTask(new ShutdownTask());
And then execute it via a POST to the following endpoint on the admin port
http://localhost:<ADMIN PORT>/tasks/shutdown
I am assuming this is not a question for your development environment but for your deployments. The answer depends on your deployment strategy. In the past we have handled deployments where the drop wizard application is bundled as a java process that can be started and the pid being recorded and forcefully kill the process. Or bundle the java process in an upstart/init script to gracefully start and shutdown the system.
On the other hand when you start a dropwizard application what it eventually does is start a jetty server. http://eureka.ykyuen.info/2010/07/26/jetty-stop-a-jetty-server-by-command. This can maybe shed some light on how you can pass the stop port as arguments when you start the dropwizard application.
Programmatically, you can also do:
environment.getApplicationContext().getServer().stop();
That's the environment you get in your Application:
#Override
public void run(ApplicationConfiguration configuration, Environment environment) throws Exception { ... }
In Java:
// In case vm shutdown
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run()
{
// what should be closed if forced shudown
// ....
LOG.info(String.format("--- End of ShutDownHook (%s) ---", "APPLICATION_NAME"));
}
});
Build your own strategy, how to shutdown your app.
Most servers like Jetty or Weblogic provide some sort of mechanism to start or shutdown the server via a call (sometimes over jmx). Can always to keep stuff running from main:
public static void main(String args[]) {
try {
Setup Spring context.....
Object lock = new Object();
synchronized (lock) {
lock.wait();
}
} catch (Exception e) {
e.printStackTrace();
}
}
And do a kill on the process PID. But thought there might a nice open-source solution hanging around in cyber space that adds JMX bells and whistles.
Maybe you want to start by looking at a combination of Commons Daemon and Commons Launcher. Not sure if there are any JMX bells and whistles in there, but this provides hooks into your OS management interface for server processes.
As the topic suggests I have a server and some clients.
The server accepts I/O connections concurrently (no queueing in socket connections) but I have this troubling issue and I do not know how to bypass it!
If I force a client to throw an I/O Exception the server detects it and terminates the client thread correctly (verified from Task Manager (Windows) and System Monitor (Ubuntu) ). But If I emulate an I/O that is "hanging" like i.e. Thread.sleep(60*1000);or
private static Object lock = new Object();
synchronized(lock) {
while (true) {
try {
lock.wait();
} catch (InterruptedException e) {
/* Foo */
}
}
}
then all subsequent I/O operations (connection & data transfer) seem to block or wait until the "hanging" client is terminated. The applications makes use of the ExecutorService so if the "hanging" client does not complete the operations in the suggested time limit then the task will time out and the client is forced to exit. The subsequent "blocked" I/Os will resume but I wonder why the server doesn't accept any I/O connections or performs any I/O operations when a client "hangs"?
NOTE:The client threading takes place in the server main like this:
while (true) {
accept client connection;
submit client task;
||
\ /
\/
// ExecutorService here in the form
// spService.submit(new Callable<Tuple<String[], BigDecimal[]>>() {
// ... code ... }}).get(taskTimeout, taskTimeUnit);
check task result & perform cleanup if result is null;
otherwise continue;
}
The Problem :
This may very well indicate that your server ACCEPTS client connections concurrently, however, it only handles these connections synchronously. That means that even if a million clients connect, successfully, at any given time, if anyone of them takes a long time (or hangs), it will hold up the others.
The TEST:
To verify this : I would toggle the amount of time a client takes to connect by adding Thread.sleep statments(1000) in your clients.
Expected result :
I believe you will see that even adding a single Thread.sleep(1000) statement in your client delays all other connecting clients by 1000.
I think I have found the source of my problems!
I do use one thread-per-client model but I run my tests locally i.e. in the same machine which means all of them have the same IP! So each client is assigned the same IP with the server! I guess that this leaves server and clients to differ only in port number but since each client is mapped to a different localport for each server connection then the server shouldn't block. I have confirmed that each client and server use different I/Os (compared references) and I wrap their sockets using <Input/Output>Streams to BufferedReaders & PrintWriters but still when a client hangs all other clients hang too (so maybe the I/O channels are indeed the same???)!I will test this on another machine and check the results back with you! :)
EDIT: Confirmed the erratic behaviour. It seems that even with remote clients if one hangs the other clients seem to hang too! :/
Don't know but I am determined to fix this. It's just that it's pretty weird since I am pretty sure I use one thread-per-client (I/Os differ, client sockets differ, IPs seem to be not a problem, I even map each client in the server to a localport of my choice ...)
May be I'll switch to NIO if I don't find a solution soon enough.
SOLUTION: Solved the problem! It seemed that the ExecutorService had to be run in a seperate thread otherwise if an I/O in a client blocked, all I/Os would block! That's strange given the fact that I've tried both an Executors.newFixedThreadPool(<nThreads>); and Executors.newCachedThreadPool(); and the client actions (aka I/Os) should take place in a new Thread for each client.
In any case, I used a method and wrapped the calls so each client instace would use a final ExecutorService baseWorker = Executors.newSingleThreadExecutor(); and created a new Thread explicitly each time using <Thread instance>.start(); so each thread would run in the background :)
I am currently setting up a Tomcat Server that is running an embedded ActiveMQ broker. I'm using Spring to configure JMS. I wrote a test application to listen to topics and when I exit the test application the ActiveMQ broker throws a SocketException. Here is the exception:
2009-06-09 13:12:48,005 DEBUG Transport:229 - Transport failed: java.net.SocketException: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
at java.lang.Thread.run(Thread.java:619)
My test application just loads the context xml file, loops sleeps every 60 seconds.
public static void main(String[] args) {
long numMillisecondsToSleep = 60000;
new ClassPathXmlApplicationContext("client-context.xml");
while (true) {
try {
logger.info("Sleeping for 60 seconds");
Thread.sleep(numMillisecondsToSleep);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
I was wondering if anyone might know why I'm getting this exception. I'm very new to ActiveMQ and JMS in general so any ideas would be greatly appreciated.
I think there's a way to shut down an embedded broker nicely via a method call. Posting your config along with your embedding code will help people figure out what's going on.
I would recommend looking in to Spring's JmsTemplates; it wraps all of the complexity of MQ into a nice package and ensures little errors like this dont cause you hours of problems.
http://activemq.apache.org/spring-support.html