Netty causing memory leak in tomcat - java

I'm using Netty 4.0.26.Final along with Tomcat 8 to implement an embedded TCP server inside a web application.
The problem is that when I stop Tomcat I'm getting this message :
The web application [app] appears to have started a thread named
[nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
I've tried using this class to shutdown the threads before closing tomcat but I'm still having the same problem.
#Component
public class MyAppServletContextListener implements ServletContextListener{
#Override
public void contextInitialized(ServletContextEvent arg0) {
}
#Override
public void contextDestroyed(ServletContextEvent arg0) {
TrackingDaemon trackingDaemon= (TrackingDaemon) ApplicationContextProvider.getBean("trackingDaemon");
trackingDaemon.bossGroup.shutdownGracefully();
trackingDaemon.workerGroup.shutdownGracefully();
}
}
BossGroup and workerGroup are:
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
Any help will be appreciated..

I have had this problem with netty 4 and tomcat 7 and solved it by capturing the future event from shutdownGracefully() and awaiting its completion.
The method below is called when closing down the application.
private void stopChannel() {
log.trace("Stopping server socket on port '{}'", port);
sendMessageService.deregisterMessageSender(endpointId);
try {
for (Channel channel : channels) {
if (channel != null) {
channel.close();
}
}
serverChannel.close().sync();
log.trace("Server socket on port '{}' stopped.", port);
} catch (InterruptedException e) {
throw new RuntimeException(e);
} finally {
Future<?> fc = connectionGroup.shutdownGracefully();
Future<?> fw = workerGroup.shutdownGracefully();
try {
fc.await(); // when shutting down in tomcat waits for the netty threads to die
fw.await();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
log.trace("Server workerGroup has shutdown successfully on port '{}'", port);
}
}

I guess you will need to do:
trackingDaemon.bossGroup.shutdownGracefully().syncUninterruptibly();
trackingDaemon.workerGroup.shutdownGracefully().syncUninterruptibly();

Related

What is the proper way to gracefully shutdown a Grizzly server? (Embedded with Jersey)

I have the following piece of code to start a basic Embedded Grizzly server running with Jersey.
private static void startServer() {
ServerResourceConfiguration configuration = new ServerResourceConfiguration();
HttpServer server = GrizzlyHttpServerFactory.createHttpServer(
URI.create(BASE_URI),
configuration,
false,
null,
false);
server.start();
if (System.in.read() > -2) {
server.shutdownNow();
}
}
This does not look like production level way to stop a server.
What is the best practice to gracefully shut it down ?
I guess a terminal command of some sort. Killing the process would work but it is not very graceful.
I am using Gradle on this project and runs the server with the gradle run command.
Could a Gradle task do the job?
Also I have seen this about gracefully terminating a grizzly transport:
http://grizzly-nio.net/2013/08/gracefully-terminating-a-grizzly-transport/
But I am not sure if I would need to use it. I don't understand how to use it.
EDIT: I came across this post:
https://stackoverflow.com/a/15391081/3982755
Is that an acceptable way to terminate an Http server in a production environment?
There is no answer so I will post my own, I implemented it with a Shutdown Hook and it works very well.
The server will wait for all connections to terminate before shutting down.
To avoid getting blocked for ever if a connection never terminates, we set a grace period(60 seconds)
After the grace period the server will force termination of all connections
Here is the code for the hook to be run when the server receives a SIGINT or SIGTERM signal.
public class GrizzlyServerShutdownHookThread extends Thread {
public static final String THREAD_NAME = "Grizzly Server Shutdown Hook";
public static final int GRACE_PERIOD = 60;
public static final TimeUnit GRACE_PERIOD_TIME_UNIT = TimeUnit.SECONDS;
private final HttpServer server;
/**
* #param server The server to shut down
*/
public GrizzlyServerShutdownHookThread(HttpServer server) {
this.server = server;
setName(THREAD_NAME);
}
#Override
public void run() {
LOG.info("Running Grizzly Server Shutdown Hook.");
LOG.info("Shutting down server.");
GrizzlyFuture<HttpServer> future = server.shutdown(GRACE_PERIOD, GRACE_PERIOD_TIME_UNIT);
try {
LOG.info(format("Waiting for server to shut down... Grace period is %s %s", GRACE_PERIOD, GRACE_PERIOD_TIME_UNIT));
future.get();
} catch(InterruptedException | ExecutionException e) {
LOG.error("Error while shutting down server.", e);
}
LOG.info("Server stopped.");
}
}
Then I register the Hook into the RunTime object this way when I setup the server:
Runtime.getRuntime().addShutdownHook(
new GrizzlyServerShutdownHookThread(server)
);
And finally, I start the server this way:
try {
server.start();
} catch (IOException e) {
throw new RuntimeException(e);
}
// wait for a SIGINT (Ctrl+c) signal to shut down
try {
LOG.info("Press CTRL^C to exit..");
Thread.currentThread().join();
} catch(InterruptedException e) {
throw new RuntimeException(e);
}

Shutdown Embedded Jetty Server programmatically in Eclipse RCP

I have embedded Jetty into my Eclipse RCP application successfully.
In my RCP application, when user click some button, a browser will be opened and some servlet page shown. The jsp files are in a separated directory, it is a angulajs web application.
I am trying to shutdown embedded Jetty server from Eclipse UI plugin when user closes the RCP.The server is started in a class named Workshop which is part of web project, so I dont have access to Server instance to call, server.stop() from Eclipse UI Plugin.I have tried below code, but in vein.
1>Configure ShutdownHook to Workshop class of web project
server = new Server();
server.setConnectors(new Connector[] { connector });
server.setHandler(handlers);
server.start();
handlers.addHandler(new ShutdownHandler(server, "abc"));
server.setStopAtShutdown(true);
server.setGracefulShutdown(7_000);
ShutdownThread.getInstance().run();
2> In my Eclipse UI Plugin, I have added
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
try {
URL url = new URL("http://localhost:" + resultPortNo + "/shutdown?token=" + shutdownCookie);
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.getResponseCode();
logger.info("Shutting down " + url + ": " + connection.getResponseMessage());
} catch (SocketException e) {
// logger.debug("Not running");
// Okay - the server is not running
} catch (IOException e) {
throw new RuntimeException(e);
}
}
});
HtppUrlConnection throws 404-NotFound response code.So Jetty server is still running. How do we handle embedded Jetty shutdown from Eclipse UI Plugin.
I did read lot of articles, but cannot find answer to my question.
Any help will be appreciated.Thank you.
Problem : Every time jettyServer.stop was called, Interrupt Exception was thrown and jetty server continued to run.
Solution : (1) Added Executor Service with daemon thread in the Servlet code to stop the Jetty server
JettyShutdownServlet.js
-----------------------
ExecutorService pool = Executors.newSingleThreadExecutor(new ThreadFactory() {
#Override
public Thread newThread(Runnable runnable) {
Thread thread = Executors.defaultThreadFactory().newThread(runnable);
thread.setDaemon(true);
return thread;
}
});
pool.execute(new Runnable() {
#Override
public void run() {
if (null != jettyServer) {
try {
jettyServer.stop();
jettyServer.join();
} catch (Exception e) {
logger.info("Error when stopping Jetty: " + e.getMessage());
}
}
}
});
(2) Adding the servlet inside the Jetty startup code.
servletContextHandler.addServlet(new ServletHolder(new JettyShutdownServlet(server)), "/shutdown");
(3) Adding shutdownhook to Eclipse UI class
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
try {
String shutdownurl = "http://localhost:" + resultPortNo + "/api/shutdown";
URL url = new URL(shutdownurl);
HttpURLConnection connection =(HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
try {
connection.getResponseCode();
connection.disconnect();
} catch (SocketException e) {
// logger.debug("Not running");
// Okay - the server is not running
connection.disconnect();
}
} catch (Exception x) {
// exception during shutdown
Activator.error("Unable to shutdown Jetty Server.\n" + x.getMessage());
}
}
});
This solved my problem and hope it will be of some help to others.

tyrus websocket connectToServer - how to clean up daemon threads

tyrus websockets ClientManager connectToServer 'Handshake response not received'
how do I retry the connection without more and more daemon and Grizzly-kernel and Grizzly-worker threads created.
Is there a call to Session or client to kill/cleanup
Thread-1 to 4 and Grizzly-kernel and Grizzly-worker threads?
Example JAVA main line which attempts forever to make and maintain a connection with a server which may not be running or is periodically restart.
public void onClose(Session session, CloseReason closeReason) {
latch.countDown();
}
enter code here
public static void main(String[] args) {
while (true) {
latch = new CountDownLatch(1);
ClientManager client = ClientManager.createClient();
try {
client.connectToServer(wsListener.class, new URI("wss://<host>/ws"));
latch.await();
}
catch (DeploymentException e) {
try {
Thread.sleep(1000);
} catch (InterruptedException ie) {
break;
}
}
catch (Exception e) {
throw new RuntimeException(e);
}
client = null;
latch = null;
// HERE... clean up
}
}
client.connectToServer returns Session instance and when you call Session.close(), client runtime should be shut down (no threads left).
You did not specify version of Tyrus you are using (I recommend 1.3.3, we made some improvements in this area). Also you might be interested in our shared container support, see TYRUS-275. You could combine it with Thread pool config and you should have much better control of number of spawned/running threads.
We are always looking for new use cases, so if you think you have something which should be better supported in Tyrus, feel free to create new enhancement request on our JIRA.
I got this exact same behavior. I was using a lot of threads and synchronization and managed to accidently get the onOpen method of the ClientEndpoint blocking which caused the handshake to time out.

How to run multiple threads concurrently for Automated Selenium testing?

I have one web application from which I can start respective testing process which is in another Java app. I am using Socket Programming for communication between Web app and Java app.
When I request for specific process from web app then SocketServer from Java app hears request and start one thread for Testing process.
Testing process will initialize FirefoxDriver and start browser and do further test process.
My problem is, when I request for another process with different process name then It again creates second thread and start firefox browser but this time it is not considering my second process, it started to do same process which is first thread have.
I do not understand what to do...For every process I created a new thread but further it will doing same process.
My inputs are correctly received at Java app.
Please help me How can I do the concurrent thread safe processing?
I am using GWT,Java, Seleniun FirefoxDriver.
here is the Server code which is running in background and listen client request:
static final int PORT = 6789;
public static void main(String args[]) {
ServerSocket serverSocket = null;
Socket socket = null;
try {
InitializeApplication application = new InitializeApplication();
application.initialize();
serverSocket = new ServerSocket(PORT);
} catch (Exception e) {
log("Exception in SocketServerExecutor !!!",e);
}
while (true) {
try {
socket = serverSocket.accept();
} catch (Exception e) {
log("Exception in SocketServerExecutor !!!",e);
}
Thread thread = new Thread(new SocketServerThread(socket));
thread.start();
}
}
and this is the Thread which start process:
private Socket client;
public SocketServerThread(Socket serverSocket) {
this.client = serverSocket;
}
/**
* Starts appropriate process depending on process name from input.
* Input string contains:
* process name
*/
public void run() {
DataOutputStream outputStream = null;
String param = null;
try{
log("Just connected to "+ client.getRemoteSocketAddress());
try {
while ((param = in.readUTF()) != null){
log("got parameteres from client (i.e. from web app): "+param);
break;
}
} catch (Exception e) { }
if(param!=null && !param.isEmpty()){
String process = params[0];
ProcessManager manager = new ProcessManager();
if(process.equals("testUser"))
manager.startUserProcess(process);
else if(process.equals("testCustomer"))
manager.startCustomerProcess(process);
}
}catch(Exception exc){
if(exc instanceof SocketTimeoutException)
log("Socket timed out! [SocketServerThread]",exc);
else if(exc instanceof BindException)
log("BindException in SocketServerThread !!!",exc);
log(Level.SEVERE, "Exception in SocketServerThread !!!",exc);
}
}
and this is ProcessManager:
public void starUserProcess(String siteName) {
ExecutorService executerService = null;
try{
Callable<Object> callable = new ProcessThread(siteName);
executerService = Executors.newCachedThreadPool();
Future<Object> future = executerService.submit(callable);
future.get();
log("[ProcessManager] Process completed for "+process);
System.exit(0);
}catch (Exception e) {
log("[ProcessManager]::Exception");
log(ex);
}
}
ProcessThread will initialize all required things and Firefox browser and start process.
Client is new every time which contains input.
One of two things that I can thing of off the top of my head could be happening.
You are passing a paramater into your run() function that links back to the initial thread or...
You are using a shared variable that is accessible to all threads and the variable is either not being updated properly, or not being updated at all.
If you could include an SSCCE, that would help us determine where the problem truly lies.
Got the solution: I created the ExecutorService using newSingleThreadExecutor() instead of newCachedThreadPool() and also setDeamon(True) for every newly created ServerThread.
Here is the Executor documentation

Handling multiple clients in Socket

I have developed a java swing client-server application. The server has many services like database service, cache service and client service talks to the clients.
The client service opens a socket on a port and listens to incoming connections. It spawns a new thread for every client connection, creates a session and reads the incoming serialized object. It maintains this session (keeps the thread alive) till the client issues a 'CLOSE_SESSION' command.
What i would like to know is if its correct to spawn a new thread for every new client-socket session. Thanks.
My client service code is as below.
Code to create server socket:
try {
ServerSocket socket = new ServerSocket(serverPort);
Socket listener = socket.accept();
Thread client = new Thread(new ClientHandler(listener));
client.start();
} catch (IOException ex) {
log.error(new Throwable(ex));
}
Code to spawn new thread for every client
class ClientHandler implements Runnable {
private static Logger log = Logger.getLogger(ClientHandler.class);
private Socket listener;
public ClientHandler(Socket listener) {
this.listener = listener;
}
public void run() {
try {
ObjectInputStream inStream = new ObjectInputStream(
listener.getInputStream());
try {
ServiceRequestResponse request = (ServiceRequestResponse) inStream
.readObject();
if (request != null && request.getServiceCommand() != null) {
ServiceCommand command = request.getServiceCommand();
log.debug("command : " + command.getCommand());
log.debug("is session alive? " + request.isAlive());
log.debug("ServiceCommand.CREATE_SESSION : "
+ ServiceCommand.CREATE_SESSION.getCommand());
if (!request.isAlive()
&& command.getCommand().equals(
ServiceCommand.CREATE_SESSION.getCommand())) {
// No session yet, and service command issued is login.
// Call login service, check credentials and create
// session.
request.setSessionId(UUID.randomUUID());
log.debug("Created user session with id : "
+ request.getSessionId());
} else {
if (command.getCommand().equals(
ServiceCommand.CLOSE_SESSION)) {
// Close session and do clean up here
}
// Here session is alive.
while (!ServiceCommand.CLOSE_SESSION.equals(command
.getCommand())) {
// Read the service command from the request
// response and
// Hand it over to the appropriate handler.
}
}
}
} catch (ClassNotFoundException ex) {
log.error(new Throwable(ex));
}
} catch (IOException ex) {
}
}
}
If your client session request can last long then thread-per-connection is a good solution.
Alternatives are:
Using NIO;
Using thread pool if client requests are short.

Categories

Resources